Automatically translate this page?

Analytics — flespi reports engine

Apply custom mathematics to extract information from device messages

Each registered device on the platform will accumulate its messages received via channels until these messages fall within the specified messages_ttl period. It is possible to access the current device state by fetching its telemetry or to read received historical messages via GET /gw/devices/{selector}/messages REST API call

Both telemetry and historical device messages are operating with parameters being sent by the device in a raw format — the one defined by the protocol. Let's take a look at a sample message from a device:

{
  "ident": "123456789012345",
  "timestamp": 1490347944.893743,
  "din": 9,
  "channel.id": 123,
"device.id": 345,
  "position.altitude": 568.49,
  "position.direction": 297,
  "position.hdop": 0.9,
  "position.latitude": -21.328481,
  "position.longitude": 47.562136,
  "position.speed": 0,
  ...
}

If the values you need are available directly in the message JSON, you can call GET /gw/devices/{selector}/messages with a specific filter (e.g. filter="position.speed>0") or limit data output to a list of specific parameter names you want to receive. This gives direct and fast access to devices' historical data.

To perform calculations with received messages, e.g. calculating the duration of an input sensor state, detecting entries to and exits from geofences, counting mileage or engine hours, and even splitting all messages into trips and parkings flespi database engine provides you with the Analytics system.

Analytics system is being in active development at the moment and all REST API calls described here are subject to change. If you intend to use this functionality for commercial projects, please let us know so we can make sure to notify you before applying incompatible API changes.

Analytics system is designed to operate in two modes: manual and automatic.

Manual mode is provided via GET /gw/devices/{selector}/intervals REST API call. It allows you to specify the way how to split device messages into intervals and how to apply custom calculations to all messages inside the interval. This mode is used to generate reports upon request with all configuration specified in REST API call parameters.

Automatic mode is doing intervals calculations automatically in the background. Once the device sends a message (even from a black box or in reverse mode) the system will recalculate intervals and their counters and merge these updated intervals with the previously calculated ones. It will also recalculate intervals if the configuration of intervals selectors or counters is changed. 

Each newly generated interval will be assigned its own ID and published with its counters as an MQTT message. Any custom handler on the client side can handle this event, perform own calculations for the specified interval with own logic and mathematics that is not possible to implement in flespi and this custom data can be attached to the interval and made available together with the counters calculated by flespi automatically. 

Automatic mode enables you to:

  • apply custom logic or make injections to interval data from own database
  • maintain calculated device data for years with minimal storage volume for raw messages
  • have instant report results even for thousands of devices
  • handle notifications about complex device state changes: e.g. current daily mileage or parking started events

Automatic analytics mode is not available at the moment. We plan to release it in 2019Q2.

To understand how analytics works we recommend starting with the manual mode because it provides instant results for your requests. Later, when you get acquainted with the manual mode, interval selectors and counters, you can copy the same configuration into the automatic system.

The internal algorithm used by the analytics system is quite simple. We have a range of messages and our task is to split these messages into intervals and calculate counters for each interval.

Each interval is represented as a JSON message with some a set of fields, where "begin", "end" and "duration" (also "id" for automated mode) are always present and other fields appear only when a counter with such name is defined in the call:

{
  "begin": 1490347944,
  "end": 1490347948,
  "duration": 4,
"mileage": 190.2,
"work_hours": 0.12345,
"max_speed": 120,
"positions": [{"t":1490347944, "lat":53.0923, "lon":27.12}, {"t":1490347944, "lat":53.0923, "lon":27.12}, ...]
  ...
}

To split messages into intervals you need to define interval selectors. And to calculate information about each interval you need to specify counters.

Let’s take a closer look at GET /gw/devices/XXX/intervals request. The expected parameters are:

{
  "from": uint32,
  "to": uint32,
  "extend": boolean,

"selectors": [{selector1-config}, {selector2-config}, ...],
"validate_message": "expression",

"counters": [{counter1-config}, {counter2-config}, ...],
"validate_interval": "expression"
}

First, you need to specify the time interval to analyze by passing “from” and “to” parameters as UNIX timestamps. To select all available messages for a device just pass zeros or simply skip these fields. Additional “extend” flag, if set to true, will automatically add one next message beyond “to” to the selected range. This is usually used to make intervals and calculations always correct whatever time borders you pass.

Next, you specify interval selectors, which task is to split all messages into intervals, where each interval has begin time, end time, and duration. It is possible to have intervals consisting of one message only, i.e. with zero duration. 

Extra option "validate_message" contains a boolean expression that will check if a message is suitable for any kind of calculations for interval selector at all. You can use it to completely skip messages that do not contain a certain parameter for example.

And finally, you specify counters that perform specific calculations with each interval (for all its messages) and generate a JSON field in the final interval with their name and calculated data, like "mileage" and "positions" in the sample above. Extra option "validate_interval" contains a boolean expression that will check generated interval JSON representation and may filter it out if validation fails.

To understand what kind of interval selectors you can use, please read more about them here

To understand what information can be extracted into counters please learn more here

And as a complex sample of analytics usage in the manual mode, we recommend following our how to split device messages onto trips or how to calculate daily engine hours guides.


See also
How to calculate vehicle engine hours on a daily basis
Instructions on how to use flespi analytics engine to extract trips from raw device messages