Analytics — flespi reports engine

Apply custom mathematics to extract information from device messages

Friendly advice: if you are new to flespi analytics, please check this page first and this comprehensive guide.

Each registered device on the platform will accumulate its messages received via channels until these messages fall within the specified messages_ttl period. It is possible to access the current device state by fetching its telemetry or reading received historical messages via GET /gw/devices/{selector}/messages REST API call.

Both telemetry and historical device messages operate with the parameters sent by the device in raw format — the one defined by the protocol. Let's take a look at a sample message from a device:

{
  "ident": "123456789012345",
  "timestamp": 1490347944.893743,
  "din": 9,
  "channel.id": 123,
  "device.id": 345,
  "position.altitude": 568.49,
  "position.direction": 297,
  "position.hdop": 0.9,
  "position.latitude": -21.328481,
  "position.longitude": 47.562136,
  "position.speed": 0,
  ...
}

If the values you need are available directly in the message JSON, you can call GET /gw/devices/{selector}/messages with a specific filter (e.g. filter="position.speed>0") or limit data output to a list of specific parameter names you want to receive. This gives direct and fast access to device historical data.

To perform calculations with received messages, e.g. calculate the duration of an input sensor state, detect geofence ins and outs, count mileage or engine hours, and even split all messages into trips and parkings flespi database engine provides you with the Analytics system.

Analytics system is designed to operate in two modes: manual and automatic.

Manual mode is provided via POST /gw/devices/{selector}/calculate REST API call. It allows you to specify how to split device messages into intervals and how to apply custom calculations to all messages inside the interval. This mode is used to generate reports upon request with the configuration specified in the REST API call parameters.

Automatic mode performs interval calculations automatically in the background based on calculator configuration and devices assigned to them. Once the device sends a message (even from a black box or in reverse mode), the system will recalculate intervals and their counters and merge these updated intervals with the previously calculated ones. It will also recalculate intervals if the calculator configuration changes.

Automatic mode enables you to:

apply custom logic or make injections to interval data from your own database
keep calculated device data for years with minimal storage volume for raw messages
have instant report results even for thousands of devices
handle notifications about complex device state changes: e.g. current daily mileage or parking started events

To understand how analytics works we recommend starting with the manual mode because it provides instant results for your requests. Later, when you get acquainted with the manual mode, interval selectors and counters, you can copy the same configuration into the automatic system.

The internal algorithm used by the analytics system is quite simple. We have a range of messages and our task is to split these messages into intervals and calculate counters for each interval.

Each interval is represented as a JSON message with a set of fields, where "begin", "end" and "duration" (also "id" and "timestamp" for automated mode) are always present and other fields appear only when a counter with such name is defined in the call:

{
  "begin": 1490347944,
  "end": 1490347948,
  "duration": 4,
  "mileage": 190.2,
  "work_hours": 0.12345,
  "max_speed": 120,
  "positions": [{"t":1490347944, "lat":53.0923, "lon":27.12}, {"t":1490347944, "lat":53.0923, "lon":27.12}, ...]
  ...
}

To split messages into intervals you need to define interval selectors. And to calculate information about each interval you need to specify counters.

Let’s take a closer look at POST /gw/devices/XXX/calculate request. The expected parameters are:

{
  "from": uint32,
  "to": uint32,
  "extend": boolean,

  "selectors": [{selector1-config}, {selector2-config}, ...],
  "validate_message": "expression",

  "counters": [{counter1-config}, {counter2-config}, ...],
  "validate_interval": "expression"

  "timezone": "timezone"
}

First, you need to specify the time interval to analyze by passing “from” and “to” parameters as UNIX timestamps. To select all available messages for a device just pass zeros or simply skip these fields. Additional “extend” flag, if set to true, will automatically add one next message beyond “to” to the selected range. This is usually used to make intervals and calculations always correct whatever time borders you pass.

Next, you specify interval selectors, which task is to split all messages into intervals, where each interval has begin time, end time, and duration. It is possible to have intervals consisting of one message only, i.e. with zero duration.

Extra option "validate_message" contains a boolean expression that will check if a message is suitable for any kind of calculations for interval selector at all. You can use it to completely skip messages that do not contain a certain parameter for example.

And finally, you specify counters that perform specific calculations with each interval (for all its messages) and generate a JSON field in the final interval with their name and calculated data, like "mileage" and "positions" in the sample above. Extra option "validate_interval" contains a boolean expression that will check generated interval JSON representation and may filter it out if validation fails.

Also with the "timezone" option, you may specify the timezone to be used in all counters and selectors that require it.

To understand what kind of interval selectors you can use, please read more about them here.

To understand what information can be extracted into counters please learn more here.

For analytics usage samples please read corresponding articles in our blog or more technical articles in KB and blog:

Was this helpful?