Calculators — algorithms to extract and calculate intervals

Preconfigured algorithms storage for intervals extraction and calculating

Calculators are the part of the flespi analytics engine used to describe custom logic to extract intervals from device messages. 

Once you have created and configured a calculator, it is possible to refer to its configuration from POST /gw/devices/{selector}/calculate REST API call instead of specifying selectors and counters in the API call parameters. Thus calculators simplify the usage of flespi analytics engine in manual mode

But the primary goal for using calculators is to let flespi do all the calculations automatically in the background. To do so, you need to assign devices to a calculator. Once the devices are assigned, their intervals calculation will start promptly. And once the calculation for the device is finished, the assigned device will get the synced=true attribute, and from that moment you can request intervals and even attach your own information to them via GET|PUT /gw/calcs/{selector}/devices/{selector}/intervals/{interval-selector} REST API calls.

Calculator configuration that you will need in both automatic and manual modes consists of:

  • interval selectors which task is to split all source messages into intervals, where each interval has begin time, end time, and duration. It is possible to have intervals consisting of one message only, i.e. with zero duration. 
  • "validate_message" contains a boolean expression that will check if a message is suitable for any kind of calculations for interval selector at all. You can use it to completely skip messages that do not contain a certain parameter for example.
  • counters that perform specific calculations with each interval (for all its messages) and generate a JSON field in the final interval with their name and calculated data, like "mileage" and "positions" in the sample above. Extra option "validate_interval" contains a boolean expression that will check generated interval JSON representation and may filter it out if validation fails.
  • "timezone" with which you may specify timezone to be used in all counters and selectors.

Additional calculator configuration for automatic mode consists of:

  • "update_delay" specifies how many seconds a calculator should wait after the moment it received the new message to start recalculation. The default value is 10 seconds. For near real-time processing, you may specify a 1-second delay while for better notifications accuracy and lower number of interval events, you may set a 60-second delay or even more. Be aware that analytics may activate additional throttling for your interval detection events if the logic in selectors or counters is too heavy, intervals are too long or update_delay value is too low. This throttling system is activated automatically and maximum additional delay it can add is 2 minutes. That's why our recommendation for update_delay value is to keep it from 10 to 30 seconds.
  • "intervals_ttl" contains the duration of history storage for calculated intervals. Intervals usually weigh much less than raw messages from devices (about 1% of it) and you may configure a calculator to store intervals for a few years while device messages can be stored for a few days or months only.
  • "intervals_rotate" — if non-zero, specifies maximum storage size for calculated intervals. If newly detected intervals overflow specified size, earlier intervals will be removed automatically from the storage. This parameter may be used to limit the maximum storage size for the calculator.
  • "update_period" specifies a valid interval update period from the current moment. During any recalculation, all intervals with end time outside this period will be ignored. Even if recalculation was triggered by the calculator parameters change. If set to standard value like 30-60 days, this parameter is a good way to ignore any outdated messages and retain all generated intervals as is until they are auto-deleted by intervals_ttl. The best usage is when update_period < messages_ttl(device) < intervals_ttl.
  • "update_onchange" — controls the behavior of analytics system when calculator configuration is changed. With TRUE value (default) the system will synchronize all devices intervals according to a new calculator configuration within update_period (10 days by default). So if you change selectors, counters, or validation fields — you will immediately see the effect with intervals created, updated, deleted events. While FALSE value will prevent any automatic synchronization to currently existent intervals and will only use the new configuration for new device messages processing. This is convenient to set in production level calculators with thousands of devices assigned to prevent flooding of your handlers with intervals update events.
  • "messages_source" allows specifying the data source for the interval calculation — it can be either raw messages from the device or interval messages from another calculator. This is usually used to calculate statistics over a date range, e.g. you may have a calculator for each individual trip and have one calculator on top that will calculate daily stats for all trips — how many trips were done, what the total duration and mileage are, etc.

If a calculator is configured to take source messages not from a device but from another calculator, make sure that your counters are referencing different parameters. For example, you may have trips calculator with a mileage counter of type="expression" and expression="mileage()". To detect trips it will use selector of type="expression" with expression="mileage()>0.01" — e.g. detect any movement based on the GPS signal. Do not forget to specify other properties in interval selector to correctly filter false trips and merge all short trips into one like: max_inactive=120, max_messages_time_diff=600, min_duration=120, merge_messages_before=true. And also validate_interval="mileage>0.5" to filter trips with short final mileage.

After assigning devices to a calculator you will have information about their detailed trips. But now you may want to have monthly stats calculated by flespi. You can do so by creating an additional calculator with messages_source={"type":"calculator","calc_id":ID} and referencing the ID of the previously created calculator. The selector will be type="datetime" with split="month". Now you want totally different counters. For example, to calculate mileage you use the counter with type="expression", expression="mileage", method="summary" to summarize all trips' mileage into a single number. And to have the total quantity of trips within a month you may use the counter with name="trips_count", type="expression", expression="1", method="summary".

Once you have configured the calculator the next step will be to assign devices to it.

See also
Apply webhooks to events from calculators to invoke your lambda upon an aggregated event happened to the device
Forwarding intervals generated by the flespi analytics engine into your solution.