Intro to flespi layers
If you monitor flespi history from the very beginning, we initially planned to deliver three layers of functionality one on top of the other: Gateway, Registry, and Analytics.
Gateway consists of flespi channels and is used to seamlessly connect to GPS trackers via REST API — either to consume the data or to control devices remotely.
Registry comes into play when you start registering devices on the platform. It provides extra functionality in both directions — splitting and storing incoming messages per device, maintaining actual device telemetry state, and managing device settings via special settings REST API.
And finally, Analytics is the system that performs automatic post-processing of data coming from devices according to the pre-configured logic. In essence, the flespi analytics engine pre-calculates information the user needs to deliver value to customers — like monthly vehicle mileage, engine hours, hours left for the driver, time spent in the geofence, etc.
Plan vs reality
If you have been following flespi evolution for a long while, you remember that the initial schedule of modules release was as follows: 2017Q1 for Gateway, 2017Q2 for Registry, and 2017Q4 for Analytics.
But the reality was quite different and not that optimistic. We did release Gateway exactly on time though.
Registry was released one quarter later than planned — at the end of 2017Q3. And Analytics was postponed for the indefinite future.
In 2018 we spent a lot of time tuning the platform which often involved completely redoing certain internal things. At the moment we are still in the process of a 1-year-long overhaul of our Gateway system. The new implementation will take into account a lot of small issues we face in our day-to-day work and will deliver a qualitatively new experience to us and our users. So, to stay in the same place we need to keep running very fast. Everybody does this, right?
Eventually, in 2019Q1 the first part of Analytics — the one working in manual mode — became available for testing via REST API. So we invite you, dear flespi users, to give it a try and provide us feedback. Soon it will be possible to set up the same configuration to work in automatic mode.
In short, Analytics allows splitting accumulated device or storage container messages into intervals according to certain custom logic (we call it interval selectors) and calculating each interval also using certain custom logic (we call it interval counters).
Strictly technical
Below goes a load of deep technical information. If you are not a developer, you may stop here to prevent brain overload.
What makes flespi Analytics different from other report engines is that it is designed to deal with data coming from GPS devices, and we know that telematics time series data differs from what normal databases operate with.
So let’s take a closer look at GET https://flespi.io/gw/devices/XXX/calculate request. The expected parameters are:
{
"from": uint32,
"to": uint32,
"extend": boolean
"selectors": [{selector1-configuration}, ...],
"validate_message": expression,
"counters": [{counter1-configuration}, ...],
"validate_interval": expression
}First, you need to specify the time interval to analyze by passing “from” and “to” parameters as UNIX timestamps. To select all available messages for a device just pass zeros or simply skip these fields. Additional “extend” flag, if set to true, will automatically add one next message beyond “to” to the selected range. This is usually used to make intervals and calculations always correct whatever time borders you pass.
Interval selectors
Next, you specify interval selectors, which task is to split all messages into intervals, where each interval has begin time, end time, and duration. It is possible to have intervals consisting of one message only, i.e. with zero duration.
We currently support the next types of interval selectors:
expression — allows specifying an interval state based on expression evaluating message parameters. Two methods are allowed for the expression selector: “boolean” (default) — meaning the expression will be evaluated as a boolean and “change” — meaning the new interval will start when the expression result changes.
datetime — allows beginning/ending an interval when the current time equals a specified hour/day/week/month/year in a specified timezone.
geofence — select intervals when message coordinates are in/out of specified geofences. Each geofence can be specified either as a polygon or as a circle.
Each interval selector has its own internal logic described above, determines the state of each analyzed message sequentially, and assigns each message one of the following:
ON — interval can continue
ON_NEW — interval should restart
OFF — interval should stop
UNKNOWN — unable to calculate
So, to detect one interval it is enough to have at least one message that will evaluate to ON or ON_NEW. Then, on top of this, there are some standard parameters for the detection algorithm, that can be specified almost for each type of interval selector:
{
"merge_message_before": boolean,
"merge_message_after": boolean,
"merge_unknown": boolean,
"max_messages_time_diff": uint32,
"invert": boolean,
"min_active": uint32,
"max_inactive": unt32,
"min_duration": uint32
}They are used to set up precise interval selection:
merge_message_after/merge_message_before — when true, the interval will be automatically extended by one message with OFF/UNKNOWN state in each direction. Usually used together with extend=true in POST /gw/devices/{dev-selector}/calculate REST API call.
merge_unknown — when true, if we have an active interval and current message state is unknown, it will be appended to the active interval. The unknown state can appear in an expression type selector when a message does not contain a parameter specified in expression or in a geofence type selector when there are no coordinates in the message.
max_messages_time_diff — if non-zero, specifies maximum timestamps difference between previous and next message to include it into the interval and to apply logic described above with merge_xxx options.
invert — when true, evaluation of interval state for the message should be inverted, .e.g. ON will become OFF.
min_active — if non-zero, all intervals with duration in seconds less than specified will be skipped.
max_inactive — if non-zero, inactive parts of intervals with duration in seconds less than specified can be merged in one combined interval.
min_duration — if non-zero, specifies minimum combined interval duration.
If multiple interval selectors are specified, they will be merged with AND logic, e.g. only ranges where all specified interval selectors are active will be present in the final intervals.
Interval counters
Once the analytics engine determines the final intervals, it will separately calculate counters for each interval.
Each counter has a unique name that will be present as a field in resulting interval JSON.
For now, you can have the following types of counters:
expression — calculate the value based on (usually) message parameters and do certain mathematics with this value.
dataset — for each message generate another JSON object message and return an array of generated points.
route — dump "position.latitude" and "position.longitude" parameters for each message within the interval into Google-encoded polyline algorithm format.
- datetime — print the user-formatted begin or end time according to specified timezone into the interval.
Counters and interval selectors widely use expressions and flespi analytics engine supports quite a lot of specific built-in functions, like mileage() and the possibility to reference parameters not only for the current message but also for the previous message of the analyzed range to operate based on changes.
The result of the analytics engine is a generated set of interval messages with begin and end time of the interval, duration, and custom counters appended to it. Like this:
{
"begin": 1522732482,
"end": 1522792804,
"duration": 60322,
"engine_hours": 0
},
{
"begin": 1522792804,
"end": 1522879226,
"duration": 86422,
"engine_hours": 6123
}A dream come true
Manual intervals calculation is just the beginning. Our ultimate goal is to perform analytics automatically in the background using MQTT to control state changes — e.g. new intervals detected, updated, deleted — and provide our users with the possibility to attach custom parameters to intervals.
Imagine, you have an engine processing all updates, even for unordered blackbox messages, doing basic math in the background, and immediately delivering the results of calculations.
Then you can do custom processing of the generated intervals using own script, refer to the local database, and attach own events and parameters to the intervals.
This is a developer’s dream — to concentrate only on actual business logic and delegate all complex message processing routines.
And this is exactly what flespi analytics will take care of starting 2019Q2 (UPD 30-05-2019: automatic analytics is already available).
Stay tuned to be among the first to test-drive the new functionality and take the burden off your shoulders.
***
For more information about Analytics, I recommend reading topical articles in our KB starting with the one describing the whole concept in detail. Or if you prefer learning by doing, check the working samples on how to split device messages onto trips and how to calculate daily engine hours.
Disclaimer: Analytics engine is highly experimental at the moment. If you intend to use it in a production environment (as some of our users already do), please let us know so that we make sure to notify you personally about any changes.