Today I would like to make a brief overview of a message lifecycle in the flespi ecosystem and which ways to access messages flespi provides. The basic idea behind flespi is to understand a message sent by a telematics device in its proprietary protocol (usually inside a TCP connection) and deliver it to a customer — an application — in a simple JSON object format.
The idea seems pretty straightforward but the devil is in the details. When we released flespi to the public back in spring 2017 we had only one method to access messages — using a REST API call to https://flespi.io/gw/channels/ID/messages. But since then we introduced many new possibilities to retrieve messages such as devices, their telemetry, push-based streams, and our MQTT API.
Three ways to access messages
Let’s now look at what happens to a message inside flespi and how to access it. On the diagram below you can see various access methods and stages they work on:
Basically, there are three different ways to access a message in flespi:
Pull-based HTTP REST — the oldest and simplest, works from any system, even from the command line. The only thing you need to specify in headers is the access token. Please refer to the REST API description for a detailed overview of the request/response structures.
Push-based stream. Flespi offers a variety of streams that can forward messages to the third-party platforms. So far the most popular is the HTTP stream that generates an HTTP request to an arbitrary URL with custom headers and messages data payload in JSON.
Push-based via an MQTT session that subscribes to certain events. This is the newest and most advanced option. An application creates one or multiple sessions that connect to flespi and subscribe to various events. When a session is offline, the broker will buffer its messages up to a certain limit and resend them to the session once it’s back online.
All these methods, except streams, can be used seamlessly not only from a backend application but also from a web browser or a smartphone app.
Going through the flespi channel
So let’s take a closer look at what happens to a message once a device sends it to [usually] a TCP connection and flespi channel parses it to a normalized JSON. A regular message consists of at least “timestamp” and “ident” parameters that a device sends, and “channel.id” and “server.timestamp” parameters that are automatically added by the channel receiving the message:
{
"ident": "123456789012345",
"timestamp": 1490347944.893743,
"din": 9,
"channel.id": 123,
"position.altitude": 568.49,
"position.direction": 297,
"position.hdop": 0.9,
"position.latitude": -21.328481,
"position.longitude": 47.562136,
...
}An “ident” parameter represents a unique device identifier — usually an IMEI, and a “timestamp” parameter represents a UNIX timestamp of the moment the message was generated in UTC with optional milliseconds in the fraction part. If the device does not send a timestamp, flespi will use the time when the message was received by the server.
Important notice regarding a “messages_ttl” channel parameter. If set to zero (seconds), a channel will not store messages in the buffer and thus the REST API method GET /gw/channels/XXX/messages will always return an empty result. The default value for “messages_ttl” is 10 days and for free flespi accounts we limit the maximum channel buffer size to 100 MB. If in 10 days you have too many messages sitting in the channel buffer that you have not consumed with the above REST API call, your channel can be periodically blocked. To avoid blocking, especially if you are using other access methods, just set “messages_ttl” for a channel to zero.
Streams and MQTT API access methods do not depend on channel messages buffer and will process each message as it comes.
On a channel level, an application will receive all messages from all devices in bulk and it is the consuming application that should use the “ident” value in each message to match a it to the proper device. The only exception here is MQTT API as it allows subscribing to messages from a specific ident on a channel level.
Optional processing in the flespi device
Next, the flespi platform will look up in the user database for a device with the corresponding ident and device type. If it can detect the correct device, the message is assigned to this device and the device level processing of the message starts.
At this stage the flespi platform performs the following operations:
Adds “device.id”, “device.name”, and “device.type.id” parameters to the original message.
If the device “messages_ttl” parameter is not zero, the copy of the message is stored in device message storage accessible anytime via GET /gw/devices/XX/messages REST API.
Publishes the message into MQTT under “flespi/message/gw/devices/XX”, where XX is device ID.
Updates device telemetry — each individual parameter is published as a retained MQTT message with the corresponding timestamp into the topic “flespi/state/gw/devices/XX/telemetry/YY”, where XX is a device ID and YY is a parameter name (e.g. “position.latitude”). Also, in the telemetry, the platform generates an artificial “position” parameter with JSON object payload containing the combined position information.
Delivers the message to streams subscribed to the given device.
Practical use
REST API allows accessing any historical message sent by a device within the specified “messages_ttl”.
To receive a push-event from the platform once it sees a new message from the device (in real-time), you need to use either streams or MQTT API. We recommend MQTT API as a mechanism that is more convenient, does not require a public IP address, and is much more configurable.
By subscribing to “flespi/message/gw/devices/#” you will be able to receive all messages from all devices registered in the flespi platform, while “flespi/message/gw/channels/#” will deliver all messages received by the channels even if devices are not registered in the flespi platform.
Another nice opportunity not only to receive updates but also to receive all current information from all devices is to subscribe to “flespi/state/gw/devices/+/telemetry/+”. This subscription will immediately deliver the latest values of all parameters from all devices and once these parameters change, you will be push-notified about the update in the same MQTT connection.
To display the current position of all devices on the map you can subscribe to “flespi/state/gw/devices/+/telemetry/position” and receive only the position data that will be updated in real-time as devices move. We have the open-source TelemetryViewer project that you may use to show telemetry parameters from selected devices.
Hint: playing around with the MQTT topics is convenient using the MQTT Board tool.
Further steps: data enrichment
Once the incoming data is saved as device messages, you can apply versatile data transformations via plugins. You can add or remove message parameters, add reverse geocoding data from Google or OSM, add LBS data, and even script a custom parsing scheme. You can explore more capabilities of plugins in the knowledge base.
Further steps: data analytics
Flespi also features a message processing pipeline — the analytics engine. Now you can aggregate or process incoming device messages by applying math and logical operations to get insights, detect events, trigger notifications, generate reports, and more. E.g. detect and tag if the device is in a geofence, split a track into trips, group trips by time, etc. Some algorithms will be built-in but some of them can be implemented by customers and on their servers. The only acceptable way to communicate with external algorithms is also via MQTT :)
***
Each month we will add more and more MQTT-based things. We are promoting the MQTT usage among developers worldwide with our free-to-use MQTT 5.0 broker and contribution of the MQTT 5.0 protocol support into the popular mqtt.js library.
So if you are picking a method to access flespi, go for MQTT — there are plenty of clients for most programming languages.