Today I would like to make a brief overview of a message lifecycle in flespi ecosystem and which methods to access a message flespi provides. The basic idea of flespi is to understand a message sent by a telematics device in its proprietary protocol (usually inside TCP stream) and deliver it to a user — an application — in a simple JSON object format.
The idea seems pretty straightforward, but the devil is in the details. When we opened the flespi platform to the public back in spring 2017, we had only one method to access messages — using REST API call to https://flespi.io/gw/channels/ID/messages which is briefly described in the flespi quick-start. But since then we introduced many new possibilities to retrieve messages such as devices, their telemetry, push-based streams, and of course our MQTT API.
Three ways to access messages
Let’s now look what happens to a message inside flespi and how to access it. On the diagram below you can see various access methods and stages they work on:
Basically, there are three different ways to access a message in flespi:
Pull-based HTTP REST — the oldest and simplest, works from any system, even from the command line. The only thing you need to specify in headers is access token. Please refer to the REST API description for the detailed overview of request/response structures.
Push-based from a stream. There are plenty of streams that resend messages from flespi platform to 3rd party applications. So far the most popular is the HTTP stream that generates an HTTP request to an arbitrary URL with custom headers and messages data payload in JSON.
Push-based via MQTT session that subscribes to certain events. This is the newest and the most advanced option. An application creates one or multiple sessions which connect to flespi and subscribe itself to various events. When a session is offline, the broker will buffer its messages up to a certain limit and resend them to the session once it’s back on.
All these methods except streams can be used seamlessly not only from backend application but from a web-browser or a smartphone app.
Going through the flespi channel
So let’s take a closer look at what happens to a message once a device sends it to [usually] a TCP connection and flespi channel parses it to a normalized JSON. A regular message consists of at least “timestamp” and “ident” parameters that a device sends, and “channel.id” parameter automatically set by the channel receiving the message:
An “ident” parameter represents a unique device identifier — usually an IMEI, and a “timestamp” parameter represents a UNIX timestamp of the moment the message was generated in UTC with optional milliseconds in the fraction part. If the device does not send a timestamp, flespi will use the time when the message was received by the server.
Important notice regarding a “messages_ttl” channel parameter. If set to zero (seconds), a channel will not store messages in the buffer and thus the REST API method GET /gw/channels/XXX/messages will always return an empty result. The default value for “messages_ttl” is 10 days and for free flespi accounts we limit the maximum channel buffer size to 100 MB. If in 10 days you have too many messages sitting in the channel buffer that you have not consumed with the above REST API call, your channel can be periodically blocked. To avoid blocking, especially if you are using other access methods, just set “messages_ttl” for a channel to zero.
Streams and MQTT API access methods do not depend on channel messages buffer and will process each message accordingly.
On a channel level, an application will receive all messages from all devices as a whole and it is the logic of the consuming application that will use the “ident” value in each message to match a given message to the correct device. The only exception here is MQTT API as it allows subscribing to messages from a specific ident on a channel level.
Optional processing in the flespi device
Next, the flespi platform will look up in the user database for a device with the corresponding ident and device type. If it can detect the correct device, the message will be assigned to the device and the device level processing of the message starts.
At this stage the flespi platform does the following operations:
Adds “device.id”, “device.name” and “device.type.id” parameters to the original message.
If the device “messages_ttl” parameter is not zero, the copy of the message is stored in devices messages storage accessible anytime via GET /gw/devices/XX/messages REST API.
Publishes the message into MQTT under “flespi/message/gw/devices/XX”, where XX is device ID.
Updates device telemetry — each individual parameter is published as a retained MQTT message with the corresponding timestamp into the topic “flespi/state/gw/devices/XX/telemetry/YY”, where XX is a device ID and YY is a parameter name, e.g. “position.latitude”. Also, in the telemetry, the platform generates an artificial “position” parameter with JSON object payload containing the combined position information.
And delivers the message to streams subscribed to the given device.
So using REST API to access messages archive for a device it will be possible to access any historical message sent by a device within specified “messages_ttl”. But to receive a push-event from the platform once it sees the message for the device, you need to use either streams or MQTT API. I personally recommend MQTT API as a mechanism that is more convenient, does not require a public IP address, and is much more configurable.
By subscribing to “flespi/message/gw/devices/#” you will be able to receive all messages from all devices registered in the flespi platform, while “flespi/message/gw/channels/#” will deliver all messages received by the channels even if the device is not registered in the flespi platform.
Another nice opportunity not only to receive updates but also to receive all current information from all devices is to subscribe to “flespi/state/gw/devices/+/telemetry/+”. This subscription will immediately deliver the latest values of all parameters from all devices and once these parameters change, you will be push-notified in the same MQTT connection. This is very convenient. For example, to display all devices on the map you can subscribe to “flespi/state/gw/devices/+/telemetry/position” and receive only the position of devices that will be updated in real-time when devices are moving. We have the open-source TelemetryViewer project that you may use to show telemetry parameters from selected devices.
Looking to plug the logic
This is not the final picture. Currently, we are thinking of how to inject this message processing pipeline with special algorithms that will be able to modify the content of a message in flespi, e.g. detect and tag if it is in a geofence, split a track into trips or just cut-off coordinates like we do in this GDPR-related python script. Some algorithms will be built-in but some of them can be implemented by flespi users and on their servers. The only acceptable way to communicate with external algorithms is also via MQTT :)
Each month we will add more and more MQTT-based things. We are promoting the MQTT usage among developers worldwide with our free-to-use MQTT 5.0 broker and contribution of the MQTT 5.0 protocol support into the popular mqtt.js library.
So if you are picking a method to access flespi, let it be MQTT — there are plenty of clients for most programming languages.