19 July, 2018

How to access messages in flespi: HTTP, MQTT, and more

A thorough explanation of the three methods to access messages in flespi — REST API, streams, and MQTT.

Today I would like to make a brief overview of a message lifecycle in the flespi ecosystem and which methods to access a message flespi provides. The basic idea of flespi is to understand a message sent by a telematics device in its proprietary protocol (usually inside TCP stream) and deliver it to a user — an application — in a simple JSON object format.

The idea seems pretty straightforward but the devil is in the details. When we opened the flespi platform to the public back in spring 2017 we had only one method to access messages — using a REST API call to https://flespi.io/gw/channels/ID/messages. But since then we introduced many new possibilities to retrieve messages such as devices, their telemetry, push-based streams, and our MQTT API.

Three ways to access messages

Let’s now look at what happens to a message inside flespi and how to access it. On the diagram below you can see various access methods and stages they work on:

ways to access messages in flespi

Basically, there are three different ways to access a message in flespi:

  • Pull-based HTTP REST — the oldest and simplest, works from any system, even from the command line. The only thing you need to specify in headers is the access token. Please refer to the REST API description for a detailed overview of request/response structures.

  • Push-based from a stream. There are plenty of streams that forward messages from the flespi platform to the 3rd party applications. So far the most popular is the HTTP stream that generates an HTTP request to an arbitrary URL with custom headers and messages data payload in JSON.

  • Push-based via an MQTT session that subscribes to certain events. This is the newest and most advanced option. An application creates one or multiple sessions that connect to flespi and subscribe to various events. When a session is offline, the broker will buffer its messages up to a certain limit and resend them to the session once it’s back online.

All these methods, except streams, can be used seamlessly not only from a backend application but also from a web browser or a smartphone app. 

Going through the flespi channel

So let’s take a closer look at what happens to a message once a device sends it to [usually] a TCP connection and flespi channel parses it to a normalized JSON. A regular message consists of at least “timestamp” and “ident” parameters that a device sends, and “channel.id” and “server.timestamp” parameters automatically set by the channel receiving the message:

  "ident": "123456789012345",
  "timestamp": 1490347944.893743,
  "din": 9,
  "channel.id": 123,
  "position.altitude": 568.49,
  "position.direction": 297,
  "position.hdop": 0.9,
  "position.latitude": -21.328481,
  "position.longitude": 47.562136,

An “ident” parameter represents a unique device identifier — usually an IMEI, and a “timestamp” parameter represents a UNIX timestamp of the moment the message was generated in UTC with optional milliseconds in the fraction part. If the device does not send a timestamp, flespi will use the time when the message was received by the server.

Important notice regarding a “messages_ttl” channel parameter. If set to zero (seconds), a channel will not store messages in the buffer and thus the REST API method GET /gw/channels/XXX/messages will always return an empty result. The default value for “messages_ttl” is 10 days and for free flespi accounts we limit the maximum channel buffer size to 100 MB. If in 10 days you have too many messages sitting in the channel buffer that you have not consumed with the above REST API call, your channel can be periodically blocked. To avoid blocking, especially if you are using other access methods, just set “messages_ttl” for a channel to zero.

Streams and MQTT API access methods do not depend on channel messages buffer and will process each message as it comes.

On a channel level, an application will receive all messages from all devices as a whole and it is the logic of the consuming application that will use the “ident” value in each message to match a given message to the correct device. The only exception here is MQTT API as it allows subscribing to messages from a specific ident on a channel level.

Optional processing in the flespi device

Next, the flespi platform will look up in the user database for a device with the corresponding ident and device type. If it can detect the correct device, the message is assigned to the device and the device level processing of the message starts.

At this stage the flespi platform does the following operations:

  • Adds “device.id”, “device.name” and “device.type.id” parameters to the original message.

  • If the device “messages_ttl” parameter is not zero, the copy of the message is stored in device message storage accessible anytime via GET /gw/devices/XX/messages REST API.

  • Publishes the message into MQTT under “flespi/message/gw/devices/XX, where XX is device ID.

  • Updates device telemetry — each individual parameter is published as a retained MQTT message with the corresponding timestamp into the topic “flespi/state/gw/devices/XX/telemetry/YY”, where XX is a device ID and YY is a parameter name, e.g. “position.latitude”. Also, in the telemetry, the platform generates an artificial “position” parameter with JSON object payload containing the combined position information.

  • And delivers the message to streams subscribed to the given device.

So using REST API to access messages archive for a device it will be possible to access any historical message sent by a device within specified “messages_ttl”. But to receive a push-event from the platform once it sees a new message from the device, you need to use either streams or MQTT API. I personally recommend MQTT API as a mechanism that is more convenient, does not require a public IP address, and is much more configurable.

By subscribing to “flespi/message/gw/devices/#” you will be able to receive all messages from all devices registered in the flespi platform, while “flespi/message/gw/channels/#” will deliver all messages received by the channels even if the device is not registered in the flespi platform.

Another nice opportunity not only to receive updates but also to receive all current information from all devices is to subscribe to “flespi/state/gw/devices/+/telemetry/+”. This subscription will immediately deliver the latest values of all parameters from all devices and once these parameters change, you will be push-notified about the update in the same MQTT connection. This is very convenient. For example, to display all devices on the map you can subscribe to “flespi/state/gw/devices/+/telemetry/position” and receive only the position of devices that will be updated in real-time when devices move. We have the open-source TelemetryViewer project that you may use to show telemetry parameters from selected devices.

Plugging the logic

This is not the final picture. We have also introduced a message processing pipeline — flespi analytics engineNow you can aggregate or process incoming device messages by applying math and logical operations to get insights, detect events, trigger notifications, generate reports, and more. E.g. detect and tag if the device is in a geofence, split a track into trips, group trips by time, cut-off coordinates as we do in this GDPR-related python script, etc. Some algorithms will be built-in but some of them can be implemented by flespi users and on their servers. The only acceptable way to communicate with external algorithms is also via MQTT :) 


Each month we will add more and more MQTT-based things. We are promoting the MQTT usage among developers worldwide with our free-to-use MQTT 5.0 broker and contribution of the MQTT 5.0 protocol support into the popular mqtt.js library

So if you are picking a method to access flespi, let it be MQTT — there are plenty of clients for most programming languages.