20 June, 2017

flespi storage system: one-of-a-kind telematics database engine

It took us two weeks of intense work to design and release flespi storage component API - the fastest IoT cloud database development seen to date.

Releasing the product

After the presentation of flespi database system we’ve seen a lot of interest in such a component among third-party developers. Indeed, why reinvent the wheel when subject matter experts have already created a solution tailored for telematics applications?

flespi CeBIT 2017 Hannover

flespi platform is a highly flexible IoT telematics backend, and making its internal components public is a straightforward process comprising the following steps:

  • describe flespi storage REST API with a swagger file;
  • add it to the automatic limitation system, so that external users can't overload it;
  • add a set of tests to the automatic test system to ensure maximum uptime of the component and maintain its highest quality;
  • market it (we are doing it now).

We've used our best efforts and have gone through all of the steps above to design and release the storage component API in less than 2 weeks! This is by far the fastest IoT cloud database development time anyone has heard of :)

flespi performance telematics database

Justifying the need for a telematics database

In essence, our storage system provides a set of REST API methods to store IoT and telematics data inside the backend. As professionals in the field we pinpoint the following key differences of telematics time series data from "normal" data:

  • data (messages) consist of a timestamp and sensor values at a given point in time; however, the combination of columns in each individual message can be dissimilar to other messages;
  • volumes are high: thousands of new messages per second and gigabytes of data in total per device;
  • each message matters: in telematics we are dedicated to reliably storing each individual message, in contrast to standard time-series database engines that are OK with dropping a few messages since this won’t affect the statistics;
  • some messages can contain additional media, such as videos, camera pictures or .ddd files from a tachograph.

Key storage elements

flespi storage system includes REST API methods designed to work with different types of elements.

The simplest element is a container. A container represents a simple key:value database where a key is a 32-bit unsigned integer and a value is a custom JSON object. According to our API guidelines you can post one or several messages with one HTTP request.

To extract messages from a container you can specify a key range (to search by key) or a value range (to search in a JSON object). For example, a container stores the floating car data where a key is a timestamp and a value is the current vehicle state and location. We can immediately fetch all the messages when the driver was not wearing the seat belt.

A container can store up to 8MB of any categorized data in a single message. Flags attribute defines the strategy for dealing with duplicates, i.e. what the database engine needs to do when we receive a message with a key already existing in the database. At the moment we have the following strategies implemented:

  • 0x0 - the new message always overwrites the currently stored message;
  • 0x1 - the currently stored message never gets overwritten and the new message is ignored;
  • 0x2 - the messages are merged, values from the new message overwrite the same keys in the currently stored message;
  • 0x3 - always register a new message to store multiple messages for one key.

In telematics and IoT applications/systems like Wialon, PositionLogic or GPSGate regular raw messages from devices use containers of 0x0, 0x1, 0x2 or 0x3 types depending on how the data need to be handled. Type 0x2 container can be used to store and update the current state of vehicles. Type 0x3 container can be used to ensure that no messages are overwritten or lost (this is how we store channel logs in gateway module).

The second type of storage elements is an abque. "Abque" is short for "asynchronous batch queue" which is a queue-type storage with the associated REST API methods for posting messages into a queue, reading and checkpointing them. A message is a JSON object with any set of custom fields of up to 8MB in binary size.

Abque is designed for multiple-writers/single-reader mode with one second granularity. Abque can receive as many messages simultaneously as you need and place them into the queue within one second. This means that if request times for the messages differ by more than one second, they will be posted into the queue in the right order. Within a 1-second interval the right order of messages is not guaranteed.

The special way of reading messages from the queue is described in the gateway quick-start. It is always a single-reader mode that goes through the following logic:

  • when a new request arrives, it confirms the messages already read by the previous request;
  • if the confirmation succeeds, the queue can safely delete those previous messages and free up the storage space for the buffer;
  • until the confirmation is successful, the messages will be available in the buffer for other requests as well;
  • if there are no messages in the buffer, the request can set the timeout in seconds that the platform will wait for another portion before sending a response.

Abque is used in telematics platforms to process messages received by gateway channels. It can also serve as an interconnection mechanism to reliably dispatch messages, commands, and events from one application to another. An open-source analogue of an abque is RabbitMQ, but its performance at high message rates is incomparably slower.

Steps going forward

Our further plans to enhance flespi storage system include:

  • a possibility to attach videos to messages and stream them;
  • specialized API to attach images to messages (alternatively, a separate image container storage type will be implemented);
  • specialized API for the dedicated storage of .ddd files containing data from tachographs;
  • a special media storage system to enable the gateway channel's protocol to operate with these media.

The release of flespi storage module marks a midpoint on the road to Registry module scheduled for Q3 2017. Registry module will empower flespi platform users to operate the gateway channels not as a single entity, but with dedicated access to the device messages and the possibility to control the devices directly over the air. This enhanced toolset will include everything that a business application needs to process and store the data from the IoT and vehicle telematics devices with ease and efficiency. More updates will follow!