After the presentation of flespi database system we’ve seen a lot of interest in such a component among third-party developers. Indeed, why reinvent the wheel when subject matter experts have already created a solution tailored for telematics applications?
flespi platform is a highly flexible IoT telematics backend, and making its internal components public is a straightforward process comprising the following steps:
We've used our best efforts and have gone through all of the steps above to design and release the storage component API in less than 2 weeks! This is by far the fastest IoT cloud database development time anyone has heard of :)
In essence, our storage system provides a set of REST API methods to store IoT and telematics data inside the backend. As professionals in the field we pinpoint the following key differences of telematics time series data from "normal" data:
flespi storage system includes REST API methods designed to work with different types of elements.
The simplest element is a container. A container represents a simple key:value database where a key is a 32-bit unsigned integer and a value is a custom JSON object. According to our API guidelines you can post one or several messages with one HTTP request.
To extract messages from a container you can specify a key range (to search by key) or a value range (to search in a JSON object). For example, a container stores the floating car data where a key is a timestamp and a value is the current vehicle state and location. We can immediately fetch all the messages when the driver was not wearing the seat belt.
A container can store up to 8MB of any categorized data in a single message. Flags attribute defines the strategy for dealing with duplicates, i.e. what the database engine needs to do when we receive a message with a key already existing in the database. At the moment we have the following strategies implemented:
In telematics and IoT applications/systems like Wialon, PositionLogic or GPSGate regular raw messages from devices use containers of 0x0, 0x1, 0x2 or 0x3 types depending on how the data need to be handled. Type 0x2 container can be used to store and update the current state of vehicles. Type 0x3 container can be used to ensure that no messages are overwritten or lost (this is how we store channel logs in gateway module).
The second type of storage elements is an abque. "Abque" is short for "asynchronous batch queue" which is a queue-type storage with the associated REST API methods for posting messages into a queue, reading and checkpointing them. A message is a JSON object with any set of custom fields of up to 8MB in binary size.
Abque is designed for multiple-writers/single-reader mode with one second granularity. Abque can receive as many messages simultaneously as you need and place them into the queue within one second. This means that if request times for the messages differ by more than one second, they will be posted into the queue in the right order. Within a 1-second interval the right order of messages is not guaranteed.
The special way of reading messages from the queue is described in the gateway quick-start. It is always a single-reader mode that goes through the following logic:
Abque is used in telematics platforms to process messages received by gateway channels. It can also serve as an interconnection mechanism to reliably dispatch messages, commands, and events from one application to another. An open-source analogue of an abque is RabbitMQ, but its performance at high message rates is incomparably slower.
Our further plans to enhance flespi storage system include:
The release of flespi storage module marks a midpoint on the road to Registry module scheduled for Q3 2017. Registry module will empower flespi platform users to operate the gateway channels not as a single entity, but with dedicated access to the device messages and the possibility to control the devices directly over the air. This enhanced toolset will include everything that a business application needs to process and store the data from the IoT and vehicle telematics devices with ease and efficiency. More updates will follow!
Evolution of communication technologies used between flespi services and the introduction of own MQTT implementation to the public.