Present-day IT solutions are the combinations of simpler ideas and concepts that evolve and grow pushed by the environment they exist in and other solutions they compete with. Just like species in nature or memes in society. The difference here is that this evolution is led by the intelligence and vision of a developer. Sometimes, one iteration is enough to create a useful solution, but this only means that the evolution had happened in the developer’s mind. However, more commonly the development process is iterative like evolution — the solution adapts based on new demands and constraints. And it might be pretty exciting to look in the rear-view mirror at the entire mental process from the beginning.
Phase 1: new lifeform
The new lifeform emerged in a small and restricted environment — we needed a dedicated place for a device to store its unique identifier and then to append it to messages for the stream to the Bulgarian Toll-road system. Adding such an option to the device configuration looked excessive for such a low-demand task. Creating a special setting with an appropriate field was architecturally ugly. So we decided to create a new instance to store this extra field to append to the TollBG stream. Luckily, we had the foresight to give a newborn a very ambitious name — “plugin”.
As often happens with evolutionary inventions (we have a thumb to hold a stick, why don't we use it to show that we like something — “thumbs up”), soon it became clear that plugins can be used to store any custom data. However, it was not very easily accessible — plugin fields did not return in response to the GET /gw/devices request so one had to perform a second REST API call GET /gw/plugin/device to obtain the custom fields.
Since flespi is a middleware, everyone consumes messages rather than static device fields, so we decided to append plugin fields to the parameters sent by the device. But some prefer to get this data from the database using GET /gw/device/messages request, some use the device telemetry, and others use MQTT API, so we let our users decide where the field should be stored and added three toggles in the plugin configuration to reflect the possible options.
Another inconvenience was that plugin parameters were prefixed with “plugin.alias.” substring — in this way we wanted to make it easy to distinguish the “real” parameter names parsed by our parsing engine from the “artificial” names added by the plugins.
Phase 2: toddler
In this state we faced another reasonable demand: non-static plugin values. Indeed, the following tasks look pretty natural and basic:
append parameter plugin.overspeeding.status=true if position.speed>60
convert position.speed to plugin.speed.miles dividing it by 1.609344
extract and decipher a certain bit from a bitmask parameter using simple binary operations.
We may conclude that the first version of plugins was unsuccessful in terms of evolution with many unmet expectations from our customers, thus it had to evolve to spread more widely. Ruthless sexual selection pressure. We needed a mutation that allowed the plugin to calculate values based on message parameters. There is such a beast in our zoo — a calculator. It may calculate interval selectors and counters using our expression engine. So we mated the calculator with our plugin and it started adding dynamically calculated values.
Our little convoluted plugin for toll roads could live in this natural equilibrium for millions of years, but the environment rapidly changed:
we’ve allowed our users to POST /gw/device/messages with any parameters (non-necessarily prefixed with some cumbersome strings),
our PVM engine became too powerful to remain the internal protocol parsing technology
we regularly face new projects where GPS trackers are used as connectivity gateways for other equipment sending raw binary data as a payload (but using expressions is a painful way to parse binary data)
post-processing channel protocols started looking like a blind branch of evolution and may even become extinct…
Phase 3: adolescent
Clearly, it was the right moment to take evolution under control, turn on genetic engineering and completely remaster the plugin structure to adapt it to current and future needs. That’s what we did: first, we remastered the current schema of plugin configuration and got rid of the “plugin.alias” prefix for static and expression-based fields. Second, we added another type of plugin for custom parsing called “msg-pvm-code” which resulted in a PVM script that runs on each incoming message. So “device sends some strange data” is not a problem anymore.
The introduction of the plugins with PVM code gave our developers the following idea: if we can run a PVM-script for each message, why can’t we run any other script in the same manner? (e.g. Lua that we use in our internal infrastructure tasks)
The only problem here was that requests to external servers may take a while and sometimes even fail. Assuming that message augmentation may be essential, we decided to solve this in a rather sophisticated way: we are used to the fact that you can see the registered message in flespi as soon as it is parsed but what happens when the device sends a packet:
data is received by the channel
data is parsed according to the protocol implementation
message is registered
(optional) acknowledge packet is sent to the device
Obviously, data augmentation with the data from third-party servers should take place between steps 2 and 3 which leads to an interesting conclusion — if something goes wrong and the plugin fails to accomplish a request or process a response, then the message won’t be registered. What’s more, the device won’t receive an ACK just like when the connection is dropped or the server is unavailable. For most device models it leads to saving the undelivered message to the black box (internal storage) and trying to deliver it again. This makes the final representation of the data more accurate and aligned with the business processes.
With this approach, we’ve easily released plugins to address the two most demanded features: LBS and reverse geocoding.
One more thing to be mentioned here is that plugins can run in sequence one by one with the customer-assigned priorities. E.g. you receive coordinates and speed from the device. The first plugin resolves the coordinates to the street address with the respective speed limit on this street. The second plugin compares the actual speed of the device with the speed limit and registers an alarm parameter that indicates an overspeeding event.
Phase 4: maturity
Plugins haven’t reached this phase yet. They are now experiencing rapid and sometimes chaotic growth and development so natural for adolescence.
***
What’s next? It depends on the demand from our customers. Do you have a project with Wi-Fi geolocation API? Or maybe you need to inject specific data from the logistics solution into flespi messages? Maybe you need to split messages from the devices based on the driver's data that can be obtained from your CRM over the API? Plugins make it all possible. So the telemetry message becomes not mere parsed data from the tracker but a real elementary business block to build all the logic on top of.
Evolution happens as new challenges arise. Challenge us with the new tasks and projects, and we will cultivate new or advance the existing species to address them.
With projects, suggestions, and questions contact us in Helpbox.