We in flespi love experimenting and trying on the new technologies. With the appearance of MQTT 5.0, we saw a number of benefits that it could bring to our platform infrastructure and started gradually transitioning to using this technology in our internal processes.
To be in control of all the ups and downs and to adhere to our in-house development principle, we created own MQTT broker with full MQTT 5.0 support and accidentally became the first to do so.
As we continued to build our ecosystem around MQTT, we came across some issues that did not satisfy us 100%.
However, we do not limit ourselves with the existing standards. We seek the best solution. And if we see room for enhancement from which our and potentials someone else’s solution may benefit, we go and enhance. This is exactly what we did with MQTT 5.0 — we introduced some extras while maintaining full compatibility with the original.
In this post, we will explain some features that you will not find in MQTT 5.0 official documentation but that may fit well into your projects or be an easy fix for some of your burning problems.
Sticky shared subscriptions
Shared subscriptions are not new for seasoned MQTT users. The idea behind the concept is that a group of clients (subscribers) share the same subscription and receive messages in an alternating fashion thus ensuring a so-called client load-balancing.
However, in some cases, it is preferable to direct specific topics to the same subscriber within the shared group unlike in the traditional use for load balancing. This approach allows for implementing more efficient cache between shared subscribers.
With sticky shared subscriptions, you can specify the portion of the topic to use for calculation of the unique topic fingerprint that will be compared upon the attribution of each incoming message.
Specify the initial word position and words count to calculate the topic fingerprint, e.g. '$share/group_name:from:count/...' (a simpler form is '$share/group_name:count/...' for zero-based 'from' topics). The broker will calculate the topic fingerprint and assign it and future topics with the same fingerprint to the same subscriber.
Shared group sessions storage
Imagine you have a shared group but for some reason, all subscribers are temporarily offline. How to ensure that the new message that arrived at the broker will eventually get delivered? This is exactly what shared group session storage is destined to do.
If the message arrives at the shared group when no sessions are online, the message will sit in special storage until any such sessions connect again.
As shared groups are mostly used for load balancing, this feature prevents any data loss when all such sessions are offline for whatever reason.
Wildcard deletion of retained messages
It may be necessary to delete a batch of retained messages with topics matching a specific topic mask. The MQTT 5.0 standard allows only one-by-one deletion, which may be time-consuming.
With flespi MQTT broker, if you publish a message with an empty payload to a topic that contains wildcards ("#" or "+"), flespi will perform bulk deletion of all retained messages that match the published topic mask in one go.
User properties
User properties is one of the features that gives MQTT 5.0 great flexibility and extendability. We introduced a few user properties into the flespi MQTT broker to adapt it to some of the telematics tasks we solve.
token_id. Interaction with the flespi broker is only possible when a publisher or subscriber authorizes using the flespi token. Therefore, having a token_id inside each message published to the broker allows identifying the sender, which may be important in some cases (e.g. when you want to group messages received from the same sender).
timestamp. This property is used to sort which message came last and will be saved in retained storage. If not set by the publisher, the flespi broker will attach this property to the published message with the current UNIX time value. In case the broker receives several messages with the same timestamp from one connection, a special "seqno" user property will be attached to messages to distinguish the most actual one. This parameter is particularly helpful for telematics projects where message ordering is important.
modified_since. The property allows filtering retained messages that are fresher than the specified time value. If a valid value is specified, the subscriber will receive only the retained messages newer than the given UNIX timestamp.
cid. If using a flespi hierarchical accounts structure, you can get the ID of the flespi account that originally generated this message. This property makes it possible to subscribe only to messages available to specified flespi account ID. This feature is useful only for Commercial Plan users which split MQTT operations into multiple subaccounts.
CONNACK. This is an auxiliary property carrying the session connection acknowledgment packet that contains additional information about the session including ACL settings for the session authorization token and its current subscriptions.
Bonus: Subscription id
Unlike the previous ones, this feature is supported in the MQTT 5.0 specification but recently we personally realized its benefits and wanted to share our experience.
One of our backend services needed to do numerous subscriptions for thousands of subaccounts. Each subscription is a one-time action, however, huge message flow is attributed to each subscription. And the resource-demanding task is to find the subscriber in the code. Subscription ID allows placing the subscribers in an array or similar easily searchable data structure, and for each incoming message find the subscriber by id in a fast and consistent manner. Without subscription id, this process required a complex analysis of the topic string to identify the recipient.
With hundreds of thousands of items requested the bandwidth savings are substantial. So is the performance boost.
***
flespi is gradually transitioning all its internal communication into MQTT and there are very few processes left unaffected. We are pushing MQTT 5.0 to the limit and then push its limits. We widely use the extra features we introduce, because they are created to respond to our specific needs.
Just bookmark this article to refer to it the next time you decide to give MQTT a try. And in case pure MQTT 5.0 is not enough.
Also, we’d be happy to answer your technical inquiries regarding the use of MQTT in general and the features we added on top.