- Webhooks intro
- MQTT intro
- Broker as a storage
- Webhooks mechanics
- Webhooks usage example
- Webhooks to third-party services
- Webooks pricing
Webhooks in flespi
Recently, we delivered a brand-new feature to the flespi platform — webhooks. Webhooks will serve as a foundational element of the Automation module and a building block for more advanced components. In this article, I want to give a short overview of their functionality and explain how to make use of them.
Webhooks are used to handle events generated in the MQTT bus of your account and execute HTTP calls to some endpoint per each received and validated MQTT message.
In flespi all user-initiated and system actions generate specific MQTT events in corresponding topics. You can find a list of the most important MQTT topics in MQTT Topics Directory section of the flespi panel or in our KB.
For a more in-depth documentation on webhooks please refer to the dedicated KB article.
For those who are unaware of MQTT, I suggest reading all MQTT Basics articles in MQTT Essentials by HiveMQ to get the idea behind MQTT. However, I can explain it in short as well to save you time ;) (if you are familiar with the MQTT concept, you can skip this section)
MQTT intro
Think of MQTT as a special bus where everybody connected to it can publish messages and subscribe to topics to receive messages published by others. In flespi messages can be published by the platform, your MQTT sessions, devices, channels, streams, plugins, webhooks themselves, calculators, and so on. Even meta-information about REST API calls that you execute on flespi is published into specialized topics.
The MQTT message consists of:
Topic — this is a hierarchical set of words split by slashes that usually identifies the reason for the event together with the source. Topic serves as a key identifier of the message.
Payload — meaningful message data. In flespi it is either UTF-8 text with JSON object or simple type like boolean, number, text, etc. Payload can be null-ed — this will also be a valid MQTT message.
User properties — additional textual tags added to the message. In flespi we add a few standard user properties to all messages:
“timestamp” — UNIX timestamp with microseconds fractional part when this message was posted to the flespi MQTT Broker by the publisher;
“cid” — user account ID to which this message was originally posted;
“token_id” — for user-generated messages contains the ID of the token user authorized at MQTT Broker with;
Sample MQTT messages that you can expect to receive in flespi are:
Device with id #100 connection event:
Topic: flespi/state/gw/devices/100/connected
Payload: true or false
New message received from device with id #200:
Topic: flespi/message/gw/devices/200
Payload: message JSON string
Broker as a storage
MQTT Broker is not only an event bus, but also a storage system. When you publish an MQTT message, you can specify a special retain attribute. If set to “true”, the broker will store this message as retained for a given topic. When a subscriber subscribes to the topic, MQTT Broker will immediately return the stored message. To clear the stored message for the topic, publish an MQTT message with empty payload and retain=true attribute.
In flespi we publish all flespi/state/… topics as retained, so if you subscribe to them, you will immediately receive the latest known message for each topic. Topics flespi/log/…, flespi/message/…, and flespi/interval/… are for real-time events only (no retain available).
When subscribing, you can use wildcards and multiwords:
“+” wildcard means accept any text in this word (part of the topic between two consecutive slashes). For example, to receive connection/disconnection events from all devices you may use this wildcard instead of device id word: flespi/state/gw/devices/+/connected. To receive JSON messages per each device, channel, stream, plugin, calculator in your account, you can use the next subscription topic: flespi/state/gw/+/+.
“#” wildcard means accept any text from here and further. This should be the last part of the subscription topic. To receive all messages related to device with id #123, use the following topic: flespi/+/gw/devices/123/#. You will receive messages with following topics in that subscription:
flespi/state/gw/devices/123
flespi/state/gw/devices/123/connected
flespi/state/gw/devices/123/telemetry/position
flespi/log/gw/devices/123/settings/report_interval/set
flespi/message/gw/devices/123
…
With comma-separated multiwords (flespi specific feature) you can specify a selection of words to match. For example, to receive connection state updates from devices with id #123, #456, #789 just use the following topic: flespi/state/gw/devices/123,456,789/connected
And so on. Basically, everything you can see in the flespi log window for any kind of object is available as an MQTT message published to the corresponding topic. I suggest you explore topics and payloads by using the integrated MQTT Board utility also available in the left-side menu of your flespi.io account.
Sorry for this off-topic explanation, however understanding of MQTT topics and payloads in flespi is extremely important in order to efficiently use webhooks. If you want more information on MQTT and how we use it, please check this selection of articles we’ve written in the course of the last 5 years and I suggest you start reading from the oldest publications.
Webhooks mechanics
So back to webhooks. As I wrote earlier, the webhook is always triggered by an MQTT message usually published by flespi itself to the corresponding topic. You can define up to 10 topics that will trigger a webhook. Once webhook receives that message, you can put together and perform a REST call either to a custom HTTP server or back to the flespi platform to the same account webhook is located in. To construct the URL of an HTTP call, its BODY and HEADERS you can use all attributes of received MQTT message — its topic and separate words in it, payload (which you may also access as JSON object), and any users properties that were added to the message.
For templating of URL, BODY, and HEADERS we use special syntax — all text inside % and % marks is evaluated as an expression with the following available properties:
timestamp — timestamp of the message, e.g. 1680680536.527043:
cid — account ID the message appeared on, e.g. 123;
token_id — token ID from which this message was published (N/A for MQTT messages generated by flespi itself);
topic — exact text of topic messages was posted to, e.g. flespi/state/gw/devices/123/connected;
topics — array of words, e.g. [flespi, state, gw, devices, 123, connected]. Zero-based index, to access device-id use topics[4] in the expression.
payload — message payload data (text). In the case of a JSON object, you can access sub-fields as json(payload, “device_id”) and so on.
user_properties — object with user properties where key is property name
responses — array with received responses. Access it with json(responses[index], "<path>") inside expression. For example use json(responses[0], "/result/0/name"]) to access the name of first element under result field returned by the first webhook in the chain.
response_codes — array with received response codes
Webhooks provide two ways to validate requests. The first allows validation before the request is sent, and the second is applied after the response is received. Depending on the validation result, the current request, or the next request in the chain, can be skipped or retried after a certain interval.
Webhooks usage example
Let’s configure webhooks for a particular task. Say we need to catch device connection/disconnection events and post them back to flespi as device messages so that they can be streamed to my integration as regular device messages or used in calculators to count device offline/online time.
There are two types of MQTT topics I can use:
flespi/state/gw/devices/XXX/connected topic with true or false payload. Flespi publishes “true” when the device is authorized and “false” when it is disconnected.
Another approach will be to handle device logs events. This is a more difficult trigger, however in that case I can also operate the IP address which device is using which is even more interesting. Here I need two topics: flespi/log/gw/devices/XXX/+/connected and flespi/log/gw/devices/XXX/+/disconnected. With ‘+’ I substitute the IP:port of the device, which is also present in the topic and which I don’t know. With the multi-word feature of flespi I will put two topics into one and use flespi/log/gw/devices/XXX/+/connected,disconnected as a trigger. The payload of both messages will contain detailed information as a JSON object.
Testing topics in MQTT Board
First, I tested the two subscriptions with MQTT Board:
When I subscribed to these two topics, I immediately received a message to the left subscription with flespi/state/gw/devices/686701/connected topic and current state of the device. Note the “retain: +” in the message — it indicates that this message comes from storage, not a real-time event.
After that, I disconnected the device and immediately received a message to both subscriptions. To the left pane with a state change and to the right pane with a corresponding log entry. In a few seconds the device reconnected, which was also reflected in both subscriptions with the respective messages.
Now that we’ve made sure the subscriptions will let us properly detect connect/disconnect events, let’s create separate webhooks for each approach.
Webhooks are available in the main left-side menu of the flespi.io panel:
Configuring first webhook
I will start with a simpler webhook — the one based on device state handling. My task is to generate a REST API call to the device which state has changed and register a message with special parameters containing a timestamp of the event. I will register messages using POST /gw/devices/XXX/messages call with a special parameter “connection.state” used to indicate connection state change.
Here is my webhook configuration:
And here is how it worked, in device messages:
Notice how device log events and new messages in its storage correlate between each other. Also note that our artificial messages do not contain “server.timestamp” but contain “rest.timestamp” properties, which is a special indication that the message is registered not from a regular channel but from a REST call. It is also colored blue in Toolbox.
Configuring another webhook
Now let’s handle the same event relying on device logs. Below is the webhook configuration and as you can notice I used “connection.log” parameter, added source address as “connection.source”, and even stored the number of bytes passed through connection for connection close event. Thus, I will be able to calculate traffic consumption for the device.
Now these two webhooks generate two separate messages, because the timestamp for the event is slightly different in its microseconds part:
As you see, webhook processing based on logs is more powerful, but at the same time more complex. You can split the logic on multiple webhooks for connect/disconnect events to handle them differently if needed.
Troubleshooting webhooks
In case you configured something wrong, webhook will report it into its logs. Like in the following example where I didn’t quote the source address as a string and flespi REST API call validation system rejected it:
Webhooks to third-party services
With webhooks you can generate HTTP calls not only to flespi but also to any third-party HTTP servers. If a cloud service has a REST API endpoint, you will be able to use it with flespi. Thus, you may generate SMS, emails, and perform any kind of automation outside flespi.
Just be aware that with webhooks you are limited in throughput as our system will run REST calls one by one. You can expect maximum webhook throughput of around 5–10 messages per second. The closer your endpoint is located, the quicker it will be. Flespi will repeat and buffer up to 10 GB of messages in case your endpoint is unavailable at the moment or reply with 500 HTTP code. For any other HTTP response code except 20X, message delivery will be marked as failed and flespi will attempt the next request.
Webhooks pricing
Webhooks are free for use, but we limit the maximum quantity of webhooks available for users depending on the pricing plan.
***
Webhooks are released in a simple and effective format. Whenever you miss any feature in telematics data processing within flespi, you may fulfill it yourself using webhooks. We plan to further expand flespi with more features and make webhooks more powerful. The vector of development will be based on demand. Once we see that some automation or feature is regularly requested, we will consider how to fit it to flespi.
For now, just play with webhooks to get the sense of what they are capable of and where they can be applied. If you realize want to accomplish something that is not currently possible, please let us know. Your feedback is greatly appreciated!