Stream — forwarding data to other platforms

Specialized outgoing pipelines to push messages to third-party platforms.

Essentials

  • Streams are used to send telematics data from flespi to cloud hubs, fleet management platforms, and other services.
  • Streams can forward data from specific devices, groups of devices, or entire channels.
  • Only newly received messages are forwarded by the stream and in the same order as they were received by the channel or device.
  • If the destination is unavailable, streams use a buffer to temporarily store the messages to be sent. Once the connection is restored all buffered messages will be flushed. Buffer stores messages for the time period specified in queue_ttl configuration field.
  • Streams operate on a batch mode sending to the target each next portion of accumulated messages only once all messages from previous batch are delivered and acknowledged. The performance of stream target can be tracked with ack_latency parameter in stream log for each such batch delivery report.
  • All streams operations with batch are reported into logs:
    • read - how many messages from subscribed devices and channels are in the batch;
    • skipped - how many messages in the batch are ignored from delivery;
    • accepted - how many messages positively acknowledged by stream target;
    • rejected - how many messages negatively acknowledged by stream target or timeouted;
  • Raw traffic between stream and its target is available for any extended diagnosis.
  • When stream is disabled - it just does not work, it does not append messages to the buffer to be delivered later. Disabling the stream does not clear the messages buffer.

You can filter out messages stream is forwarding by utilizing validate_message field with expression. Only messages that satisfy the expression will be processed. For example to forward only messages with non-zero speed and coordinates present in message use next expression: "position.speed > 0 && exists('position.latitude') && exists('position.longitude')".

By using convert_message field you may replace dots in messages to different format, for example position.speed can be converted into PositionSpeed or position_speed with appropriate option selected. 

Also convert_message option has fields parameter by utilizing which you may leave in the forwarded message only specified fields and remove all other. For example if you specify there ["device.id", "position.*"] messages sent by the stream will contain only device.id and position.latitude, position.longitude and position.speed parameters. This option is able to remove flespi system parameters as well.

How streams work

Streams push messages from flespi to other platforms (see how to create a stream [VIDEO]). A stream will push only newly received messages from channels or devices subscribed to the stream and in the same order as they were received. If the target platform is unavailable, the stream stores unpushed messages in its internal buffer and pushes them as soon as the target platform is up again. flespi supports several types of streams.

Note: if a stream is subscribed to a channel and a device (working over the same channel), the messages in the stream will not be duplicated and the message from the device will be forwarded.

IP whitelisting: If you protect your integration with an IP filter or firewall, please be sure to whitelist 185.213.2.10 and 185.213.2.110 IP addresses (or the whole subnet 185.213.2.0/24 that is operated solely by flespi).

Stream types

Common streams

HTTP, MQTT, Wialon Retranslator are the protocols with the open specification that can be used in your custom solution:

  • HTTP — sends POST requests with a list of messages to the specified URI. Max number of messages in one request can be set in the stream configuration. As soon as the server replies with 200 code, the sent messages are removed from the stream buffer.

  • MQTT — publishes messages to the MQTT connection (credentials specified in stream configuration). The stream uses QoS=1 to ensure that the message is removed from the stream buffer as soon as ACK from the broker is received.

  • AMQP - publishes messages to the AMQP Broker, such as RabbitMQ.

  • Pulsar - publishes messages to the Apache Pulsar messages platform.

  • Wialon Retranslator — sends telemetry collected from several sources (has open specification). This protocol is widely accepted by various telematics platforms.

  • Flespi gateway — sends data between different flespi accounts via HTTPS.

  • Zoho Catalyst — full-stack cloud development platform.

Streams to cloud IoT hubs

flespi allows easily send your data to the major cloud services::

These platforms can accept the special data format implemented in flespi streams. So flespi user simply needs to configure the stream with the appropriate IoT hub credentials.

Streams to fleet management & GPS tracking platforms

Streams to specialized platforms

flespi offers ready-made solutions to push data to such platforms as:

How to create a stream?

  1. Log in to the flespi panel
  2. Navigate to Telematics hub -> Streams and click the "+" button in the bottom right corner to add a new stream:
    create a stream
  3. Give the new stream a name, pick the proper stream type depending on the destination platform, configure it, and click Save:
    configure a flespi stream
  4. For the stream to start sending messages it has to be subscribed to device(s) or channel(s). Go to the respective tab on the stream card and assign desired items to the stream:
    assign device to stream
  5. Once the assigned device starts sending messages, the stream with start forwarding them to the target platform. You can check the Logs & Messages for the stream to see the "stream has sent messages" records:
    check stream logs

Streams API

To perform any operations with the streams, use the streams API.

Troubleshooting

In case of any issues, check the Logs tab on the stream screen:

stream logs and messages

The accepted, read, rejected, and skipped columns show the respective number of packets. The ack_latency parameter shows how long it took to send a batch of messages.

You can also view stream traffic — full data exchange between flespi and the 3rd party platform:

flespi stream traffic

You can quickly navigate to a raw traffic associated with certain log event by right clicking on a log message and selecting "Show Traffic" in the context menu. This action will switch the view to a traffic view and navigate to the corresponding traffic generated by the stream and related to this event in stream logs.


See also
Sending telemetry data from flespi to Pulsar Broker
Sending telemetry data from flespi to RabbitMQ or other AMQP 0-9-1 Broker.