Video Telematics features preview

Please check modern variant of video telematics integration guide in our KB.

Almost a year ago, we announced in our roadmap the integration of video telematics functionality into flespi. This year, since July, we have occasionally published the progress of this task on LinkedIn. And now, I’m finally ready to put everything in one place and provide a brief overview of what you should expect from flespi in the video telematics segment.

First of all, I want to note the fact that flespi is a backend, not a full-featured platform. Despite a set of ready-to-use tools, most of which are available in open-source form, it is expected that you build your application on top of flespi or use its specific telematics services. The same applies to video telematics features. We provide you with functionality, but still, you have to perform its integration into your application either by utilizing our specific tools or using REST API and MQTT services directly. Keep this in mind, but do not be afraid. With flespi, you can integrate video telematics into your application and release it into production within a single working day!

By the way, our video telematics system design was strongly influenced by the marshmallow BBQ party this summer; that's why, in the very end, it became so sweet and nice :)

In short, we provide:

Device media storage to fetch or play video clips on the Internet via a simple link.
Durable device commands for video clip retrieval. The file is provided via the link and stored in device media storage.
Live video streaming functionality.
Take picture commands to operate with pictures as well.
Embeddable open-source UI tool to seamlessly integrate video telematics into your application.
Unification of photo/video data access patterns among different hardware manufacturers.

Despite a cool set of features, we decided not to charge anything additional for this functionality, which means that video telematics features are available to all flespi users for free — you only pay for the storage.

Device Media Storage

Device media storage is used to store any additional media data related to the device. Initially, we are implementing it for video and picture storage; however, it will be possible to store any type of information related to the device there—for example, .ddd files from tachographs.

Currently, up to 10GB of storage can be used for device media files. We're considering extending this limit if there is a corresponding demand.

You control the actual volume of device media storage by using the 'media_ttl' and 'media_rotate' properties in the device's configuration:

For example, if you set the media storage rotation size to 1GB, whenever a new file is uploaded and the total media storage size exceeds 1GB, older files will be automatically deleted. We do not recommend setting it below 500 MB, as you will not be able to retrieve large video clips or you will be very limited in video streaming functionality, which also uses device media storage during its operation.

You may protect some files from automatic deletion by TTL or rotation by size by setting the 'protected' attribute for each particular file.

The REST API to access device media files can be found here. The following operations are available for each device:

Search for some files or list all files.
Delete all or specific files.
Add to the file your own set of properties, or enable/disable file protection from automatic deletion.

All files have a unique UUID property, which is used to identify the file, and a unique link by which you may download the file or access it directly from your application. When you access a file with this link, there's no need to use tokens for authorization — you just use it as is. The link is available with GET and HEAD HTTP/1.1 methods, and you may also request a preview and JSON representation of the file. Compare these three links of the same media file:

File as is: https://media.flespi.io/A731FE26BCEEA97200003CD37796680F620000000000000047C9C57BFA56BD08
Small resolution preview image: https://media.flespi.io/A731FE26BCEEA97200003CD37796680F620000000000000047C9C57BFA56BD08?preview=jpeg
Metadata in JSON: https://media.flespi.io/A731FE26BCEEA97200003CD37796680F620000000000000047C9C57BFA56BD08?preview=json

These three representations of each file are made available to you for easier integration of device media into your application:

JSON: Allows you to access file metadata and render corresponding controls.
Preview: Needed to enhance the UI experience.
Standard link: Provides access to the full contents of the file with the corresponding media viewer (we also offer one for your use, as mentioned below).

All operations with media files are logged in the device log. You can access this log from the device’s Logs & Messages tab or via the toolbox. To monitor any changes to media files in real-time, you may subscribe to the following MQTT topics:

flespi/log/gw/devices/+/media_file_uploaded to catch when a new file is uploaded to media storage.
flespi/log/gw/devices/+/media_file_updated to catch when a media file is updated.
flespi/log/gw/devices/+/media_file_deleted to catch when a media file is deleted due to user action.
flespi/log/gw/devices/+/media_file_expired to catch when a media file is automatically deleted due to TTL or rotation settings.

The JSON representation of a typical file is:

{
  "uuid": "unique identifier",
  "url": "link to access the file",
  "name": "file name",
  "size": file size in bytes,
  "mime": "MIME type",
  "created": timestamp of file creation on the device,
  "modified": timestamp of last modification,
  "uploaded": timestamp of upload to flespi media storage,
  "protected": boolean flag for protection status,
  "shared": boolean flag for sharing status,
  "preview": {
    "jpeg": "BASE64 encoded preview image..."
  },
  "meta": {
    "height": image height,
    "type": "file type",
    "width": image width
  },
  "channel_id": associated channel ID,
  "device_id": associated device ID
}

Important points to note in the JSON:

url: The link you use to access the file.
created: The time when the file was created on the device, which typically corresponds to the recording time.
uploaded: The time the file was uploaded to the flespi media storage.

If you add attributes to the file by using the PUT /gw/devices/XXX/media API method, your attributes will be combined with the above.

Video Download Command

To fetch a video clip from a remote device at a specific time, use the standard command named 'request_video'. This command may contain manufacturer-specific options for fine-tuning and always includes three standard options: camera channel index, recording time, and clip duration:

The typical flow that flespi executes when you queue such a command is illustrated by the slide from our conference, where this functionality was announced:

All the complexity is managed behind the scenes. In most cases, you simply queue the command once via REST API, and it durably downloads the video from the device into media storage on flespi, notifying you via corresponding logs and messages. A typical REST API call to queue video looks like this:

curl -X POST --header 'Authorization: FlespiToken XXX' -d '[{"name":"request_video","properties":{"channel":1,"duration":300,"from":1701084647},"ttl":86400}]' 'https://flespi.io/gw/devices/YYYY/commands-queue'

With this call, you instruct flespi to download a video from device YYYY recorded at the specified GMT timestamp (in GMT this is Monday, November 27, 2023 11:30:47) with a duration of 300 seconds. This command will remain in the command queue for up to one day (86400 seconds), awaiting the device to connect or be able to serve this command. Once the device connects, the command will direct it to send the requested recording via a secondary connection, store the file in device media storage, and complete the command with the link to the video file in the command response. Here's a screenshot demonstrating how this sequence looks in the real world, showing the order and types of events you can handle with flespi, all reported in MQTT in real-time on the corresponding topics.

With an MDVR type of device equipped with a powerful CPU and 4G or 5G connectivity, you will typically receive the video clip in a matter of seconds. In one specific case, once the command was queued via the REST API, it took just 4 seconds to end up with a 6MB file in video storage, with all listeners notified. This enables the functionality to request video in real-time through the REST API command, which will pause its response and wait for this process to complete. This means it's incredibly simple to retrieve video in a blocking call and receive a URL to the file for further processing. To illustrate, a 10-second video clip in full quality was downloaded in less than 8 seconds:

Command to retrieve a 1-minute video from another MDVR with sub-quality took 21 seconds, resulting in a 44MB file.

However, remember that real-time commands are great when you have a device online, and its connectivity is stable. For durable media retrieval, it's recommended to queue commands instead. More information on remote device commands can be found here.

As a result of the command's execution, you receive a link to the video in the command result, and flespi automatically registers a device message with full information about the video clip at the timestamp when the video was recorded. On the downstream application or platform, you can process such messages and present these videos to users or copy them to your private storage as needed. The JSON representation of such a message will always have 'media=true', and 'media.{type}.X' field contains video or photo data from channel X:

For full automation on the flespi side, it is recommended to use webhooks to request video from the device automatically upon certain events. You set up a webhook to the corresponding MQTT topic and payload (for instance, a harsh driving event reported by the device), and it instructs flespi to automatically download the associated video clip and register it as a message back to the device. Your downstream application then consumes these messages with a link to the video. This approach was illustrated in a presentation about flespi analytics at the flespi conference.

If you want to store media files outside of flespi, you can transfer them to your local storage and then delete them. Once deleted, the file's link will result in a 404 Not Found HTTP response code.

Live Video Streaming Functionality

Live video streaming from the device is based on HLS technology. You initiate streaming with the 'start_videostream' command, where the most crucial option is the channel number selection.

The command will wait for the device to connect (if not yet connected) and instruct the selected device camera to start streaming to flespi with the specified quality. Real-time commands are suggested here as this is typically interactive, and you want an immediate response with all required information to show streaming to the user. With an MDVR on a table, it took just 5 seconds to begin streaming. In the command response, you receive a link to an HLS playlist file, which can be loaded directly into a media player:

Streaming data is also available in the device connections. Each stream will have its own secondary connection, and you can access its data directly via the REST API:

To stop streaming, you simply delete the connection associated with it via the REST API. If the HLS playlist file is not accessed within 60 seconds, flespi will automatically delete it and stop the corresponding streaming.

Streaming activity is also reflected in the device logs:

To summarize: you initiate streaming with a command and halt streaming by deleting the associated connection or by ceasing to watch the .m3u8 playlist file.

Under the hood, streaming utilizes device media storage and stores all video chunks sent by the device. These video chunks are automatically deleted with a 2-minute timeout since they were last accessed. Keep in mind that if your device's media storage is blocked or full, live video streaming will not function properly.

Mediabox UI Tool

Everything described above has been packaged into a special UI tool called mediabox. This open-source tool can be integrated into your own application to seamlessly make use of flespi's video telematics features.

Mediabox is currently in development mode, and its UI is expected to change once we integrate a listing of connected cameras in the device configuration. It's not yet available as open source, but we plan to release it on flespi’s GitHub once it's stabilized.

You can integrate mediabox via an iframe with a corresponding token to outsource all work with device media, or pass file UUID directly to play or show only that file.

When integrated via an iframe with the token and device specified, mediabox will handle everything related to device media:

Browse available device media files in a calendar-like mode.
Display images, and play any video file or live video stream.
Queue commands to take pictures from cameras.
Queue the 'request video' command to download video from the device.
Start or stop live streaming.
Watch live streaming initiated by others for the given device.

The link for the iframe has the format https://mediabox.flespi.io/#/device/XXXXX?token=YYYYY where 'XXXXX' is the device ID, and 'YYYYY' is the token that grants access to this device. You'll get an interface like this:

There are buttons within the UI for requesting that the device upload video from history to flespi, or to start a live video stream (two streams are live on the right pane).

You can click on videos or streams to view or play them. If you want to control commands and have a good player for flespi media files you can embed mediabox with the UUID of a specific file (use the 'Share' button to list all available options for that):

Here is a sample video shared with mediabox in an 'iframe':

Well, that's all! Remember, the UI is not finalized, but integrating video telematics features into your application is super-easy - just as embedding the iframe at the desired location with the provided mediabox link.

Supported Manufacturers and Devices

This integration is essential as flespi without devices wouldn't be functional. Initially, in our strategy, we have decided to focus on several most popular hardware manufacturers rather than trying to support video from all possible sources.

Currently, video telematics features are supported with devices from:

These manufacturers have proven reliable, with a complete set of features and high performance in our tests. They came from the MDVR world with corresponding components on the device capable of high-volume video processing and this was feasible during integration, but drawbacks include their price and overall telematics data extraction from vehicles.

We've also integrated video and photo download commands from:

However, live video streaming is not available for these. Teltonika's limitation comes from the slower data transfer rates due to connecting cameras via a serial port. I'd suggest starting with short, 5-10 sec video clips to prevent device overload.

In collaboration with Jimi IoT (Concox), we are in the process of integrating JC450 and JC181. We've enabled live video streaming and the automatic download of event-based video if configured on the device. The integration of the historical video download command is based on the "ancient" FTP protocol, and we're developing the technological platform expected to be finalized in a month.

Upon the successful implementation of this technology into flespi, we plan to integrate other Concox devices and add Queclink to our list of video telematics manufacturers. If you can't find your device on the list of supported - feel free to reach us in chat.

You can also emulate video telematics device using Telegram protocol channel. There we can not request photo or video however any media uploaded by the user to the channel bot will pass through the same flow as the media from a normal device and will result in a media file link in device messages.

Please notice that direct integration via device-specific protocol is our requirement, hence we will not integrate with middleware video telematics platforms - it doesn't make any sense as they're mostly too unreliable.

Supported Video Codecs

We recommend starting with the H264 video codec due to its compatibility with all media viewers and compact file size. Infrastructure for automatic video conversion between codecs, such as the super compact H265, is in progress. Ultimately, flespi will manage codec complexities automatically. It's recommended to start from the H264 codec and proceed further as long as it's successfully tested.

Privacy

Our servers are located in Europe and comply with European GDPR regulations, and you can store as much data in flespi as needed. Alternatively, flespi can be used as a middleware or temporary storage. E.g., once the video is received from the device, you can catch this event via MQTT or a webhook, transfer the downloaded videos to private storage, and then delete them from flespi.

Currently, regardless of the codec conversion, flespi does not modify the video files — registration numbers and human faces are not blurred, but we may add AI-based video processing for this purpose if there is a demand.

Looking ahead, we aim to refine our video telematics functionality throughout the final month of 2023 and into Q1 and Q2 of 2024, based on our experiences with various devices from different manufacturers.

The key message of this long post is that with flespi, you can effortlessly integrate video telematics functionality into your application, without the need for extensive engineering resources. It simply works, and it works perfectly, at a price tag of 0.05-0.50 EUR per device. Wow, it's cool. This could revolutionize the telematics platform landscape over the next decade. It's just the beginning.