Video Telematics Integration

 A comprehensive overview of video telematics functionality integration into your application with flespi.
api
devices
video

Flespi is a back-end platform that offers a complete set of tools and technologies to swiftly integrate advanced video telematics functionality into your application.

Key features:

Device media storage

All video and photo functionality in flespi is device-centric and requires a registered device.

Device media storage is used to store persistent or temporary media data related to the device. 

Each device is allocated its dedicated storage, with up to 10GB available. You manage the actual volume of device media storage by utilizing 'media_ttl' and 'media_rotate' properties in the device configuration:

For instance, if you set the media storage rotation size to 1GB, whenever a new file is uploaded and the total media storage size exceeds 1GB, older files will be automatically deleted. We do not recommend setting it below 1 GB. Either you will not be able to retrieve large video clips or will be very limited on video streaming functionality which also uses device media storage during its operation.

You can safeguard specific files from automatic deletion due to TTL or size rotation by setting a protected attribute for each particular file.

The REST API for accessing device media files can be found here. The following operations are available for each device:

  • Search for some files or list all files.
  • Delete all or specific files.
  • Add to the file your own set of properties or enable/disable file protection from automatic deletion.

All files possess a unique UUID property used for identification, accompanied by a unique link that allows you to download or access the file directly from your application. Accessing a file using this link doesn't require tokens for authorization. The link is accessible through GET and HEAD HTTP methods, and you can also request a preview and JSON representation of the file. Consider these three links for the same media file:

These three representations of each file facilitate the integration of device media into your application. With JSON you can access file metadata and render corresponding controls. The preview link is crucial for creating a user-friendly interface, while the normal link provides access to the complete contents of the file with the corresponding media viewer (which we also provide, see below).

All operations with media files are reported in the device log. To monitor real-time changes to media files, you can subscribe to the following MQTT topics:

  • flespi/log/gw/devices/+/media_file_uploaded to catch when a new file is uploaded to the media storage
  • flespi/log/gw/devices/+/media_file_updated to catch when a media file is updated
  • flespi/log/gw/devices/+/media_file_deleted to catch when a media file is deleted due to user action
  • flespi/log/gw/devices/+/media_file_expired to catch when a media file is automatically deleted due to TTL or rotate settings

JSON representation of a typical file is:

  {
  "uuid": "C7F49F4EB8EE560A0000104148A3780FB1F80B00000000009D3031141F0BE3F4",
  "url": "https://media.flespi.io/C7F49F4EB8EE560A0000104148A3780FB1F80B00000000009D3031141F0BE3F4",
  "name": "photo.jpg",
  "size": 533799,
  "mime": "image/jpeg",
  "created": 1701080990,
  "modified": 1701080991.36393,
  "uploaded": 1701080991.36393,
  "protected": false,
  "shared": true,
  "preview": {
  "jpeg": "BASE64 OF PREVIEW IMAGE..."
  },
  "meta": {
  "height": 1280,
  "type": "image",
  "width": 960
  },
  "channel_id": 1172707,
  "device_id": 5298643
}

Here are a few important points to note:

  • 'url' is the link you use to access the file;
  • 'created' is the time when the file was created on the device (typically, this corresponds to the recording time);
  • 'uploaded' is the time the file was uploaded to the flespi media storage;

You can add some attributes to the file by using the PUT /gw/devices/XXX/media API method. This allows you to annotate the file with your custom data.

Video download

To retrieve a video clip from the remote device at a specific time, you use the standard command named 'request_video'. This command may contain manufacturer-specific options, providing fine-tuning capabilities when necessary. It always includes three standard options: camera channel index, video recording time, and clip duration.

The camera specifications can be included directly in every command, or the user may pre-define it in device configuration tab in the Cameras field. In this case you may specify "Configured camera" option in the video functionality command and specify only the index of pre-configured camera:

The typical sequence that flespi executes when such a command is queued is illustrated below:

In most cases, a single queuing of the command via the REST API suffices. It robustly downloads the video from the device into the media storage on flespi and notifies of this action through corresponding logs and messages. A regular REST API call to queue a video looks like this:

curl -X POST --header 'Authorization: FlespiToken XXX' -d '[{"name":"request_video","properties":{"channel":1,"duration":300,"from":1701084647},"ttl":86400}]' 'https://flespi.io/gw/devices/YYYY/commands-queue'
  • With this call, you instruct flespi to download video from device YYYY recorded at 1701084647 (in GMT this is Monday, November 27, 2023, 11:30:47) and duration of 300 seconds.
  • This command will stay in the command queue for up to 1 day (86400 seconds) waiting for the device to be connected and be able to serve this command.
  • Once the device is connected the command will instruct it to send the requested recording via the secondary connection, store this file in device media storage, and finish the command with the link to the video file in the command response.

Here is how this sequence looks in the real world. Pay attention to the order and type of events you can handle with flespi. They all reported into MQTT in real-time in corresponding topics.

With an MDVR type of device equipped with a powerful CPU along with 4G/5G connectivity, you typically receive the video clip within a matter of seconds. This makes it even possible to request video in a real-time REST API command that will pause its response, wait for this flow to be fully processed, and return a link to the downloaded video file:

However, it’s important to note that real-time commands are great when the device is online and its connectivity remains stable. For durable media retrieval, we highly recommend executing commands via the device commands queue. 

Should you wish to submit a video download request for multiple channels within a single API call, you can pass multiple commands at once. Just send an array of commands to the device:

curl -X POST ‘https://flespi.io/gw/devices/{device-id}/commands-queue’ -d ‘[{channel1-command-json}, {channel2-command-json}, …]’

Once the command is executed you get the link to the video in the command result. Flespi also automatically registers a device message with full information about the video clip at the recording time as a timestamp. On the downstream application or platform, you can just process such messages and show these videos to users or just copy them to your private storage whatever is needed. JSON representation of such message will always have 'media=true' field and 'media.{type}.X' field contains video or photo data from channel X:

Automatic event-based video retrieval

If your device allows configuration for automatically uploading photo or video data to the backend server upon specific events, flespi fully supports this functionality.

For devices lacking such configuration options, webhooks can be utilized to automatically handle the triggering event in the cloud and subsequently request video from the device in response.

In both variants, your downstream application will immediately receive the device message via stream or MQTT with two parameters:

  • ‘media=true’ indicates that this message includes media data.
  • ‘media.photo.X’ or ‘media.video.X’ holds video metadata information, including the link to the video from channel X. This link can be used to download video to your local storage or directly share it with the user in your application.

Should you prefer not to store media files in flespi, you have the option to delete them after copying them to your storage or enable auto-rotation of media data based on size or time. Once the file is deleted, attempting to access its link will result in a 404 Not Found HTTP response code.

Typically, such messages primarily contain the recording timestamp as reported by the device and parameters holding media data. However, before pushing a message to the stream or MQTT, you can enrich it by incorporating full telemetry data from the associated device storage using the 'msg-merge' plugin.

Live video streaming

Live video streaming functionality from the device is based on HLS technology. You instruct the device to start streaming with ‘start_videostream’ command where the most important option is to select the channel number:

The command will await the device’s connection (if it’s not already connected) and instruct the selected device camera to start streaming to flespi with the specified quality. In this scenario we recommend using real-time commands, as this action is usually interactive and you need a response with the REST API call, providing all the necessary information required for displaying the streaming content to the user. Below is a sample call that initiates video streaming on the device and returns a link to the HLS m3u8 playlist file. This link can be loaded into a media player:

The corresponding streaming information is also accessible within the device connections. Each stream will possess its secondary connection, and you can directly access its data through the REST API:

To stop streaming, you simply delete the associated connection using the REST API. If you don't access the HLS .m3u8 playlist file within 60 seconds, flespi will automatically delete it, thus terminating the corresponding streaming session.

You can track video streaming, whether initiated by you or other users, with MQTT subscriptions to specific device topics:

  • flespi/state/gw/devices/+/connections/+: retain topic to track device connections;
  • flespi/log/gw/devices/+/mediastream/+: real-time only topic to track live video streaming ‘start’ and ‘stop’ events.

For starting live streaming across multiple channels within a single API call, you can pass to a device multiple commands at once. Just send an array of commands and then wait: 

curl -X POST ‘https://flespi.io/gw/devices/{device-id}/commands’ -d ‘[{channel1-command-json}, {channel2-command-json}, …]’

Or you can add them to the commands queue to avoid blocking API calls.

Streaming functionality utilizes device media storage and stores all video chunks transmitted by the device. These chunks are automatically deleted after a 2-minute timeout since the last access. It's important to note that if your device experiences media storage blockage or overflow, live video streaming won't be possible. Thus, we recommend reserving at least 1 GB, or even more, in automatic media rotation size.

Playback video

In addition to video file download and live video streaming functionality, there is an option to view video files recorded directly from the device. This is commonly known as the “video playback” feature.

To start playback you use ‘playback_video’ command. It will require the same set of options as ‘request_video’ command described above, with the only difference that playback activation command result will contain a link to the HLS live stream playing back this file through flespi directly from the device. Thus you can start watching the video almost immediately.

Playback start time does not depend on the file size or device network bandwidth, so you can effectively use real-time commands execution REST API instead of queue-based commands execution REST API to receive HLS stream URL in REST API call response.

Video timeline

You can use the ‘video_timeline’ command to ask the device for the list of available time intervals with video data recorded on the MDVR device storage. This command requires two options - time from and time to. In the result you will receive a JSON array with objects representing every available interval with recorded video.

Its typical JSON representation is:

[
    {
      "from": 1709114905,
      "to": 1709119537,
      ...
    },
    ...
]

In addition to the listed fields there might be additional protocol-specific non-required fields to filter out specific channels (filter_channels) or streamtype (filter_streamtype).

Objects in result array will contain fields begin and end represented as timestamps, and may contain additional manufacturer-specific fields like size, filename or streamtype.

Mediabox UI tool

Mediabox is an open-source tool that you can integrate into your own application to use all video telematics features provided by flespi seamlessly and with minimal effort.

Integration via iframe is done by using corresponding token to delegate all device media-related work to mediabox or, alternatively, passing the file UUID directly to play or display only a specific file.

When integrated via iframe with the token and specified device, mediabox streamlines the handling of device media right out of the box. You can:

  • Browse available device media files in a calendar-like mode
  • Display images and play any video file or live video stream
  • Queue commands to capture pictures from cameras
  • Queue a request video command to download video from a device
  • Start or stop live-streaming
  • Watch live streaming initiated by others for the given device;

To integrate mediabox via an iframe with the token and device specified, use the following link format:

https://mediabox.flespi.io/#/device/XXXXX?token=YYYYY, 

where XXXXX represents the ID of the device, and YYYYY is the token granting access to this device. Upon integration, you'll access an interface similar to this:

For ACL-type tokens you should grant access to specific device submodules:

  • commands, commands-queue, commands-result
  • connections
  • media
  • logs

However, if you prefer to manage commands independently and solely need an efficient player for flespi media files, you can embed mediabox by providing only the file UUID without a token. Within mediabox, each file features a 'Share' button that provides a list of the following options:

In this mode mediabox is able to automatically detect and visualize the contents of media file by the link - be it an image, video clip or live video stream from the camera. You just embed it into a page and it will show corresponding media content to user.

Supported manufacturers and devices

You can use all flespi video telematics functions with the following manufacturers:

Specific device models compatible with video functionality can be found in this list of supported devices.

Should you require integration of video telematics devices from other manufacturers, you can request it. Please note, we do not integrate video functionality from middleware hubs or platforms. Direct integration between the device and flespi is our requirement.

Supported video codecs

Flespi mitigates all complexities with video codecs seamlessly. Our standard video codec is H264, compatible with all media viewers and relatively compact in size.

If a device reports video in another codec, such as the highly compact H265, flespi will automatically convert it to H264 before storing or streaming. However, auto codec conversion may not be available for certain protocols. To make testing easier when you first connect a new type of device, it is recommended that you initially configure it with the H264 codec. After successful testing, you can switch to an alternative codec.

Cost and limitations

Video telematics functionality is not charged separately and is included in the standard flespi device cost of €0.02/month. 

Additionally, you are charged €1/month for each extra GB of device messages and media data stored in flespi. If you are using live streaming functionality, it will also temporarily count towards device media storage. You have complete control over the amount of data stored per each device.

And for each device, we measure and limit the device media traffic that was generated either by the device itself during the media upload process or by your users or applications during media retrieval via the REST API. 

Media traffic is limited on a per-device basis. When the device media traffic limit is overflowed, only operations with device media will be unavailable. You can identify such occasions by the "device media blocked" errors in the device log for certain operations.

To increase the media traffic limit, consider upgrading your plan to the next level.

See also
How to add calculated interval data to device telemetry
How to add data generated by calculators to device messages
analytics
devices
messages
How to use FormBox?
How to automate the process of forms creation for your web-project.
api
tutorials