How to verify dashcam ADAS and DSM alarms with AI?

Inexpensive dashcams raise a lot of false ADAS and DSM alarms. The ai-driving-events plugin checks an alarm against the footage the device uploads for it: when a message with a video or an image is registered, the plugin passes the file to the AI classification of driving events, adds the review to the message and resets the alarms that the footage rejects. Streams, webhooks and calculators receive a message that is already verified.

The plugin works with messages that carry a file of the device media storage in a media.video.X or media.image.X parameter, other messages pass through untouched. For a message with footage the plugin:

• Finds the alarms: the raised adas.*.event and dsm.*.event parameters of the message, or its adas.event.name and dsm.event.name. Most devices report the alarm and upload the clip in separate messages, so when the media message has no alarm, the plugin takes the alarms the device registered during the clip: from one second before the timestamp of the media message to one second after the end of the video, and within 5 seconds of an image. A clip whose parameter reports no duration is assumed to last 15 seconds.
• Reviews the footage: each file is reviewed together with the alarms and the speed: the one of the media message, or the one at the alarm when the media message has none. Up to 4 files of one message are reviewed in parallel, the first four by the parameter name.
• Finalizes the message: the review is added as the driving.events.review parameter, the event parameters are corrected according to the mode. In the verify and classify modes the alarms found in the device storage are copied into the message together with the other adas.* and dsm.* parameters of their message, such as the alarm level, unless all the alarms of that group are rejected.

The alarm message is already registered by the time the clip arrives and stays as the device sent it. Treat it as the claim of the device and the media message as the verified event: build notifications and reports on messages with driving.events.review. A calculator that counts events in all messages sees such an event twice, exclude one of the two with a media check in the selector.

How to use

To create a new plugin, navigate to the Telematics Hub > Plugins section and click on the '+' button. Name the plugin, select 'ai-driving-events' as the 'Type id'. Fill in the 'Validate message' field with the expression 'media==true', so that only messages with media reach the plugin and its log. The expression can be narrower, for example 'exists('media.video.1')'.

The Configuration section of the plugin has two options:

• Mode: what footage is reviewed and what is changed in the message.
  • 'Verify alarms' (verify, default): footage that comes with an alarm is reviewed, and an alarm with the rejected verdict is reset to false. Footage without an alarm is not reviewed and costs nothing.
  • 'Verify alarms and detect events' (classify): all footage is reviewed. Rejected alarms are reset, and each of the detected events with a high or medium confidence is raised as its standard parameter, for example dsm.driver.phone.event=true, unless the review rejected it as an alarm. A plain camera without ADAS and DSM functions then produces the messages of an ADAS and DSM device.
  • 'Report only' (report): footage that comes with an alarm is reviewed and the review is added, nothing else is changed. Use it to decide on your own, for example in a msg-pvm-code plugin with a lower priority.
• Channels: camera channels to review, all channels if empty. A multi-channel device uploads a clip from every camera for one alarm: list the channel of the cabin camera to verify DSM alarms, or of the road camera for ADAS alarms.

An alarm with the undetermined verdict is never changed, for example a seatbelt alarm checked against the clip of the road camera. With several files an alarm is reset only when at least one file rejects it and none confirms it. The adas.event.name or dsm.event.name parameter is removed when it is rejected itself, or when all the raised events of its group are rejected. The original claim of the device stays visible in the reported list of the review.

Now assign the plugin to a device: navigate to the Devices tab of the plugin, click on the '+' icon and select the device from the list. Leave the required flag of the plugin off, otherwise a message with footage that cannot be reviewed is not registered at all.

As the device uploads footage, the messages are processed by the plugin. A message of a cabin camera that reported a distraction and an unfastened seatbelt looks like this after the verify mode:

{
  "timestamp": 1789926449.924622,
  "media": true,
  "media.video.1": {"url": "https://media.flespi.io/XXXXXXXX", "mime": "video/mp4", "size": 154955, "meta": {"channel": 1, "duration": 10}},
  "position.speed": 40,
  "dsm.distraction.event": true,
  "dsm.seatbelt.event": false,
  "driving.events.review": [
    {
      "media": "media.video.1",
      "view": "cabin",
      "quality": "good",
      "summary": "The driver is driving while wearing his seatbelt, but looks down away from the road for several seconds.",
      "events": [
        {
          "parameter": "dsm.distraction.event",
          "at": 3,
          "confidence": "high",
          "evidence": "The driver looks down away from the road for more than 3 seconds."
        }
      ],
      "reported": [
        {
          "parameter": "dsm.distraction.event",
          "verdict": "confirmed",
          "evidence": "The driver looks down away from the road from about second 03 to second 06."
        },
        {
          "parameter": "dsm.seatbelt.event",
          "verdict": "rejected",
          "evidence": "The driver has a seatbelt visibly fastened across his chest."
        }
      ]
    }
  ]
}

The clip confirms the distraction and shows a fastened seatbelt, so dsm.seatbelt.event is reset. The driving.events.review parameter is an array with one item per reviewed file, media names the parameter of the file. It is absent when nothing was reviewed. The fields of the review are described in AI classification of driving events.

Cost

Each reviewed file consumes AI credits of the account that owns the plugin: 1 credit for an image, 2 credits for a video of up to 20 seconds, 4 credits up to a minute and 10 credits up to 5 minutes. The duration is read from the header of an MP4, MOV or 3GP file: a video in another container, or one that declares no duration, costs 10 credits. A refused file and a failed review are not charged. The calls of the plugin are listed in /ai/logs with tg.plugin.ai-driving-events.ID in the app field, where ID is the ID of the plugin.

Troubleshooting

What happened to each message is written to the Logs tab of the plugin with the reason:

Performance and stability considerations

• The registration of a media message waits for the review, which typically takes 4 to 15 seconds.
• When nothing is reviewed, the message is registered as the device sent it. With the required flag set, such a message is rejected instead.
• Only the alarms that come with footage are verified. Devices upload footage for a part of their alarms, and an alarm without footage stays as reported.
• The review is an opinion of an AI model. Events in the cabin are recognized more reliably than road events, which depend on estimating a distance: the verdict on a close following or a collision warning may differ between two reviews of the same clip.

Change log

Subscribe to the ai-driving-events plugin change log to stay in sync with any updates.

More plugin types

Find a comprehensive list of available plugin types here.