AI classification of driving events

Video telematics devices raise ADAS and DSM alarms and upload the footage that goes with them. A fleet manager then has to watch the clips to tell a real event from a false alarm. The classify-driving-events AI tool does this review for you: it looks into a video clip or an image of a vehicle camera, lists the driving events it observes, and gives a verdict on each alarm the device reported with that footage.

The tool is a part of the flespi AI integration. It is available as a REST API endpoint, POST /ai/tools/classify-driving-events, and as a tool of the support MCP server.

Essentials

The tool works with footage stored in flespi. The usual source is the device media storage: a message with media=true carries the file in its media.video.X or media.image.X parameter, and the url field of that parameter is what you pass to the tool. Footage from any other source can be reviewed once it is uploaded to the flespi CDN.

There are two ways to use it:

• Detect events on your own: pass the link alone. The tool reports every event from its list that it observes in the footage. This works for any camera, including the ones that upload video without any ADAS or DSM parameters.
• Verify the alarms of the device: pass the link together with the device message. Each ADAS or DSM event raised in that message gets a verdict: confirmed, rejected or undetermined.

The review is done by an AI model, so treat its answer as the opinion of an attentive assistant, not as a measurement. Events in the cabin are recognized more reliably than road events, which depend on estimating a distance.

Usage

The classify-driving-events tool is called with a POST request to /ai/tools/classify-driving-events.

Request

• url: required, the link to the video or image in the device media storage (https://media.flespi.io/...) or in the flespi CDN (https://cdn.flespi.io/file/...).
• message: optional, the device message the footage came with, as a JSON object. Its raised adas.* and dsm.* event parameters are verified and position.speed is taken into account. Other parameters are ignored, so you can pass the message as is. A verdict is given for up to 16 reported events.

A video and an image are told apart by the content type of the file. Supported are MP4, MOV, 3GP, AVI, WebM, MPEG, FLV and WMV video, and JPEG, PNG, WebP and GIF images.

An example request that verifies two alarms of a cabin camera:

curl -X POST "https://flespi.io/ai/tools/classify-driving-events" \
  -H "Authorization: FlespiToken XXXXXXXX" \
  -d '{
    "url": "https://media.flespi.io/XXXXXXXX",
    "message": {
      "dsm.distraction.event": true,
      "dsm.seatbelt.event": true,
      "position.speed": 40
    }
  }'

Many devices report the alarm first and upload the clip a few seconds or minutes later, in a separate message. In that case pass the alarm message of the same device that is the closest in time to the clip.

Response

Like every AI tool, the response carries the result and the credits consumed by the call:

{
  "credits": 2,
  "result": {
    "footage": {"type": "video", "duration": 10, "size": 154955},
    "view": "cabin",
    "quality": "good",
    "summary": "The driver drives with the seatbelt fastened and looks down away from the road for several seconds.",
    "events": [
      {
        "parameter": "dsm.distraction.event",
        "at": 0,
        "confidence": "high",
        "evidence": "The driver looks down away from the road from the start of the video until about 00:06."
      }
    ],
    "reported": [
      {
        "parameter": "dsm.distraction.event",
        "verdict": "confirmed",
        "evidence": "The driver looks down away from the road for over six seconds."
      },
      {
        "parameter": "dsm.seatbelt.event",
        "verdict": "rejected",
        "evidence": "The driver wears the seatbelt across his chest throughout the footage."
      }
    ]
  }
}

Detected events

The classify-driving-events tool names the events after the standard flespi message parameters, so the result can be written back to a message or compared with it directly. An event is reported only for the matching camera view.

Verification is not limited to this list: any raised adas.* or dsm.* event parameter of the message gets a verdict. When the message raises no such parameter but names the alarm in adas.event.name or dsm.event.name, the verdict is given for that name.

Cost and limitations

The classify-driving-events tool consumes AI credits by the footage it reviews:

A typical event clip of a dashcam lasts 10 to 15 seconds and costs 2 credits. A call reserves 10 credits and refunds the unused ones once the footage is reviewed, so the credits limit of the account needs room for 10 credits at the moment of the call. Nothing is charged when the link or the footage is refused, or when the review fails.

Devices that record continuously upload hundreds of clips a day. To keep the cost in line with the value, review the footage that comes with an alarm rather than every file.

Limitations

• A video of up to 5 minutes and 50 MB, an image of up to 10 MB.
• The duration of a video is read from the header of an MP4, MOV or 3GP file, and the video is reviewed up to that duration. A video in another container, or one that declares no duration, costs 10 credits and is reviewed up to its 5th minute.
• Event detection is tuned for short event clips: a video that declares a duration of up to a minute is reviewed at 4 frames per second, a longer one or one that declares no duration at 1 frame per second, where a brief event may fall between the frames.
• An image is a single frame. What it shows counts as the event even where the event is defined by how long it lasts, with a confidence below high.

Troubleshooting

A request to classify-driving-events that cannot be served returns HTTP 400 with the reason:

HTTP 401 means that the request carries no valid token: the call needs POST ai/tools in an ACL token, or POST ai/mcp when the tool is called through MCP. HTTP 403 means that the AI credits of the account are exhausted, HTTP 429 that the per-minute ai_calls rate limit is exceeded.

Every call is recorded to /ai/logs with the credits it consumed. Use the tool=="classify_driving_events" filter to see the calls of this tool.