Expressions — the way to analyze message

How to use expressions to extract information from telematics message using own mathematics

Analytics system uses expressions everywhere — in interval selectors, in interval counters, and even in message and interval validators. Expression operates with a message and the result of an expression is a typed number, string, boolean or null value. All numbers are operated as floating-point with double precision, thus limiting integers to 53 bits in size.

An expression consists of numbers, operators, parameters references, and functions. If an expression contains a reference to a parameter that is not present in the message or division by zero, its result is undefined or OFF in most boolean expressions. In the case of boolean-type expressions (e.g. validation or interval test expressions), a non-zero resulting value means the expression is valid and zero, false or null will deny validation of the message or interval. 

Important! We have released a handy expressions testing tool that may greatly simplify debugging of the flespi expressions.

For example, we sequentially analyze the following messages from devices:

{
  "device.name": "123456789012345",
  "timestamp": 1490347944.893743,
  "din": 9,
  "channel_id": 123,
  "altitude": 568.49,
  "direction": 297,
  "hdop": 0.9,
"speed": 10,
  "lat": -21.328481,
  "lng": 47.562136,
  "accelerations": [1, 2, 3.3, 0],
  ...
}

Expressions samples that can be used for such kinds of messages are:

  • device.name == "12*45"
  • timestamp > 1490347940 && timestamp < 1490347950
  • (hdop < 1 && (din & 0x1)) || speed == 0
  • if(altitude > -500 && altitude < 500, altitude, error())

In an expression, all references to message parameters that contain a number, boolean, textual, or null value will equal this value. All references to other types of parameters (e.g. array, nested object, or binary) will make expression evaluation impossible.

It is possible to use true, false, and null values in the expression. The type of value can be tested with functions: isnull, isnumber, isstring, isboolean, isjson and safely converted to another type with functions tonumber, toboolean, tostring.

In mathematical comparison operators, the maximum level of automatic type casting is used: null and false are cast to zero, true or any valid string — to one.

For textual parameters you can use comparison operators like '==' or '!=', for example: "device.name == '123456789012345'" to check the actual value. When comparing two strings, you may use wildcards, for example: "device.name == 'Telto*'". For case-insensitive comparison, use a special operator "~", e.g. "device.name ~ 'TeLTo??kA'".

If you need to access nested (JSON) objects or arrays, use a notation object-key['json-path'], for example ble.sensors['/0/rssi'] to access BLE sensor with first index and its rssi value. If unable to extract value under path, it will always return null. See more examples and explanations for json function in the table below.

To reference the value of the same parameter in the previous message, use previous("parameter-name") function. For example, you can use "speed - previous('speed')" to calculate the speed change from message to message. Caveat: re-registered messages cannot be referred to by the previous() function; previous values of parameters are only available for messages that are newer (fresher) than all the existing ones.

Important note on using references to previous message parameters in calculators: as the system does not know the value of the previous parameter for the very first message, it means that the very first message in such reference will always return an unknown or OFF value. In order to correctly and consistently calculate intervals, we recommend using merge_message_before=true in selector configuration, which expression uses the reference to previous message parameters.

Another form of parameter specification is by preceding them with a dollar sign ($). In that case, if the parameter is not present in the message, a null value will be put instead and the expression can be evaluated further. For example, "abs($speed)>=0" will be always valid and return a boolean true.

You can use mathematical operators in expression and add brackets to control the order of operations. The priority of operations is the same as in the C programming language and notation, in general, is also very similar to what the C language defines. You can use the following operators:

operator

explanation

+sum
-diff
/divide
*multiply
|binary OR
&binary AND
&&logical AND
||logical OR
^binary XOR
==equal; for strings will match with wildcards
!=not-equal; the inverted value of equal operator
~case insensitive match of a string with wildcards
<less
<=less or equal
>greater
>=greater or equal
>>right shift by the specified amount of bits (in the range from 0 to 64)
<<left shift by the specified amount of bits (in the range from 0 to 64)

If you need more operators, please contact us

Here are some sample expressions and their resulting values:

expression
evaluation
1 + 2 * 4 / 25
(1 + 2) * 4 / 26
(4 & 1) > 0false, because 4 AND(binary) 1 is 0
(4 && 1) > 0true, because 4 AND(logical) 1 is 1

It is also possible to use some functions inside expressions, for example, "abs(speed-#speed)>10" will always be true if the instant speed value change between two sequential messages is different by more than 10 km/h in any direction. Here is the full list of functions:

function
explanation
previous(X)
previous known value of X or null if not yet available
abs(X)an absolute value of X
round(X)round X to the closest integer
ceil(X)round X up to the closest integer
floor(X)round X down to the closest integer
sqrt(X)the square root of X
min(X, Y)return a minimum value between X and Y
max(X, Y)return a maximum value between X and Y
if(X, Y, Z)if X is non-zero, evaluate to Y otherwise evaluate to Z
mileage()mileage in kilometers from the previous message, calculated using position.latitude, position.longitude and position.altitude parameters
distance(lat1, lon1, lat2, lon2)distance in kilometers between two points X and Y with coordinates lat1, lon1 and lat2, lon2 where latX is latitude of point X and lonX is longitude of point X
month(X)month value for X with ranges 1-12, where X is UNIX timestamp, e.g. month(timestamp) will return month number for the current message
day(X)day of month value for X with ranges 1-31, where X is UNIX timestamp
hour(X)hour for X with ranges 0-23, where X is UNIX timestamp
minute(X)minute value for X with ranges 0-59, where X is UNIX timestamp
strftime(X, Y)format date into text, where X is UNIX timestamp and Y is a format string similar to strftime.
now()current time as UNIX timestamp with microseconds granularity
json(X, Y) or short form: X[Y]
fetch the value from the sub-element inside a JSON object or array.

Sample JSON: {"x":{"y":1, "z":2}, "a":[3, 5, {"keyA":"valueA"}]}

To fetch value under key "y" inside object under key "x" use json(x, 'y') or x['y'] which will return 1.

To access arrays use json(a, 0), json(a, '/0') or a[0] which all will return 3.

JSON path can be used for as well for fetching values from complex objects/arrays: json(a, '/2/keyA') or a['/2/keyA'] will return "valueA".

If unable to extract value under path will return null.
hex(X[, Y, Z])Convert string value X from hexadecimal format into 64-bit unsigned integer and take Z bits starting from Y. If Z is not defined, all bits will be taken, if Y is not defined, zero is assumed (first bit).

Samples: 
  • hex("FF") will return 256.
  • hex("08", 0, 2) will return 0.
  • hex("F8", 2, 2) will return 2.
exists('X')return 'true' if the specified parameter with name 'X' exists in the message and 'false' otherwise
error()generate an error and fail expression evaluation. Can be used in conditional evaluation like: if(position.valid, mileage(), error())
not(X)invert argument value - if zero/false, return non-zero/true and vice versa
tonumber(X), tostring(X), toboolean(X)convert value to another type
isnumber(X), isstring(X), isboolean(X), isnull(X), isjson(X)
test if value is of the specified type and return 'true' in such case

If you need more functions, please contact us.


See also
Forwarding intervals generated by the flespi analytics engine into your solution.
A comprehensive guide on the key concepts and principles of the flespi analytics engine.