HTTP vs MQTT performance tests

We, developers, love benchmarking. Every time I speed test my Internet connection I feel like I am whizzing at full throttle.

As flespi stands for flexible and speedy, and quantifying flexibility is complicated, we test our solutions for speed. We decided to add MQTT broker to flespi because on some tasks MQTT protocol is faster and easier to use than HTTP. HTTP is universal and well-known. MQTT is fresh, lightweight, and designed for M2M and IoT communication. We checked how these two perform in different scenarios.

Case 1. Consuming messages from flespi in real time

This is probably the most evident case where the publish-subscribe concept of MQTT is more suitable for the task than the HTTP-based REST API. Let’s compare the two algorithms of receiving new channel messages in real time:

REST API:

1. perform GET /channels/XX/messages request

2. iterate over messages from “result” array field of the response to handle each of them

3. get a timestamp from “next_key” field of the response

4. perform the next GET /channels/XX/messages request with the specified next key

MQTT:

1. subscribe to flespi/message/gw/channels/# topic

2. handle each message as soon as it’s received by the MQTT client

Don’t your guts already tell you that the second algorithm is more efficient?

Test 0

By pure chance, the first algorithm is exactly what flespi_receiver does. It’s developed with python 3.5 based on the asynchronous architecture and should be rather fast. I’ve slightly modified the handler that just prints new messages to measure the delay between messages receipt by flespi and by the python script. To test the same period for Mosquitto MQTT client, I used a bash script like this (bash code): 

mosquitto_sub -h mqtt.flespi.io -p 8883 -V mqttv311 \
  --cafile /etc/ssl/certs/ca-certificates.crt -t "flespi/message/gw/channels/#" \
  -u "FlespiToken XXXXXX" |
/* we subscribed to 'messages' topic with mosquitto MQTT client */
while IFS= read -r line /* and run handler every time new message appears */
do
  echo $line /* print received message */
  NOW_TIME=$(date +%s.%N) /* get current timestamp with nanoseconds */
  RECV_TIME=$(echo $line | jq -r '.timestamp') /* get timestamp from received message */
  bc <<< "$NOW_TIME-$RECV_TIME" /* bash floating point subtraction of two timestamps */
done

Note: mosquitto_sub tool is not using root certificates file installed in your OS by default. Thus, to connect to flespi MQTT broker using secure connection, you have to provide additional command-line argument --cafile with path to your OS root certificates files.
For Linux-based OS it looks this: --cafile /etc/ssl/certs/ca-certificates.crt.
In addition, you may download "GlobalSign Root R1" certificate only for flespi.io on the GlobalSign site.

The message was just a simple JSON {"test":1234}. Ping result to the testing server is:

rtt min/avg/max/mdev = 59.227/65.902/73.218/4.110 ms

The result of tests was the following:

Conclusion: MQTT is on average 25 times faster even though the python module uses urllib3 and reuses keep-alive connection if possible.

Case 2. Measuring the amount of data received over the wire

MQTT is often called a protocol for the Internet of Things. Which means that it must be more lightweight for network usage. The experts in MQTT solutions also note that it’s especially efficient in wired data transmission.  Let’s see what network-related data we can get from packet sniffers to compare MQTT over SSL and HTTPS.

Test 1. Comparison of protocols service part

I’ve measured the number of bytes and packets required to establish a connection, send/receive data (simple JSON {"test":1234}) and close the connection. Here’s what I’ve got:

Conclusion: MQTT service part requires only 10% less traffic than HTTP. The advantage of MQTT service part over Ethernet vs Wireless is negligible.

Test 2. Real use case example: transmitting a bunch of messages

The previous test looks a bit synthetic. So I put together a more realistic use case. Imagine a hub that collects data from telemetry devices. The task is to feed this data to a business application via one of the following methods:

1. MQTT client keeps connection and publishes each piece of data to MQTT broker

2. HTTP keep-alive connection with POST request for each data piece

3. A single HTTP request with the entire pack of data

Yes, methods 1 and 2 have an advantage of delivering telemetry data to the platform as soon as it appears. But there are some possible applications where this advantage is not significant. So we’ll compare 3rd case out of competition :)

Conclusion: MQTTS is 20 times faster and requires 50 times less traffic on the task of posting consistent time-valuable data.

Case 3. Raspberry Pi 

I am not the first to compare the power consumption of MQTT and HTTP. Several years ago Stephen Nicolas did wonderful research on this topic with the next conclusion: in the long run, MQTT beats HTTP in energy consumption, but MQTT consumes more power than HTTP to establish a session. While the first conclusion correlates with my Test 2, I could not understand why MQTT is less power-efficient than HTTP. According to my Test 1, MQTT requires fewer TCP packets and less traffic. So I did more testing.

Test 3. Power consumption test

The first question was: “How to measure the power consumption?”. It’s obvious that I can’t measure exact power consumption for MQTT and HTTP sessions on my laptop, so I took a Raspberry Pi (2 model B). Raspberry is rather energy efficient, works over SSL for both MQTT and HTTP, allows disabling all unneeded Linux services, etc. Besides, everyone likes Raspberries, so more people will read this! The question became “How to measure the power consumption of Raspberry Pi?”. I decided not to play with ammeters and voltmeters, took a 5V 2000mAh battery, tuned Raspberry for an infinite loop of MQTT or HTTP sessions (like in Test 1) and measured:

1. the number of sessions until the battery dies

2. the lifetime of the test

3. average CPU usage

What do we get from this test? Two valuable parameters: mAh/session and CPU resource required for a session. Here are the test results:

Conclusion: the result is exactly what I’ve expected. Even in the short run, MQTT is up to 22% more energy efficient and 15% faster. And it does not depend on whether the connection type is 3G or Ethernet.

Testing outcomes

We have tested HTTPS applied to the flespi platform compared to MQTT over SSL connecting to the flespi MQTT broker. This is what we conclude:

1. Use MQTT to get data from the flespi platform: messages, logs, streams or storage events etc. It is much faster and easier to use.

2. If you want to build a data collecting system where messages consistency and time compliance are important, use MQTT. Free online public MQTT broker by flespi can be a reliable companion in this venture.

3. MQTT over SSL consumes less power than HTTPS in both wired and wireless connections. So we recommend using MQTT in standalone and wearable devices alike.

Interested in the topic? Read our new IoT-focused use case based on ESP8266.