Automatically translate this page?
23 January, 2018

HTTP vs MQTT performance tests

Comparison of HTTP and MQTT protocols for performance, power consumption, and CPU usage in different scenarios on a laptop and Raspberry Pi.

We, developers, love benchmarking. Every time I speed test my Internet connection I feel like I am whizzing at full throttle.

http vs mqtt performance test


As flespi stands for flexible and speedy, and quantifying flexibility is complicated, we test our solutions for speed. We decided to add MQTT broker to flespi because on some tasks MQTT protocol is faster and easier to use than HTTP. HTTP is universal and well-known. MQTT is fresh, lightweight, and designed for M2M and IoT communication. We checked how these two perform in different scenarios.

Case 1. Consuming messages from flespi in real time

This is probably the most evident case where the publish-subscribe concept of MQTT is more suitable for the task than the HTTP-based REST API. Let’s compare the two algorithms of receiving new channel messages in real time:

REST API:

  1. perform GET /channels/XX/messages request

  2. iterate over messages from “result” array field of the response to handle each of them

  3. get a timestamp from “next_key” field of the response

  4. perform the next GET /channels/XX/messages request with the specified next key

MQTT:

  1. subscribe to flespi/message/gw/channels/# topic

  2. handle each message as soon as it’s received by the MQTT client


Don’t your guts already tell you that the second algorithm is more efficient?

Test 0

By pure chance, the first algorithm is exactly what flespi_receiver does. It’s developed with python 3.5 based on the asynchronous architecture and should be rather fast. I’ve slightly modified the handler that just prints new messages to measure the delay between messages receipt by flespi and by the python script. To test the same period for Mosquitto MQTT client, I used bash script like this (bash code): 

mosquitto_sub -h mqtt.flespi.io -p 8883 -V mqttv311 \
--cafile /etc/ssl/certs/ca-certificates.crt -t "flespi/message/gw/channels/#" \
-u "FlespiToken XXXXXX" |
# we ve subscribed to 'messages' topic with mosquitto MQTT client
while IFS= read -r line # and run handler every time new message appears
do
echo $line # print received message
NOW_TIME=$(date +%s.%N) # get current timestamp with nanoseconds
RECV_TIME=$(echo $line | jq -r '.timestamp') # get timestamp from received message
bc <<< "$NOW_TIME-$RECV_TIME" # bash floating point subtraction of two timestamps
done

The message was just a simple JSON {"test":1234}. Ping result to the testing server is:

rtt min/avg/max/mdev = 59.227/65.902/73.218/4.110 ms

The result of tests was the following:

Delay from flespi to client, seconds

REST API python module

Mosquitto MQTT client

Average

0.768

0.0322

Max

1.274

0.0346

Min

0.379

0.0315

Conclusion: MQTT is on average 25 times faster even though the python module uses urllib3 and reuses keep-alive connection if possible.


Case 2. Measuring the amount of data received over the wire

MQTT is often called a protocol for the Internet of Things. Which means that it must be more lightweight for network usage. The experts in MQTT solutions also note that it’s especially efficient in wired data transmission.  Let’s see what network-related data we can get from packet sniffers to compare MQTT over SSL and HTTPS.

Test 1. Comparison of protocols service part

I’ve measured the number of bytes and packets required to establish a connection, send/receive data (simple JSON {"test":1234}) and close the connection. Here’s what I’ve got:

Secure session 

Outcoming bytes

Incoming bytes

Number of packets

HTTPS

1734

4186

20

MQTT over SSL (WiFi)

1274

4159

20

MQTT over SSL (Ethernet)

1186

4075

18


Plain TCP session 

Outcoming bytes

Incoming bytes

Number of packets

HTTP

675

431

10

MQTT (WiFi)

615

352

11

MQTT (Ethernet)

601

342

11

Conclusion: MQTT service part requires only 10% less traffic than HTTP. The advantage of MQTT service part over Ethernet vs Wireless is negligible.


Test 2. Real use case example: transmitting a bunch of messages

The previous test looks a bit synthetic. So I put together a more realistic use case. Imagine a hub that collects data from telemetry devices. The task is to feed this data to a business application via one of the following methods:

  1. MQTT client keeps connection and publishes each piece of data to MQTT broker

  2. HTTP keep-alive connection with POST request for each data piece

  3. A single HTTP request with the entire pack of data

Yes, methods 1 and 2 have an advantage of delivering telemetry data to the platform as soon as it appears. But there are some possible applications where this advantage is not significant. So we’ll compare 3rd case out of competition :)

1K messages

Bytes transmitted

Number of packets

Time, seconds

MQTT over SSL

1 publish per message, single session, QoS = 1

283,743

265


5.911

HTTPS

1 POST per message, keep-alive connection 

15,474,263

12,079

115.669

HTTPS

1 POST with 1000 messages

20,515


27

0.307

Conclusion: MQTTS is 20 times faster and requires 50 times less traffic on the task of posting consistent time-valuable data.


Case 3. Raspberry Pi 

I am not the first to compare the power consumption of MQTT and HTTP. Several years ago Stephen Nicolas did wonderful research on this topic with the next conclusion: in the long run, MQTT beats HTTP in energy consumption, but MQTT consumes more power than HTTP to establish a session. While the first conclusion correlates with my Test 2, I could not understand why MQTT is less power-efficient than HTTP. According to my Test 1, MQTT requires fewer TCP packets and less traffic. So I did more testing.

Test 3. Power consumption test

The first question was: “How to measure the power consumption?”. It’s obvious that I can’t measure exact power consumption for MQTT and HTTP sessions on my laptop, so I took a Raspberry Pi (2 model B). Raspberry is rather energy efficient, works over SSL for both MQTT and HTTP, allows disabling all unneeded Linux services, etc. Besides, everyone likes Raspberries, so more people will read this! The question became “How to measure the power consumption of Raspberry Pi?”. I decided not to play with ammeters and voltmeters, took a 5V 2000mAh battery, tuned Raspberry for an infinite loop of MQTT or HTTP sessions (like in Test 1) and measured:

  1. the number of sessions until the battery dies

  2. the lifetime of the test

  3. average CPU usage

What do we get from this test? Two valuable parameters: mAh/session and CPU resource required for a session. Here are the test results:

Raspberry power consumption

3G modem Internet connection

Ethernet internet connection


REST API via curl

Mosquitto MQTT client

REST API via curl

Mosquitto MQTT client

Number of sessions

9243

11728

20964

22864

Test lifetime

2h 39min

2h 50min

5h 53min

6h 21min

mAh per session

0.2164

0.17

0.0954

0.089

Messages per second

0.97

1.15

0.99

0.98

CPU usage, %

64

60

69

64

Conclusion: the result is exactly what I’ve expected. Even in the short run, MQTT is up to 22% more energy efficient and 15% faster. And it does not depend on whether the connection type is 3G or Ethernet.


Testing outcomes

We have tested HTTPS applied to the flespi platform compared to MQTT over SSL connecting to the flespi MQTT broker. This is what we conclude:

  1. Use MQTT to get data from the flespi platform: messages, logs, streams or storage events etc. It is much faster and easier to use.

  2. If you want to build a data collecting system where messages consistency and time compliance are important, use MQTT. Free online public MQTT broker by flespi can be a reliable companion in this venture.

  3. MQTT over SSL consumes less power than HTTPS in both wired and wireless connections. So we recommend using MQTT in standalone and wearable devices alike.

Interested in the topic? Read our new IoT-focused use case based on ESP8266.




Get the latest updates and monthly newsletters from flespi in your inbox

19 June, 2018 | flespi features | Jan Bartnitsky

What’s up with my flespi account? It’s in flespi/state/#

Collect custom data relevant to your project via flespi MQTT API and visualize it effectively.

10 April, 2018 | flespi features | Anton Kulichenko

First MQTT 5.0-compliant broker on the market. flespi takes the lead

The latest standard for IoT connectivity now supported. Shared subscriptions, retained messages, response topic, maximum packet size, and more. Plus bonuses from the flespi team!