Raspberry Pi and Redis

This summer we did something amazing: we took our enterprise appliance, a very powerful server and we tried to run same software, same analytic kernel and all the other things in something smaller and lot simpler, like this:

Light, mobile appliance

And we did it ! Being powered by OpenResty and Lua we were able to size our appliance easily and be up and running quickly. Low factor, low power consumption, very cheap, these ARM based devices are getting more and more attention and become more popular nowadays:

• Banana Pi - A Highend Single-Board Computer
• 4 x Raspberry Pi + Arduino = UDOO

What ? Analytics on a Raspberry PI ... !?

Yep. First of all, we wanted to experiment with ARM based boards using few number of hosts and few data messages per host as input. The immediate trouble was that we should fit in 512MB RAM with all our software and customer data. Then these boards don't run on latest Intel specs but something like 700 Mhz CPU as a system on chip with 100Mbps NIC and shared USB ports.

But still we were thrilled by this idea of having, few hosts and messages per appliance and be able to deploy such appliance quicker than a larger one for different setups. For example

• Enterprise Appliance: up to 5000 hosts, 5 messages per host, 25.000 messages
• PI Appliance: up to 5 hosts, 2 messages per host, 10 messages

And we went Raspberry Pi. Nothing different on the appliance itself, except some configuration changes and lots of testing. And here comes the fun...

Redis on Raspberry PI

A central part of our analytic software is Redis, a very popular and powerful NoSQL in cache memory data structure server. It runs on majority of Linux distributions, *BSD and many other UNIXes. It has a nice and friendly community and you get lots of help around. For our analytic software Redis was a natural choice: several ways to structure data in memory, very fast, no threading and simple.

The RBPi model B+ used for this test:

As storage we have selected these two cards: SanDisk and Kingston 16GB , Class 10. We used default ext4 file system on both card, with default options.

king-msdhc6-c10

[    2.112778] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[    2.123339] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[    2.233281] Waiting for root device /dev/mmcblk0p2...
[    2.284667] mmc0: read SD Status register (SSR) after 3 attempts
[    2.305649] mmc0: new high speed SDHC card at address 0007
[    2.319414] mmcblk0: mmc0:0007 SD16G 14.9 GiB 
[    2.327567]  mmcblk0: p1 p2

SanDisk 16GB
Class 10

[    2.131742] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[    2.142791] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[    2.262561] Waiting for root device /dev/mmcblk0p2...
[    2.293773] mmc0: read SD Status register (SSR) after 2 attempts
[    2.316876] mmc0: new high speed SDHC card at address aaaa
[    2.332197] mmcblk0: mmc0:aaaa SU16G 14.8 GiB
[    2.345618]  mmcblk0: p1 p2

Transcend 32GB
Class 10 MLC

[    2.167920] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[    2.180580] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[    2.358084] mmc0: read SD Status register (SSR) after 2 attempts
[    2.381255] Waiting for root device /dev/mmcblk0p2...
[    2.392431] mmc0: new high speed SDHC card at address 59b4
[    2.418636] mmcblk0: mmc0:59b4 USDU1 29.7 GiB 
[    2.430805]  mmcblk0: p1 p2

After several weeks of testing and testing again, we found several issues with Redis on Raspberry PI when using Kingston cards:

• slow storage
• lots of Lua client errors
• high CPU Utilization

Types of Errors

2014/09/11 11:53:57 [error] 2590#0: *43736 lua tcp socket read timed out, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx"
2014/09/11 11:53:57 [error] 2590#0: *43736 attempt to send data on a closed socket: u:B6925060, c:00000000, ft:0 eof:0, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx"

[3305] 18 Oct 12:52:31.024 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

2014/10/26 21:17:58 [error] 2712#0: *14598 lua tcp socket read timed out, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx"  2014/10/26 21:17:58 [error] 2712#0: *14598 [lua] messages.lua:154: deposit_stats_data(): Error setting expiration for key: stats:3f87c584-2312-42e5-67e4-93005181870d:eth0:cpd-linux-nicrec:rxpkt:2338:10800:MIN | expire at: 1415232000 | err: timeout, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx" 2014/10/26 21:17:58 [error] 2712#0: *14598 attempt to send data on a closed socket: u:B5FA41B8, c:00000000, ft:0 eof:0, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx"

Latency Doctor

Redis has something amazing called Latency doctor. All things explained here: Redis latency monitoring. Using that we were able to see that majority of all our timeouts were caused by a slow storage media, in fact the SD and MicroSD cards. And we start asking the doctor:

1. aof-write: 160 latency spikes (average 1312ms, mean deviation 726ms, period 19.63 sec). Worst all time event 5686ms. 

2. aof-write-alone: 160 latency spikes (average 1355ms, mean deviation 749ms, period 20.49 sec).  Worst all time event 5686ms. 

3. aof-write-active-child: 11 latency spikes (average 1088ms, mean deviation 656ms, period 614.18 sec). Worst all time event 4160ms.

After all tuning we were still seeing things like these and lots of timeouts on our Lua clients. The usage of the appliance was kind of high for 5 messages per second.

Kingston 16GB, 3-5 messages per second

We changed cards. This is SanDisk 16GB. This time we were able to run without no Lua timeouts nor any other errors in the error log. Still visible a high CPU utilization:

SanDisk 16GB, 3-5 messages per second

Final configuration

The final settings OS and Redis for Raspberry Pi B+ model with SanDisk 16 GB MicroSD:

Item	Status	Description
/etc/sysctl.conf	vm.swappiness=1 vm.overcommit_memory=1 vm.min_free_kbytes=16384	Virtual Memory Tuning
/etc/fstab	noatime	/dev/mmcblk0p2 root
redis.conf	port 0 unixsocket   kernel.sock unixsocketperm 755 appendfsync no no-appendfsync-on-rewrite yes	Redis configurations

Conclusions

Running on a RBPi board is cool since we can look even deeper how things could be improved and tuned for better performance and reduce utilization. We learned:

• Running Redis on MicroSD, SD cards is a bit complicated
• You need to measure what is slow, welcome Latency Doctor
• To measure where we waste time: Lua, Redis, both  ?
• To profile all our Lua scripts