Creating a distributed Redis system using Docker

A common problem I face on a daily basis is a lack of hardware / resource in order to test things out to the fullest. For example, in days gone by I’d have needed 3 servers for what i’m about to do – and in more recent times, 3 virtual machines. I dont have the time to continuously build these items, nor the resource if we were going physical. This is where my new found interest in Docker can help me out!

What I want to do on my Ubuntu ‘host’ server is create 3 Docker containers running Redis, and link them all together so that I can then develop and test the best way to monitor h-scaled Redis. Below I will show you how i’ve done it, and the benefits (even beauty) of it!

For my base images, I use the excellent ‘phusion’ image which adds in a lot features that Docker omits (by design or not), including ordered startup of services and more. For more information on the phusion-base image, click here.

First, lets pull the phusion image as below, using Docker:

root@server:/home/sam# docker pull phusion/baseimage

On completion, you should be able to run ‘docker’ images and see the new image as below:

root@server:/home/sam# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
phusion/baseimage 0.9.15 cf39b476aeec 3 months ago 289.4 MB
phusion/baseimage latest cf39b476aeec 3 months ago 289.4 MB

This image is great as you can tell it easily to startup certain services on start of the container, which is a bit of a pain in the arse otherwise (in my other blog i outlined how when using a standard base image you need to use bash to essentially start things!). One of the things I wanted to ADD to this image is the ability to login via SSH using a password instead of keys; given this is only going to be running on a locally contained box im not too worried around security!

To do this on phusion, we need to create an instance of phusion, login via SSH and modify the config, then restart SSH. We will then be able to login using the root user.

First, create a new container from the phusion base image:

root@server:/home/sam# docker run -d --name redis phusion/baseimage /sbin/my_init --enable-insecure-key

The ‘/sbin/my_init’ is what allows phusion to start your services on boot of the container. The ‘enable-insecure-key’ allows us to ssh into the new container using an ‘insecure key’ (see below).

Now that the container has been deployed, we need to get its IP address. To do this, use ‘docker inspect’ as below:

root@server:/home/sam# docker inspect node1 | grep IPA
"IPAddress": "172.17.0.46",

Next, lets pull down the ‘insecure key’ and using it to login to our new container:

root@server:/home/sam# curl -o insecure_key -fSL https://github.com/phusion/baseimage
root@server:/home/sam# docker/raw/master/image/insecure_key
root@server:/home/sam# chmod 600 insecure_key
root@server:/home/sam# ssh -i insecure_key [email protected]
The authenticity of host '172.17.0.52 (172.17.0.52)' can't be established.
ECDSA key fingerprint is aa:bb:cc:xx:xx:xx:xx:xx:xx:xx:xx:yy:zz:04:bf:04.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.52' (ECDSA) to the list of known hosts.
root@c36b4bba7dd4:~#

Congratulations, you are now ssh’d into the container! Next we need to modify SSH so we dont need to use the insecure key in the future. To do this, open up /etc/ssh/sshd_config and find and uncomment the following line:

PermitRootLogin yes

Then save the file and exit the text editor. Finally, we need to give our root user a password. To do this run ‘passwd’ as below:

root@c36b4bba7dd4:~# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
root@c36b4bba7dd4:~#

And thats that for SSH on our base image. Next we will install redis on the container.

Now that we have our base image configured, we will need to download and install Redis. For my example I am using the latest version of redis, which can be downloaded from the page here. For my example i will be using Redis 3.0.0 RC1 from this link – https://github.com/antirez/redis/archive/3.0.0-rc1.tar.gz .

Note: You will need to install wget, gcc, make and a few other tools – these are properly base images 🙂

First we will need to download the Redis 3.0.0 RC1:

root@e3919192d9e3:/home# wget https://github.com/antirez/redis/archive/3.0.0-rc1.tar.gz

Next, lets unpack it and make it:

root@e3919192d9e3:/home# tar -zxvf 3.0.0-rc1.tar.gz
root@e3919192d9e3:/home# cd redis-3.0.0-rc1/
root@e3919192d9e3:/home/redis-3.0.0-rc1# make

Finally, move the binaries into /usr/bin/ so we can access them system-wide (easier!):

root@e3919192d9e3:/home/redis-3.0.0-rc1# cd src
root@e3919192d9e3:/home/redis-3.0.0-rc1/src# mv redis-cli redis-server redis-sentinel /usr/bin/
root@e3919192d9e3:/home/redis-3.0.0-rc1/src# cd ..
root@e3919192d9e3:/home/redis-3.0.0-rc1# mkdir -p /etc/redis/
root@e3919192d9e3:/home/redis-3.0.0-rc1# cp redis.conf /etc/redis/redis.conf

Now open up /etc/redis/redis.conf, find the line ‘daemonize no’ and change it to ‘daemonize yes’. Next, lets start it up!

root@e3919192d9e3:/home/redis-3.0.0-rc1/src# redis-server /etc/redis/redis.conf
root@e3919192d9e3:/home/redis-3.0.0-rc1/src#

And thats that – Redis is now installed and running on your container, using the config file at /etc/redis/redis.conf!

Now that our container is running redis and has SSH access, we need to tell it to start redis automatically on ‘start’ of the container.

To do this using the phusion base image, its actually remarkably simple. First, because we are starting docker using /sbin/my_init it will run anything we put in /etc/service/* on start which is excellent. SO, for our situation we need to create a new folder and ‘run’ file for docker here, as below:

root@e3919192d9e3:/etc/service# cd /etc/service
root@e3919192d9e3:/etc/service# mkdir redis
root@e3919192d9e3:/etc/service# cd redis
root@e3919192d9e3:/etc/service/redis# nano run

Within run, we need to paste the following:

#!/bin/sh
set -e
exec /usr/bin/redis-server /etc/redis/redis.conf

Finally, remember to set the run file to be executable:

root@e3919192d9e3:/etc/service/redis# chmod +x run

And thats all we need to do! Now, lets commit this image to our local library so we can start deploying it en-mass!

Now that our ‘redis’ container is working a treat, we want to save it as a pseudo ‘template’ so we can deploy it over and over again on Docker – VERY quickly.

To do this is remarkably simple: we just ‘docker commit ..’ the image to our local library, as below:

root@server:/home/sam# docker ps -as
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
e3919192d9e3 phusion/baseimage:0.9.15 "/sbin/my_init --ena 3 hours ago Up 3 hours redis 164.9 MB
root@server:/home/sam# docker commit redis redis-cluster-node
root@server:/home/sam# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
redis-cluster-node latest babfb02edf4d 5 hours ago 561.2 MB

There we have it – our image is saved locally. Next we can deploy it over and over to ours hearts content, as below.

For claritys sake, I want to delete my ‘build’ container now as it is confusing to have around in the future. To do this first ‘docker stop redis’ and ‘docker rm redis’ as below:

root@server:/home/sam# docker stop redis
redis
root@server:/home/sam# docker rm redis
redis
root@server:/home/sam#

Simples. Next, lets deploy 3 instances of our template. I like to port forward using docker so i can access it using my ‘host servers’ IP. So for 3 instances, I am going to associate:

• node1 with hostip:7001
• node2 with hostip:7002
• node3 with hostip:7003

So that i can hit ‘redis-cli -h 192.168.0.2 -p 7001’ and it will redirect to ‘172.17.0.x -p 6379’, for example. Below, I have deployed 3 instances from my image:

root@server:/home/sam# docker run -d --name node1 -p 7001:6379 redis-cluster-node /sbin/my_init
cd1c1f96346bdf9c1cec04333c2e849992ecbc4375dcea6b30902dd9842d8c99
root@server:/home/sam# docker run -d --name node2 -p 7002:6379 redis-cluster-node /sbin/my_init
cd1c1f96346bdf9c1cec04333c2e849992ecbc4375dcea6b30902dd9842d8c99
root@server:/home/sam# docker run -d --name node3 -p 7003:6379 redis-cluster-node /sbin/my_init
cd1c1f96346bdf9c1cec04333c2e849992ecbc4375dcea6b30902dd9842d8c99
root@server:/home/sam#

Now I can do ‘docker ps -as’ and view all 3 of my new, shiny Redis containers:

root@server:/home/sam# docker ps -as
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
a532b4ac60d9 redis-cluster-node:latest "/sbin/my_init" 5 hours ago Up 5 hours 0.0.0.0:7003->6379/tcp node3 2.267 MB
6c8a87a0a76a redis-cluster-node:latest "/sbin/my_init" 5 hours ago Up 5 hours 0.0.0.0:7002->6379/tcp node2 2.318 MB
39e02633ccf8 redis-cluster-node:latest "/sbin/my_init" 5 hours ago Up 5 hours 0.0.0.0:7001->6379/tcp node1 2.334 MB
root@server:/home/sam#

Here we can see 3 redis containers, named node1, node2 and node3 – all with a dedicated ‘7001-7003’ port that maps straight to the Redis servers port.

To test this is working, we can use ‘redis-cli’ on another server and try and login to the ‘host server’ using the port 7001 and the host servers 192.168.x.x IP, as below:

root@server:/home/sam# redis-cli -h 192.168.0.2 -p 7001
redis 192.168.0.16:7001>
root@server:/home/sam# redis-cli -h 192.168.0.2 -p 7002
redis 192.168.0.16:7002>
root@server:/home/sam# redis-cli -h 192.168.0.2 -p 7003
redis 192.168.0.16:7003>
root@server:/home/sam# redis-cli -h 192.168.0.2 -p 7005
Could not connect to Redis at 192.168.0.2:7005: Connection refused
not connected>

As you can see, I can connect through to the 3 instances using their port + the host servers IP (all docker IPs’ are on 172.17.0.0/24 range). When i try an incorrect port it fails (just to prove that ALL ports dont end up in a redis server!!).

Now we have 3 individual redis servers, we want to link them together so that we can test scalability/monitoring of clusters or whatever our reason is for setting this all up! 🙂

We will need to configure node2 and node3 to be slaves of node1 which will be our master; we can do this easily by modifying /etc/redis/redis.conf.

To do this we will take advantage of our SSH configuration earlier – simply find out the IP address of the containers for node2 and node3, SSH into them, modify the config and then restart the container, as below:

root@server:/home/sam# docker inspect node1 | grep IPA
"IPAddress": "172.17.0.46",
root@server:/home/sam# docker inspect node2 | grep IPA
"IPAddress": "172.17.0.47",
root@server:/home/sam# ssh [email protected]
[email protected]'s password:
Last login: Tue Jan 13 11:47:31 2015 from 172.17.42.1
root@6c8a87a0a76a:~# nano /etc/redis/redis.conf

In this config file we need to find the line ‘slaveof’ and uncomment and modify it to the IP address of node1 – similar to below:

root@6c8a87a0a76a:~# cat /etc/redis/redis.conf | grep slaveof
# Master-Slave replication. Use slaveof to make a Redis instance a copy of
slaveof 172.17.0.46 6379
root@6c8a87a0a76a:~#

Finally, restart the container using ‘docker restart node2’ (for example), and it should now be a slave of the master Redis server on node1. To verify this, use redis-cli to login to node1 and node2 on seperate terminals and use ‘set hello world’ on node1, and ‘get hello’ on node2. If it works it should look like the following:

root@server:/home/sam# redis-cli -h 192.168.0.16 -p 7001
redis 192.168.0.12:7001> set hello world
OK
redis 192.168.0.12:7001> exit
root@server:/home/sam# redis-cli -h 192.168.0.16 -p 7002
redis 192.168.0.12:7002> get hello
"world"
redis 192.168.0.12:7002> exit
root@server:/home/sam#

And there you have it! 3 redis servers setup in a h-scale fashion (well, using slaves!) in docker – using your own phusion-based image. Marvellous!

In my next blog, I will show you how to monitor your Redis cluster using Opsview so that you can get a nice pretty dashboard for your Redis stats, as below: