Navigate / search

Installing Redis 2.4 (or latest) on dotCloud

UPDATE: Redis 2.4 is now supported as a dotCloud service, but this article applies to any custom version of Redis on dotCloud

I’ve been using dotCloud for about 6 months now and I think it’s a terrific service. Unfortunately, their services usually lag a version or 2 behind (no judgement, I’m sure it takes incredible manpower to keep so many services functioning, much less up-to-date). Someone over there did have the foresight to allow custom build scripts for almost any service you can think of though. It’s one of these custom scripts we’ll be creating today.

Version 0.8.0 of Nohm, an ORM for node.js and Redis, adds a lot of really nice features like built-in sorting (actually added in 0.7.0 I think) and Pub/Sub at the model-level.  I’m currently building the user dashboard for Netmarks, and both of these features were a perfectly timed godsend.  However, Nohm 0.8.0 requires Redis 2.4, and dotCloud’s Redis implementation is only version 2.2 at the time of this post.

This brings us to today, where I’ll go through how to install Redis 2.4 (or whatever the latest version is) on dotCloud and migrate your existing data. (the completed files are on Github if you don’t want to read the rest).  First, we’ll create the dotcloud.yml file necessary to create any service on dotCloud.

dotcloud.yml

redis:
  type: custom
  buildscript: build
  systempackages:
    - gcc
  ports:
    redis: tcp
  process: ~/redis-server ~/redis.conf

Simple, right? If you aren’t familiar with the dotCloud build file you should read this first, otherwise, to the breakdown!

Looks easy enough, but what the heck is going on here? Thanks so much for asking. I’ll tell you.

type: custom

This tells the dotCloud builder to use the custom script we’ll write in a minute to install and initialize the service, not a pre-packaged dotCloud service.

buildscript: build

The name of the BASH script we’ll use to install Redis on the dotCloud instance. This is a relative url, so be sure to provide the full path if it doesn’t reside in the same directory.

systempackages:
  - gcc

Redis requires GCC be installed. We’re just making sure the server we’re pushing to installs it if necessary.

ports:
  redis: tcp

This part actually took me a bit to figure out. What we’re asking dotCloud to do here is open a TCP port named PORT_REDIS in the environment files and assign the port number to an environment variable with the same name. As you’ll see later though, this is not the port on which we connect to Redis. It looks like dotCloud proxies requests from a public port (named “DOTCLOUD_DB_REDIS_PORT” if you copy my configuration) to PORT_REDIS. In the postinstall script below you’ll see we send that info as output to make it easier on the user.

process: ~/redis-server ~/redis.conf

Start Redis. The redis.conf referenced there is the next thing we’ll create. The build script moves all the relevant files to the home directory so they’re easily referenced and executed.

redis.conf

daemonize yes

save 900 1
save 300 10
save 60 10000

requirepass change_me

 * notice the extra line breaks at the bottom, they’re intentional

I’m not going to go into too much detail here.  In the Redis Quickstart Guide there’s a section titled “Installing Redis More Properly” which talks about managing processes and such. Since dotCloud handles that for us with supervisord the only thing we take away is adding daemonize yes to the config.

The next set of lines determine how often Redis writes the content in memory to the disk based on the number of transactions that occur. The Redis website explains all of this better than I ever could.

Lastly, we add a password for our database. It’s one line in the config and takes 2 seconds, please add a password. Especially if there’s the slightest possibility you’ll ever use it in production. Trust me, it’s a good idea.

Like I said, the Redis website and installation have documentation on the default config. If you want to get the most out of Redis, it’s seriously worth your time to read through it. Now, let me show you where the magic happens ;)

The build file

#!/bin/bash

wget http://download.redis.io/redis-stable.tar.gz
tar -xvf redis-stable.tar.gz
cd redis-stable
make

cd ..
cp -a . ~/

cd redis-stable/src
cp redis-server ~/
cp redis-cli ~/

If you’ve never installed Redis before and you skipped the Quickstart Guide above, now is the time to read it.

#!/bin/bash

We’re creating a bash script here. Safe bet that this will work on any Linux installation.

wget http://download.redis.io/redis-stable.tar.gz
tar -xvf redis-stable.tar.gz
cd redis-stable
make

Download, unzip, and make Redis. Pretty standard stuff here.

cd ..
cp -a . ~/

Navigate up to the temp directory that we’re installing from and copy all the files to the dotcloud user’s home directory. Not only are some of these files going to be necessary in the following steps, but it also helps to have access to these files if something goes wrong. dotCloud doesn’t like to let you out of your sandbox to poke around.

cd redis-stable/src
cp redis-server ~/
cp redis-cli ~/

Jump back into the installation directory and copy our executables into the home directory. The process: ~/redis-server ~/redis.conf in dotcloud.yml relies on these files being in the home directory.

postinstall

#!/bin/bash

#assign the port in the ENV vars to the redis.conf
echo port $PORT_REDIS >> ~/redis.conf

#output the info to the CLI user
echo '********************'
echo 'Use the following url to access this service:'
echo $DOTCLOUD_DB_REDIS_URL
echo '********************'

This is just a little script that runs after Redis has finished installing and performs 1 very important function: adding the port number assigned by dotCloud to the redis.conf.

echo port $PORT_REDIS >> ~/redis.conf

This ensures that Redis is listening on the correct port when the requests are proxied from the public port which we find in $DOTCLOUD_DB_REDIS_URL.
The output at the bottom of postinstall prints the host and port to the console so you, the user, can easily update your app without having to SSH into the dotCloud service and look it up. You’re welcome :)

I was going to go into migrating your data from an existing Redis installation on dotCloud, but this post is starting to get really long.  If you’re interested in migrating your data then feel free to shoot me an email or look into the Redis SAVE command to get you started.  Feel free to leave a comment below if you’re having trouble, and I’ll try to help out as much as I can.

UPDATE: I found this article, which is the best way I’ve seen to migrate your data from one Redis server to another.


If you enjoyed this post, please follow me on Twitter.

  • http://www.dotcloud.com Solomon Hykes

    Hey David, nice write-up! A few quick notes:

    - If you’re interested in seeing other examples of custom services, we have a whole bunch of them at http://github.com/dotcloud.

    - Don’t forget that custom services are still in beta, so the API might change in the future. But given the number of awesome custom services already built (including by us!), we’ll make sure any future changes are as painless as possible.

    - If you have any complaints or suggestions to make it easier to create custom services, we’d love to hear it!

    Happy hacking
    Solomon

  • Pingback: 3 Reasons Why You Should Build Something | David Dripps

  • http://gravatar.com/maaaritzzz maaaritzzz

    Hey, creator of nohm here.

    Glad to hear you like it, currently I’m working on an admin interface. But once the basic functionality for that is done, I’ll get back to getting nohm to v1.0.

    If you have any feature requests / issues / questions, feel free to ask. (here, github issues, mail/pm, irc #node.js mAritz) :)

    • http://www.facebook.com/daviddripps David Dripps

      Hey, it’s awesome to get a comment from you. Nohm’s doing everything I need it to right now. Seriously, thanks.
      I think an admin interface would be great. I haven’t found a nice GUI for interacting with Redis in any capacity. I’ll fork your nohm-admin repo and send some pull requests your way if I can contribute anything.
      Thanks again.