How To Migrate Redis Data with Replication on Ubuntu 18.04

Introduction

Redis is an in-memory, key-value data store known for its flexibility, performance, wide language support, and built-in features like replication. Replication is the practice of regularly copying data from one database to another in order to have a replica that always remains an exact duplicate of the primary instance. One common use of Redis replication is to migrate an existing Redis data store to a new server, as one might do when scaling up their infrastructure for better performance.

This tutorial outlines the process of using Redis’s built-in replication features to migrate data from one Ubuntu 18.04 server (the “source”) to another (the “target”). This involves making a few configuration changes to each server, setting the target server to function as a replica of the source, and then promoting the replica back to a primary after the migration is completed.

Prerequisites

To complete this tutorial, you will need:

• Two servers running Ubuntu 18.04. Each server should have a user configured with administrative privileges and a firewall set up with ufw. To set up this environment, follow our initial server setup guide for Ubuntu 18.04 for both servers.
• The latest version of Redis installed on each server. To set this up, follow our guide on How To Install Redis from Source on Ubuntu 18.04.

Step 1 — (Optional) Loading Your Source Redis Instance with Sample Data

This optional step involves loading your source Redis instance with some sample data so you can experiment with migrating data to your target instance. If you already have data that you want to migrate over to your target, you can move ahead to Step 2 which will go over how to back it up.

To begin, connect to the Ubuntu server you’ll use as your source Redis instance as your non-root user:

ssh sammy@source_server_ip

Then run the following command to access your Redis server:

redis-cli

If you’ve configured your Redis server to require password authentication, run the auth command followed by your Redis password:

auth source_redis_password

Next, run the following commands. These will create a number of keys holding a few strings, a hash, a list, and a set:

mset string1 "Redis" string2 "is" string3 "fun!"
hmset hash1 field1 "Redis" field2 "is" field3 "fast!"
rpush list1 "Redis" "is" "feature-rich!"
sadd set1 "Redis" "is" "free!"

Additionally, run the following expire commands to provide a few of these keys with a timeout. This will make them volatile, meaning that Redis will delete them after a specified amount of time (7500 seconds, in this case):

expire string2 7500
expire hash1 7500
expire set1 7500

With that, you have some example data you can export to your target Redis instance. Keep the redis-cli prompt open for now, since we will run a few more commands from it in the next step to back this data up.

Step 2 — Backing Up Your Source Redis Instance

Any time you plan to move data from one server to another, there’s a risk that something could go wrong and you could lose data as a result. Even though this risk is small, we will use Redis’s bgsave command to create a backup of your source Redis database in case you encounter an error during the replication process.

If you don’t already have it open, start by opening up the Redis command line interface:

redis-cli

Also, if you’ve configured your Redis server to require password authentication, run the auth command followed by your Redis password:

auth password

Next, run the bgsave command. This will create a snapshot of your current data set and export it to a dump file held in Redis’s working directory:

bgsave

Following that, you can close the connection to your Redis instance by running the exit command:

exit

If you need it in the future, you can find the data dump file in your Redis instance’s working directory. Recall how in the prerequisite Redis installation tutorial you set your Redis instance to use /var/lib/redis as its working directory.

List the contents of your Redis working directory to confirm that it’s holding the data dump file:

sudo ls /var/lib/redis

If the dump file was exported correctly, you will see it in this command’s output. By default, this file is named dump.rdb:

Output
dump.rdb

After confirming that your data was backed up correctly, you’re all set to configure your source Redis server to accept external connections and allow for replication.

Step 3 — Configuring Your Source Redis Instance

By default, Redis isn’t configured to listen for external connections, meaning that any replicas you configure won’t be able to sync with your source instance unless you update its configuration. Here, we will update the source instance’s configuration file to allow for external connections and also set a password which the target instance will use to authenticate once replication begins. After that, we’ll add a firewall rule to allow connections to the port on which Redis is running.

Open up your source Redis instance’s configuration file with your preferred text editor. Here, we’ll use nano:

sudo nano /etc/redis/redis.conf

Navigate to the line that begins with the bind directive. It will look like this by default:

. . .
bind 127.0.0.1
. . .

This directive binds Redis to 127.0.0.1, an IPv4 loopback address that represents localhost. This means that this Redis instance is configured to only listen for connections that originate from the same server as the one where it’s installed. To allow your source instance to accept any connection made to its public IP address, such as those made from your target instance, add your source Redis server’s IP address after the 127.0.0.1. Note that you shouldn’t include any commas after 127.0.0.1:

. . .
bind 127.0.0.1 source_server_IP
. . .

Next, if you haven’t already done so, use the requirepass directive to configure a password which users must enter before they can interact with the data on the source instance. Do so by uncommenting the directive and setting it to a complex password or passphrase:

. . .
requirepass source_redis_password
. . .

Be sure to take note of the password you set here, as you will need it when you configure the target server.

Following that change, you can save and close the Redis configuration file. If you edited it with nano, do so by pressing CTRL+X, Y, then ENTER.

Then, restart the Redis service to put these changes into effect:

sudo systemctl restart redis

That’s all you need to do in terms of configuring Redis, but if you configured a firewall on your server it will continue to block any attempts by your target server to connect with the source. Assuming you configured your firewall with ufw, you could update it to allow connections to the port on which Redis is running with the following command. Note that Redis is configured to use port 6379 by default:

sudo ufw allow 6379

After making that final change you’re all done configuring your source Redis server. Continue on to configure your target Redis instance to function as a replica of the source.

Step 4 — Configuring your Target Redis Instance

By this point you’ve configured your source Redis instance to accept external connections. However, because you’ve locked down access to the source by uncommenting the requirepass directive, your target instance won’t be able to replicate the data held on the source. Here, you will configure your target Redis instance to be able to authenticate its connection to the source, thereby allowing replication.

Begin by connecting to your target Redis server as your non-root user:

ssh sammy@target_server_ip

Next, open up your target server’s Redis configuration file:

sudo nano /etc/redis/redis.conf

If you haven’t done so already, you should configure a password for your target Redis instance with the requirepass directive:

. . .
requirepass target_redis_password
. . .

Next, uncomment the masterauth directive and set it to your source Redis instance’s authentication password. By doing this, your target server will be able to authenticate to the source instance after you enable replication:

. . .
masterauth source_redis_password
. . .

Lastly, if you have clients writing information to your source instance, you will want to configure them to write data to your target instance as well. This way, if a client writes any data after you promote the target back to being a primary instance, it won’t get lost.

To do this, though, you will need to adjust the replica-read-only directive. This is set to yes by default, which means that it’s configured to become a “read-only” replica which clients won’t be able to write to. Set this directive to no to allow clients to write to it:

. . .
replica-read-only no
. . .

Those are all the changes you need to make to the target’s configuration file, so you can save and close it.

Then, restart the Redis service to put these changes into effect:

sudo systemctl restart redis

After restarting the Redis service your target server will be ready to become a replica of the source. All you’ll need to do to turn it into one is to run a single command, which we’ll do shortly.

Step 5 — Starting and Verifying Replication

By this point, you have configured your source Redis instance to accept connections from your target server and you’ve configured your target Redis instance to be able to authenticate to the source as a replica. With these pieces in place, you’re ready to turn your target instance into a replica of the source.

Begin by opening up the Redis command line interface on your target Redis server:

redis-cli

Run the auth command to authenticate the connection:

auth password

Next, turn the target instance into a replica of the source with the replicaof command. Be sure to replace source_server_ip with your source instance’s public IP address and source_port with the port used by Redis on your source instance:

replicaof source_server_ip source_port

From the prompt, run the following scan command. This will return all the keys currently held by the replica:

scan 0

If replication is working as expected, you will see all the keys from your source instance held in the replica. If you loaded your source with the sample data in Step 1, the scan command’s output will look like this:

Output
1) "0"
2) 1) "string3"
   2) "string1"
   3) "set1"
   4) "string2"
   5) "hash1"
   6) "list1"

While you have the connection open, you can also confirm that the keys you set to expire are still volatile. Do so by running the ttl command with one of these keys as an argument:

ttl hash1

This will return the number of seconds before this key will be deleted:

Output
5430

Once you’ve confirmed that the data on your source instance was correctly synced to your target, you can promote the target back to being a primary instance by running the replicaof command once again. This time, however, instead of following replicaof with an IP address and port, follow it with no one. This will cause the target instance to stop syncing with the source immediately:

replicaof no one

To confirm that the data replicated from the source persist on the target, rerun the scan command you entered previously:

scan 0

You should see the same keys in this command’s output as when you ran the scan command when the target was still replicating the source:

Output
1) "0"
2) 1) "string3"
   2) "string1"
   3) "set1"
   4) "string2"
   5) "hash1"
   6) "list1"

With that, you’ve successfully migrated all the data from your source Redis instance to your target. If you have any clients that are still writing data to the source instance, now would be a good time to configure them to only write to the target.

Conclusion

There are several methods besides replication you can use to migrate data from one Redis instance to another, but replication has the advantages of requiring relatively few configuration changes to work and only a single command to initiate or stop.

If you’d like to learn more about working with Redis, we encourage you to check out our tutorial series on How To Manage a Redis Database. Also, if you want to move your Redis data to a Redis instance managed by DigitalOcean, follow our guide on how to do so.