How To Migrate Linux Servers Part 2 - Transfer Core Data

Introduction

There are many scenarios in which you might have to move your data and operating requirements from one server to another. You may need to implement your solutions in a new datacenter, upgrade to a larger machine, or transition to new hardware or a new VPS provider.

Whatever your reasons, there are many different considerations you should make when migrating from one system to another. Getting functionally equivalent configurations can be difficult if you are not working with a configuration management solution such as Chef, Puppet, or Ansible. You need to not only transfer data, but also configure your services to operate in the same way on a new machine.

In the last article, you prepped your server for data migration. At this point, your target and source system should be able to communicate — the target system should have SSH access to the source system. You should also have a list of software and services that you need to transfer, including version numbers. In this guide, you’ll continue where you left off and begin the actual migration to your new server.

Step 1 – Creating a Migration Script

This tutorial and the next one will be focused on creating and adding to a migration bash shell script as you go along. You will be using a number of low-level Linux tools, and rather than try to run them all interactively, your goal should be to end up with a reproducible set of steps that can capture the relevant parts of your server configuration.

As you write this script, you should be able to run it iteratively as you go along. Most of the tools used in this tutorial, such as rsync, will only transfer data if it has been changed since the last run, so that you can safely repeat commands without needing to worry about making them redundant. Because you configured SSH to enable connecting to the original (source) machine from the new (target) server, you should be working from the target server throughout this tutorial.

You can create this script in your home directory, using nano or your favorite text editor:

nano ~/sync.sh

On the first line of the file, add a script heading, also known as a shebang. This tells the script which interpreter to run with by default. #!/bin/bash means the script will default to the bash shell, which is the most powerful and widely-supported shell available on most systems.

#!/bin/bash

Save and close the file for now. If you are using nano, press Ctrl+X, then when prompted, Y and then Enter.

Back on the command line, make the script executable by using chmod:

chmod 700 ~/sync.sh

For an in-depth overview of how chmod and Linux permissions work, you can refer to An Introduction to Linux Permissions.

After having made the script executable and adding the shebang, you can run it by calling it directly:

~/sync.sh

It will not produce any output yet, as the script is empty. You should test the script regularly through the rest of this tutorial as needed. As in the prior tutorial in this series, you may need to run it with sudo permissions, depending on the steps that you add to the script.

Step 2 – Installing Needed Programs and Services

The first step you’ll add to your migration script will be to restore the packages that you marked for migration in the previous tutorial.

Add Additional Repositories

Before doing that, you’ll want to connect to your original (source) server again in a separate terminal, to check whether you’ve installed software from any third-party repositories. If so, you won’t be able to reinstall those packages in your new environment without first configuring those additional package sources.

In Ubuntu and Debian environments, you can see if alternative repositories are present on your source system by investigating a few locations:

cat /etc/apt/sources.list

Output
…
## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu impish partner
# deb-src http://archive.canonical.com/ubuntu impish partner

deb http://security.ubuntu.com/ubuntu impish-security main restricted
# deb-src http://security.ubuntu.com/ubuntu impish-security main restricted
deb http://security.ubuntu.com/ubuntu impish-security universe
# deb-src http://security.ubuntu.com/ubuntu impish-security universe
deb http://security.ubuntu.com/ubuntu impish-security multiverse
# deb-src http://security.ubuntu.com/ubuntu impish-security multiverse

This is the main package source list — because it’s a single file, you can use cat to output its contents. If the last line of the file contains a ubuntu.com address, then you probably haven’t added any third-party repositories to this file. Additional repositories can also be listed in the sources.list.d directory:

ls /etc/apt/sources.list.d

Output
droplet-agent.list  elastic-7.x.list  nodesource.list

If this directory is not empty, you can cat the individual files to check each of the repositories:

cat /etc/apt/sources.list.d/elastic-7.x.list

Output
deb https://artifacts.elastic.co/packages/7.x/apt stable main

This will tell you the URL of the repository that you’ll need to re-add to your target machine. In most cases, you can do that with the add-apt-repository command:

sudo add-apt-repository repo_url

On RHEL, Rocky, or Fedora Linux, you can instead use dnf to list the repositories configured for the server:

dnf repolist enabled

You can then add additional repositories to your target system by using dnf config-manager:

sudo dnf config-manager --add-repo repo_url

If you make any changes to your source list, add them as comments at the top of your migration script back on your target system. This way, if you have to start from a fresh install, you will know what procedures need to happen before attempting a new migration.

nano ~/sync.sh

#!/bin/bash

#############
# Prep Steps
#############

# Add additional repositories to /etc/apt/source.list
#       deb http://example.repo.com/linux/deb stable main non-free

Then, save and close the file.

Specifying Version Constraints and Installing

You now have your package sources updated on your target machine to match your source machine.

On Ubuntu or Debian machines, you can now install the versions of the software that you need on your target machine by typing:

sudo apt update
sudo apt install package_name=version_number

If the version of the package you are trying to match is more than a few months old, it may have been removed from the official repositories. In this case, you could try to hunt down the older version of the .deb packages (for example, by browsing older upstream repositories, or third-party PPAs) and their dependencies and install them manually with:

sudo dpkg -i package.deb

However, you should do this very sparingly to avoid creating a situation where you have too many packages with version mismatches. If older versions of software are not readily available, test the newest available releases first to see if they still meet your needs, to avoid imposing out-of-date requirements.

For RHEL, Rocky, or Fedora Linux, you can install specific versions of software by typing:

Sudo dnf install package_name-version_number

If you need to hunt down rpm files that have been removed from the repository in favor of newer versions, you can install them with dnf :

dnf install package_name.rpm

Again, keep track of what operations you are performing here. You can include them as comments in the script you are creating:

nano ~/sync.sh

#!/bin/bash

#############
# Prep Steps
#############

# Add additional repositories to /etc/apt/source.list
#       deb http://example.repo.com/linux/deb stable main non-free

# Install necessary software and versions
#       apt-get update
#       apt-get install apache2=2.2.22-1ubuntu1.4 mysql-server=5.5.35-0ubuntu0.12.04.2 libapache2-mod-auth-mysql=4.3.9-13ubuntu3 php5-mysql=5.3.10-1ubuntu3.9 php5=5.3.10-1ubuntu3.9 libapache2-mod-php5=5.3.10-1ubuntu3.9 php5-mcrypt=5.3.5-0ubuntu1

Again, save and close the file.

Step 3 – Start Transferring Data

The actual transfer of data is usually not the most labor intensive part of the migration, but it can be the most time-intensive. If you are migrating a server with a lot of data, it is probably a good idea to start transferring data sooner rather than later.

Rsync is a powerful tool that provides a wide array of options for replicating files and directories across many different environments, with built-in checksum validation and other features. Identify any directories whose data you want to transfer, and add rsync commands to your migration script.

A sample rsync command looks like this:

rsync -azvP --progress source_server:/path/to/directory/to/transfer /path/to/local/directory

-azvP is a typical set of Rsync options. As a breakdown of what each of those do:

• a enables “Archive Mode” for this copy operation, which preserves file modification times, owners, and so on. It is also the equivalent of providing each of the -rlptgoD options individually (yes, really). Notably, the -r option tells Rsync to recurse into subdirectories to copy nested files and folders as well. This option is common to many other copy operations, such as cp and scp.
• z compresses data during the transfer itself, if possible. This is useful for any transfers over slow connections, especially when transferring data that compresses very effectively, like logs and other text.
• v enables verbose mode, so you can read more details of your transfer while it is in progress.
• P tells Rsync to retain partial copies of any files that do not transfer completely, so that transfers can be resumed later.

You can find out more about how to create appropriate rsync commands by reading this article. In some cases, you may have to create the parent directories leading up to your target destination prior to running rsync .

With the addition of rsync commands, your sync script might now look like this:

#!/bin/bash

#############
# Prep Steps
#############

# Add additional repositories to /etc/apt/source.list
#       deb http://example.repo.com/linux/deb stable main non-free

# Install necessary software and versions
#       apt-get update
#       apt-get install apache2=2.2.22-1ubuntu1.4 mysql-server=5.5.35-0ubuntu0.12.04.2 libapache2-mod-auth-mysql=4.3.9-13ubuntu3 php5-mysql=5.3.10-1ubuntu3.9 php5=5.3.10-1ubuntu3.9 libapache2-mod-php5=5.3.10-1ubuntu3.9 php5-mcrypt=5.3.5-0ubuntu1

#############
# File Transfer
#############


# Rsync web root
rsync -azvP --progress source_server:/var/www/site1 /var/www/

# Rsync home directories
. . .

Remember that these commands can be re-run and will not transfer any new data unless the source files have changed, so you can add to this script incrementally as you are testing it. Be conservative and iterative about which directories you include.

Migrating Databases and other Non-File Data

Note that you cannot necessarily copy all of your data using rsync without any additional prep. Many applications such as databases store their relevant data across multiple actual “files” in your filesystem, to optimize access using techniques like Database Sharding. These files are generally not meant to be accessed or copied as-is; a database exposes data through a query interface instead.

Fortunately, almost all applications that implement their own storage will include some mechanism of exporting and importing data into ordinary files, so that they can be copied as normal during migrations like this. For example, if you are using MySQL, you can review How to Import and Export Databases. You can then transfer these exports across servers using rsync or scp.

Step 4 – Modifying Configuration Files

Although some software will resume working gracefully after transferring the relevant configuration details and data from the original server, many configurations will need to be modified.

This presents a slight problem for the syncing script. If you run the script to sync your data, and then modify the values to reflect the correct information for its new destination, these changes will be wiped out the next time you run the script. To solve this problem, you can add additional steps to the script which will modify that data in place after transferring it.

Linux includes a number of core utilities that are very useful for this kind of text scripting. Two of these are sed and awk. In general, sed is more straightforward to use if you are making modifications to unstructured text using regular expressions, and awk is more useful for more complex parsing of formatted text or tabular data. Beyond this tutorial, you can also learn more about using sed, or learn more about using awk.

This way, your sync script can perform sed or awk commands immediately after rsync, so that your files are automatically modified as needed after being transferred.

sed syntax looks like this:

sed -i 's/string_to_match/string_to_replace_it_with/g' file_to_edit

The -i flag means that the file will be modified in place rather than creating a separate output file. The s and g do not change and are a regular sed convention. You can also use regular expressions within the string_to_match. Try adding a sed command to your sync.sh:

rsync -avz --progress source_server:/etc/mysql/* /etc/mysql/

# Change socket to '/mysqld/mysqld.sock'
sed -i 's/\/var\/run\/mysqld\/mysqld.sock/\/mysqld\/mysqld.sock/g' /etc/mysql/my.cnf

This will change every instance of /var/run/mysqld/mysqld.sock in /etc/mysql/my.cnf to /mysqld/mysqld.sock/g. The \ character is used to precede the / characters because they would otherwise be parsed as the end of your sed expression. This is known as escaping special characters. Make sure that your sed commands come after the rsync commands.

You can use awk for formatted text the same way you used sed for unstructured text. For instance, the /etc/shadow file is divided into tabs delimited by the colon (:) character, which look like this:

vault:!:18941::::::
stunnel4:!:18968:0:99999:7:::
sammy:$6$bscTWIVxvy.KhkO8$NJNhpABhJoybG/vDiRzQ2y9OFEd6XtqgCUG4qkuWfld97VEXH8/jUtc7oMtOC34V47VE7HjdpMMv37Aftdb7C/:18981:0:99999:7:::

You could use awk to remove the data from the second “column” (i.e., between the first and second : character), like so:

awk 'BEGIN { OFS=FS=":"; } $1=="root" { $2=""; } { print; }' /etc/shadow > shadow.tmp && mv shadow.tmp /etc/shadow

This command is telling awk that both the input and the output delimiter should be parsed as :. It then specifies that if column 1 is equal to “root”, then column 2 should be set to an empty string. Unlike sed, awk doesn’t directly support editing files in place, so this script performs equivalent steps of writing to a temporary file then using mv to overwrite the original input with the temporary file.

You can always add comments to your migration script (on lines preceded by #) to document in-progress fixes or changes to your files.

Conclusion

You should now have all the information you need to migrate your application environments and your data to your new server. You should also have good, reproducible documentation for this process should you ever need to redeploy your stack onto a new system.

In the final tutorial in this series, you’ll review how to transfer and test any lingering system services on your new server.