Rsync, which stands for remote sync, is a remote and local file synchronization tool. It uses an algorithm to minimize the amount of data copied by only moving the portions of files that have changed.
In this tutorial, we’ll define Rsync, review the syntax when using rsync
, explain how to use Rsync to sync with a remote system, and other options available to you.
Deploy your frontend applications from GitHub using DigitalOcean App Platform. Let DigitalOcean focus on scaling your app.
In order to practice using rsync
to sync files between a local and remote system, you will need two machines to act as your local computer and your remote machine, respectively. These two machines could be virtual private servers, virtual machines, containers, or personal computers as long as they’ve been properly configured.
If you plan to follow this guide using servers, it would be prudent to set them up with administrative users and to configure a firewall on each of them. To set up these servers, follow our Initial Server Setup Guide.
Regardless of what types of machines you use to follow this tutorial, you will need to have created SSH keys on both of them. Then, copy each server’s public key to the other server’s authorized_keys
file as outlined in Step 2 of that guide.
This guide was validated on machines running Ubuntu 20.04, although it should generally work with any computers running a Linux-based operating system that have rsync
installed.
Rsync is a very flexible network-enabled syncing tool. Due to its ubiquity on Linux and Unix-like systems and its popularity as a tool for system scripts, it’s included on most Linux distributions by default.
The syntax for rsync
operates similar to other tools, such as ssh
, scp
, and cp
.
First, change into your home directory by running the following command:
Then create a test directory:
Create another test directory:
Now add some test files:
There’s now a directory called dir1
with 100 empty files in it. Confirm by listing out the files:
Outputfile1 file18 file27 file36 file45 file54 file63 file72 file81 file90
file10 file19 file28 file37 file46 file55 file64 file73 file82 file91
file100 file2 file29 file38 file47 file56 file65 file74 file83 file92
file11 file20 file3 file39 file48 file57 file66 file75 file84 file93
file12 file21 file30 file4 file49 file58 file67 file76 file85 file94
file13 file22 file31 file40 file5 file59 file68 file77 file86 file95
file14 file23 file32 file41 file50 file6 file69 file78 file87 file96
file15 file24 file33 file42 file51 file60 file7 file79 file88 file97
file16 file25 file34 file43 file52 file61 file70 file8 file89 file98
file17 file26 file35 file44 file53 file62 file71 file80 file9 file99
You also have an empty directory called dir2
. To sync the contents of dir1
to dir2
on the same system, you will run rsync
and use the -r
flag, which stands for “recursive” and is necessary for directory syncing:
Another option is to use the -a
flag, which is a combination flag and stands for “archive”. This flag syncs recursively and preserves symbolic links, special and device files, modification times, groups, owners, and permissions. It’s more commonly used than -r
and is the recommended flag to use. Run the same command as the previous example, this time using the -a
flag:
Please note that there is a trailing slash (/
) at the end of the first argument in the syntax of the the previous two commands and highlighted here:
This trailing slash signifies the contents of dir1
. Without the trailing slash, dir1
, including the directory, would be placed within dir2
. The outcome would create a hierarchy like the following:
~/dir2/dir1/[files]
Another tip is to double-check your arguments before executing an rsync
command. Rsync provides a method for doing this by passing the -n
or --dry-run
options. The -v
flag, which means “verbose”, is also necessary to get the appropriate output. You’ll combine the a
, n
, and v
flags in the following command:
Outputsending incremental file list
./
file1
file10
file100
file11
file12
file13
file14
file15
file16
file17
file18
. . .
Now compare that output to the one you receive when removing the trailing slash, as in the following:
Outputsending incremental file list
dir1/
dir1/file1
dir1/file10
dir1/file100
dir1/file11
dir1/file12
dir1/file13
dir1/file14
dir1/file15
dir1/file16
dir1/file17
dir1/file18
. . .
This output now demonstrates that the directory itself was transferred, rather than only the files within the directory.
To use rsync
to sync with a remote system, you only need SSH access configured between your local and remote machines, as well as rsync
installed on both systems. Once you have SSH access verified between the two machines, you can sync the dir1
folder from the previous section to a remote machine by using the following syntax. Please note in this case, that you want to transfer the actual directory, so you’ll omit the trailing slash:
This process is called a push operation because it “pushes” a directory from the local system to a remote system. The opposite operation is pull, and is used to sync a remote directory to the local system. If the dir1
directory were on the remote system instead of your local system, the syntax would be the following:
Like cp
and similar tools, the source is always the first argument, and the destination is always the second.
Rsync provides many options for altering the default behavior of the utility, such as the flag options you learned about in the previous section.
If you’re transferring files that have not already been compressed, like text files, you can reduce the network transfer by adding compression with the -z
option:
The -P
flag is also helpful. It combines the flags --progress
and --partial
. This first flag provides a progress bar for the transfers, and the second flag allows you to resume interrupted transfers:
Outputsending incremental file list
created directory destination
source/
source/file1
0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=99/101)
sourcefile10
0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=98/101)
source/file100
0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=97/101)
source/file11
0 100% 0.00kB/s 0:00:00 (xfr#4, to-chk=96/101)
source/file12
0 100% 0.00kB/s 0:00:00 (xfr#5, to-chk=95/101)
. . .
If you run the command again, you’ll receive a shortened output since no changes have been made. This illustrates Rsync’s ability to use modification times to determine if changes have been made:
Outputsending incremental file list
sent 818 bytes received 12 bytes 1660.00 bytes/sec
total size is 0 speedup is 0.00
Say you were to update the modification time on some of the files with a command like the following:
Then, if you were to run rsync
with -azP
again, you’ll notice in the output how Rsync intelligently re-copies only the changed files:
Outputsending incremental file list
file1
0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=99/101)
file10
0 100% 0.00kB/s 0:00:00 (xfer#2, to-check=98/101)
file2
0 100% 0.00kB/s 0:00:00 (xfer#3, to-check=87/101)
file3
0 100% 0.00kB/s 0:00:00 (xfer#4, to-check=76/101)
. . .
In order to keep two directories truly in sync, it’s necessary to delete files from the destination directory if they are removed from the source. By default, rsync
does not delete anything from the destination directory.
You can change this behavior with the --delete
option. Before using this option, you can use -n
, the --dry-run
option, to perform a test to prevent unwanted data loss:
If you prefer to exclude certain files or directories located inside a directory you are syncing, you can do so by specifying them in a comma-separated list following the --exclude=
option:
If you have a specified pattern to exclude, you can override that exclusion for files that match a different pattern by using the --include=
option:
Finally, Rsync’s --backup
option can be used to store backups of important files. It’s used in conjunction with the --backup-dir
option, which specifies the directory where the backup files should be stored:
Rsync can streamline file transfers over networked connections and add robustness to local directory syncing. The flexibility of Rsync makes it a good option for many different file-level operations.
A mastery of Rsync allows you to design complex backup operations and obtain fine-grained control over how and what is transferred.
Need highly available block storage? Attach secure, scalable NVMe- and SSD-based Volumes Block Storage to your DigitalOcean virtual machine in seconds. We’ll make sure your data is reliably stored and secure.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Is it possible to backup multiple folders are once over SSH? Locally, this command works fine:
rsync -av /{folder1, folder2} /backup
But when run over ssh, it fails. Any ideas? The paths are correct and work when run individually. I am on OS X if that matters.
Yes, you can back up multiple folders at once over SSH using
rsync
, but you need to handle it slightly differently than running it locally. The syntax and approach vary a bit when dealing with remote servers, especially when you need to include multiple source directories.To back up multiple directories to a remote server using
rsync
, you can list each directory separately and specify the remote destination. Here’s the correct approach:-a
stands for archive mode (preserves permissions, timestamps, etc.).-v
is for verbose output.-z
enables compression.-e "ssh -p port_number"
specifies the SSH command and port if not using the default port 22.Regards
@TheUnknown: “But when run over ssh” - what command do you try to run exactly?
Both of these fail:
rsync -av root@server.com:/{folder1, folder2} /backup rsync -av -e ssh root@server.com:/{folder1, folder2} /backup
Realized I put spaces in my examples.
This works: rsync -av /{folder1,folder2} /backup
These fail: rsync -av root@server.com:/{folder1,folder2} /backup rsync -av -e ssh root@server.com:/{folder1,folder2} /backup
Oh. Glad it works now! :]
I am still not able to transfer multiple folders over SSH, I just made a typo in my example. Sorry if I was not clear.
To transfer multiple folders to a remote server, list each directory in the
rsync
command. Make sure to use the correct syntax forrsync
.Example Command:
Explanation:
-a
stands for archive mode (preserves permissions, timestamps, etc.).-v
enables verbose output.-z
enables compression during transfer.-e "ssh -p port_number"
specifies the SSH command and port if you’re using a non-default port.Assuming you have directories
projectA
andprojectB
in/home/user
and want to transfer them to/var/www
on the remote serverexample.com
, here’s how you’d do it:Command:
bashCopy code
rsync -avz -e "ssh -p 22" /home/user/projectA /home/user/projectB username@example.com:/var/www/
This command will transfer both
projectA
andprojectB
to the/var/www/
directory on the remote server.If you have a list of directories, first create
directories.txt
:projectA projectB
Then use:
This will sync the directories listed in
directories.txt
from/home/user
on the local machine to/var/www/
on the remote server.Regards
Is this a permanent connection so if file 1 changes on server 1 then its sent to directory2 on server 2? Or is it temporary and must be done manually?
Is it similar to lncron? Is one better to have change automatically updated on remote host?
Thanks.
No, rsync is not a permanent connection. It’s like a one-time command that syncs the files from one directory to another(can be remote-local or the other way around as well). Meaning that if the file changes, the rsync command won’t update the file automatically.
There are services like syncthing that can help you with that:
https://syncthing.net/
@phillip: rsync only runs when you run the command. It doesn’t automatically sync both servers, take a look at <a href=“https://code.google.com/p/lsyncd/”>lsyncd</a>.
Most useful post Kamal. Thanks a lot !!
My Apologies Justin. I meant you not Kamal. Thanks for this most informative post.
This comment has been deleted
A nice tip when you need to send files through a specific SSH port:
nice :)
This Should be added to the main Article - was a critical point for me Thanks for sharing
Thanks ,
That is a great catch.
I usually use
rsync
like this:-a
stands for archive mode, which preserves permissions and recursively copies directories.-v
stands for verbose, to provide detailed output.-z
enables compression during transfer.-e "ssh -p port_number"
specifies the SSH port.Regards
I’m having a problem backing up some various configuration files from my droplet to my local machine. This problem is not specific to rsync, but that is my tool of choice for this purpose, so I’m asking here.
The problem is a permissions issue. The files I need to copy usually require elevated privileges, things like /etc/dovecot/*, which are owned by root. So,
fails, because my user has insufficient privileges. In an ssh session, I would simply use sudo to elevate my privileges. Also,
does not work, because I’ve disabled root login on my droplet.
What is the best way to accomplish this? So far I’ve managed to alarm my droplet’s firewall to the point where it blocked my own ip.
In situations like this you can use Sudo as mentioned, that would be the best way to go about it.
In case you need or want to skip Sudo asking you for permissions, you’ll need to set a suduers file and configure commands that the said user can execute with sudo without providing a password.
I’ve written this mini-tutorial here:
https://www.digitalocean.com/community/questions/a-deep-dive-into-the-sudoers-file
It should help in that situation.
Dear Justin:
Using the --delete option, I would like to “store” (keep) the deleted files in a different directory I choose. How can I get it?
mm
Using the
--delete
option withrsync
typically removes files from the destination directory that no longer exist in the source directory. If you want to move deleted files to a different directory rather than just deleting them,rsync
does not directly support this functionality. However, you can achieve this with a combination ofrsync
and other tools.You can sync the directories using
rsync
with the--delete
option to identify which files will be deleted.Example Command:
-a
for archive mode.-v
for verbose output.--delete
to delete files in the destination that are no longer in the source.--log-file=/path/to/rsync.log
to log the files being processed.You can then perform the
rsync
operation with the--delete
option, knowing that deleted files have been backed up.Example Sync Command:
Regards
@tech: You can use the
--backup
and--backup-dir
arguments to do that.Example:
Make sure
--backup-dir
is set to a directory that is outside of the tree that is being synced (source
).I’m looking to do a complete “clone” of one droplet to another, for the purpose of moving from a larger droplet to a smaller droplet. That is, copy the entire root to root.
Can someone advise me on this command? Thanks in advance!
rsync -avP /* root@xxx.xxx.xxx.xxx:/
This is not really a recommended way of going about it. You can use the upgrade process rather than going through this complex process which will possibly fail anyway.
You can follow this tutorial on how to upgrade your Droplet:
https://docs.digitalocean.com/products/droplets/how-to/resize/
@bcabalic: Try using the following command:
This should make sure the permissions are properly set and that
/dev
,/proc
, and/sys
aren’t transferred.Another great Digital Ocean tutorial. Thanks a lot @jellingwood! The only thing I have missed is the mysterious
rsyncd
orrsync --daemon
which initially seemed to me something the keeps running all the time watching for live file changes and it would be great to link to a folder. Could you please help me on clarifying what is thatrsyncd
really does?I saw that @kamaln7 pointed to lsyncd which seemed pretty neat. However I still got that big question mark over my head when speaking about
rsyncd
. :)Thanks again
PS: The way I currently work with live syncing is with fsmonitor npm with the following command:
This is a interesting topic.
rsyncd
is for running an rsync server that clients can connect to for file synchronization. It’s useful for setting up centralized file syncing services.lsyncd
is for real-time synchronization between directories on local or remote systems, providing continuous sync based on file system changes.If your need is for real-time synchronization or continuous sync between directories,
lsyncd
is a great choice. For setting up anrsync
server to provide access to files over the network,rsyncd
is suitable.Regards
What do i have do run if both servers has the same public key and the password auth is disabled ?
If both servers have the same public key and password authentication is not available, one option is to copy your SSH private key to each computer to allow password-less authentication. This would be placed in the hidden
~/.ssh
directory in your user’s home directory.Another option is to create an additional SSH key pair for use between these two hosts. You can create a key pair on one of the hosts and copy the files over to the other. You can then add the host to the
authorized_hosts
file within the~/.ssh
directory to allow authentication. You can have as many keys as you would like in this file, so you wouldn’t need to remove your local authentication information.A third option, that I personally find quite elegant, is to use an SSH agent on your local computer. OpenSSH (the SSH implementation available on almost every Linux distribution) comes with an SSH agent by default. You can add your private key to the agent, and then forward it to the host you are connecting to, allowing you to use your home credentials from within the host you are connected to.
The procedure for doing this starts on your local computer. First, you must start up your SSH agent by typing:
This will start the agent process and put it into the background. Next, you need to add your SSH private key to the agent. This will save the key in memory for the duration of the terminal session:
Finally, when you connect to your first server, add the
-A
flag. This allows you to use your local SSH key pair to authenticate through the server you are connecting to.This will connect you to your host, as usual. The difference is that you can now hop to from the first host to another host that your SSH key pair has access to, without having to provide any additional authentication information:
This should solve the problem of authenticating with your second host.
If both servers have the same public key and password authentication is disabled, you should be able to connect between them using SSH key-based authentication.
You need to ensure that the ssh-key pair is correct, also the same goes for the keys permissions and ownership.
Regards
I think the statement on excluding multiple files/comments by listing them separated with comma is dead wrong… doesn’t work… here is the right way Examples to Exclude Multiple Files and Directories using exclude-from (segment 6)
so use something like
rsync -az --delete --exclude=.git --exclude=data source destination
Hi dear all. Is there a way to backup files on different folder to different folder. For example I need to backup following directories to another directories. rsync -avz /usr/local/squidGuard 192.168.1.7::website rsync -avz /etc/squid 192.168.1.7::website Above commands backups files inside directory to /etc/squid . I need rsync to backup /usr/local/squidGuard into /usr/local/squidGuard on remote server. And /etc/squid into /etc/squid on remote server. What should I do? Thanks in advanced!
Hello guys, I need to know how could I have everything logged in a log file (deleted files, updated file and created files). The log file should be timestamped. Anyone can help me ?
rsync
has built-in logging capabilities using its verbose (-v
) and other flags. To capture detailed logs, you can redirect output to a log file. For including timestamps, you’ll need to manage that separately.Basic
rsync
Command with Logging:Explanation:
-a
: Archive mode (preserves permissions, timestamps, etc.).-v
: Verbose mode (shows detailed output).--delete
: Deletes files in the destination that are no longer in the source.>> /path/to/rsync.log
: Appends the output to the specified log file.2>&1
: Redirects both stdout and stderr to the log file.Regards
help!! my terminal does not recognise new folders, files ever: my mac is new.
sl1$ ls dir1 ls: dir1: No such file or directory
so i never really got to the rsync part, which i need to keep my skipped files.
any ideas would be of great help
If you’re unsure where the directory might be, you can search for it:
This command will search the entire filesystem for
dir1
. The2>/dev/null
part suppresses permission-denied error messages.Sometimes the Finder and Terminal may not immediately reflect new changes. Try refreshing or restarting them.
Command + R
to refresh Finder.Regards
Thanks for the detailed answers! Here’s something I’d like to share. I am trying to rsync files and directories which are bound with permissions for a non-root user to be the only one allowed to read it (600 perms) and root will not sync those files, as the NFS mounts won’t allow root to get the job done. I need to preserve these permissions, so here was my workaround - I used sudo to become the user, and perform the copy for me.
sudo -u www-data rsync -a /old/path/ /new/path
( no closing / here on last one…) I will probably add rsync to the crontab for user www-data, but I needed to sync 100GB of info from the command line, to prove it all worked first… In retrospect, a simple ‘at’ command would have sufficed.
You’re right in using
sudo -u
to runrsync
as a non-root user (www-data
in your case) to handle files with restrictive permissions. This allows you to preserve permissions correctly while syncing files between directories.When syncing files with
rsync
across NFS mounts, you might encounter issues with root permissions due to NFS’s root squashing feature. Usingsudo -u
bypasses these issues by executingrsync
as a user with appropriate permissions.Regards
Very good technical writing! Thank you very much for it.
Hi! Is rsync daemon is necessary on remote machine while using SSH? I have no problem while sync on local VPS but when I try to push folder from Ubuntu VPS to remote Windows machine running Bitwise SSH Server, rsync says:
rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.0]
Meanwhile I’m able to ssh and sftp on that Windows machine. The problem only with rsync.
Heya,
No, rsync is not actually a service so it doesn’t have a daemon. It’s just a command that executes.
The error you are receiving could be due to a couple of reasons.
Nice, but what solution do you provide for windows users? I tryed with putty, but I´m not able to install the putty tools on my ubuntu 14.04 droplet.
What about having a master directory that syncs (overwrites) multiple “slave” directories? Would Rsync work for that? I have a repository that I pull newest versions to a main directory, I have multiple apps that use the same updates but on different directories on the VPS… I’d like my master directory to write all updates to the other app directories… is this possible?
Yes, you can use
rsync
to synchronize a master directory with multiple “slave” directories on a VPS. The master directory can be set up to automatically overwrite or update the slave directories with the latest changes.To sync a master directory with a slave directory, you can use a simple
rsync
command. If you have a master directory and want to sync it with several slave directories, you can runrsync
multiple times, once for each slave directory.Regards
I have to sync one entire server to another. Here i have created a image of live server installed it another one in which i have only changed the IP. but both the systems are same application & configurations also. only the mac & ip are different. here my task is that i have to do live sync after each 5 minute. The server includes LAMP . my question is here how will i do with this with --delete option which means whenever i sync directories from local to remote if the files under source isn’t available which is in destination then that should be deleted in destination. so our live sync will be success. Thanks in Advance . Waiting for you reply…
I wouldn’t recommend using rsync to do so. There are other smarter ways to go about this.
First, what you can use is Snapshots:
https://docs.digitalocean.com/products/images/snapshots/
and more importantly this:
https://docs.digitalocean.com/products/images/snapshots/how-to/create-and-restore-droplets/#:~:text=Restore a Snapshot on an,snapshots available in your account.
Additionally, the LAMP data is easier recreated than running rsync and making it work. Installing Apache, MySQL and PHP is way easier. If you just rsynced them over your system wouldn’t recognise them and you’ll have one big mess on your hands.
Greate guide, thanks!
Thank you for this post however,
I am trying to automate this process with a shell script and I am having troubles on the enter password part. I need the password part to be automated too so I read up that creating public and private keys would be the best option and then install the public key on the host that I am trying to grab files from that is the one in question asking for the user password when I run the command.
Unfortunately, I am still unsuccessful as it still asks me for a password when I run my script. I even tried ‘ssh ip-address’ and it still prompts me for a password. Any ideas? Other work arounds? I need to automate this process to grab the data/output directory from a MotionEye OS on a Raspberry Pi 3. It also needs to be secure and this seemed like the best idea however I need a little more help please!
Thank you, Appletatoes
That would be the case if you are running the commands with
sudo
. It will always ask for the password of the user.In case you need or want to skip Sudo asking you for permissions, you’ll need to set a suduers file and configure commands that the said user can execute with sudo without providing a password.
I’ve written this mini-tutorial here:
https://www.digitalocean.com/community/questions/a-deep-dive-into-the-sudoers-file
It should help in that situation.
rync is very slow for big directories.
I have a directory containing hundreds of subdirectories and 1300GB of user uploaded files. How do I keep a backup of this directory? rsync over ssh takes several hours to examine each file to determine if it needs to be sync’d or not.
Heya,
It’s not necessarly that it’s slow, it’s the number of files. Before rsync runs, it makes like a list of everything it needs to copy over and since you have a lot of files, it takes time. After that the download/upload speed will be determined by your network connection.
Now, regarding backups, I’ll recommend using a backup solution like these 2:
https://docs.digitalocean.com/products/images/snapshots/
Snapshots are on-demand disk images of DigitalOcean Droplets and volumes saved to your account. Use them to create new Droplets and volumes with the same contents.
and
https://docs.digitalocean.com/products/images/backups/
Backups are automatically-created disk images of Droplets. Enabling backups for Droplets enables system-level backups at weekly intervals, which provides a way to revert to an older state or create new Droplets.