Tutorial

How To Mirror Local and Remote Directories on a VPS with lsyncd

Published on August 30, 2013
How To Mirror Local and Remote Directories on a VPS with lsyncd

What is lsyncd?

When managing a web server or an application, there are many situations that require synchronization between directories. While a number of tools can help you accomplish this, lsyncd is a strong candidate.

This article will discuss how to configure lsyncd to mirror changes between directories on a single machine and how to mirror between remote hosts.

For this guide, we will be using an Ubuntu 12.04 VPS, but any modern distribution should function in a similar way.

How to Install lsyncd

Fortunately, Ubuntu includes lsyncd in its default repositories.

We can install lsyncd with the following commands:

sudo apt-get update
sudo apt-get install lsyncd

This will install lsync, but it will not provide you with a default configuration. We will create one later in the article.

How to Find lsyncd Configuration Examples

While lsyncd does not provide a configuration file by default, it does include some examples that we can look at to get ideas.

See the examples by checking out the files in the "/usr/share/doc/lsyncd/examples" directory:

cd /usr/share/doc/lsyncd/examples
ls
lbash.lua  lgforce.lua      lpostcmd.lua  lrsyncssh.lua
lecho.lua  limagemagic.lua  lrsync.lua

You can look at these text files to get a sense of what can be done with configuration.

We can check out one of the more basic configurations by opening the "lrsync.lua" file:

sudo nano 
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync.
--
settings = {
        statusFile = "/tmp/lsyncd.stat",
        statusIntervall = 1,
}

sync{
        default.rsync,
        source="src",
        target="trg",
}

The lines beginning with two dashes (--) are comments. They are not interpreted by lsyncd.

The lsync configuration file is written in the Lua programming language. You can learn more about Lua here.

Setting Up the Environment

We will sync two local directories in our first example. Let's create them with the following commands:

sudo mkdir -p /source/to/copy
sudo mkdir /dest

We will then add some files to the first directory so that we can check if the sync is operating correctly:

cd /source/to/copy
sudo touch file{1..100}

The above commands create 100 files in the "/source/to/copy" directory.

Additionally, we can create a log directory and some files for lsyncd to use:

sudo mkdir /var/log/lsyncd
touch /var/log/lsyncd/lsyncd.{log,status}

Next, we can create the lsyncd configuration directory:

sudo mkdir /etc/lsyncd

We will create a configuration file inside of this directory called "lsyncd.conf.lua", with nano:

sudo nano /etc/lsyncd/lsyncd.conf.lua

How to Sync Two Local Folders with lsyncd

The configuration file we are making will sync the "/source/to/copy" directory to the "/dest" directory.

Configuring the Global Section

The general settings are all configured in a section called "settings". Our section will contain a few basic settings:

settings = {
	logfile = "/var/log/lsyncd/lsyncd.log",
	statusFile = "/var/log/lsyncd/lsyncd.status"
}

These options tell lsyncd to use the files that we created earlier.

Configuring the Sync Section

The next section specifies how you want the sync operations to be performed. Since this is a local transfer, we will use regular rsync to complete the transfer.

This configuration is defined by the "default.rsync" option. This configuration means that lsyncd will wait 20 seconds or collect 1000 separate sync events and then call rsync with the changes that are needed.

According to the lsyncd manual, the rsync instance that is called is equivalent to the command:

rsync -ltsd --delete --include-from=- --exclude=* SOURCE TARGET

This means that rsync operates in the following way:

  • -l: copy symbolic links
  • -t: copy modification time
  • -s: no space-splitting; wildcard characters only
  • -d: transfer directories without recursing

The only thing we need to do to implement these changes is to specify the sync type and the source and target directories. The sync section will look like this:

sync {
	default.rsync,
	source = "/source/to/copy",
	target = "/dest"
}

If you want to add options to rsync to modify its behavior, you can do so by passing the "rsyncOpts" variable an array containing comma-separated strings, each representing an rsync option:

sync {
	default.rsync,
	source = "/source/to/copy",
	target = "/dest",
	rsyncOpts = {"rsync option1", "rsync option2", "rsync option3"}
}

We now have enough of a configuration to test our setup. Save and close the file.

Testing Local Syncing

Let's go to the target directory and verify that there are no files in the "/dest" directory at this point:

cd /dest
ls

The last command should return no output because the "/dest" directory should be empty.

We can start the lsyncd service by issuing the following command:

sudo service lsyncd start

Check the directory again with ls:

ls
file1    file18  file27  file36  file45  file54  file63  file72  file81  file90
file10   file19  file28  file37  file46  file55  file64  file73  file82  file91
file100  file2   file29  file38  file47  file56  file65  file74  file83  file92
file11   file20  file3   file39  file48  file57  file66  file75  file84  file93
file12   file21  file30  file4   file49  file58  file67  file76  file85  file94
file13   file22  file31  file40  file5   file59  file68  file77  file86  file95
file14   file23  file32  file41  file50  file6   file69  file78  file87  file96
file15   file24  file33  file42  file51  file60  file7   file79  file88  file97
file16   file25  file34  file43  file52  file61  file70  file8   file89  file98
file17   file26  file35  file44  file53  file62  file71  file80  file9   file99

We can see that all of the files are instantly synced.

If we add some files to the source directory, we will experience the 20 second delay that we mentioned when discussing "default.rsync":

sudo mkdir /source/to/copy/hello{1..100}
ls

We won't see the files initially, but after the timeout is reached, rsync will sync the files. Check again by reissuing the "ls" command.

How to Configure Remote Syncing with lsyncd

With a few changes to our configuration file, we can configure remote syncing.

First we need to be able to sign into the mirroring machine from the original machine through password-less ssh.

How to Log Into the Remote Machine with SSH Keys

If you have not already done so, generate ssh keys on your machine with lsyncd by following this guide.

Make a key pair for your root user, since the lsyncd service will run as root. You can then copy the key file to the remote mirror machine with the following commands:

sudo su
ssh-copy-id remote_mirror_ip_address

Your key file should now allow you to log into the remote mirroring server as the root user.

Try it now so to test that it works and to create a destination directory on the remote host:

ssh remote_mirror_ip_address

We will create a directory called "/remotesync" to act as our target directory:

mkdir /remotesync

Exit out of the remote session and the local root session by typing "exit" twice:

exit
exit

How to Configure lsyncd to Mirror Remotely

We can open the lsyncd configuration file again with the following command:

sudo nano /etc/lsyncd/lsyncd.conf.lua
settings = {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
        default.rsync,
        source = "/source/to/copy",
        target = "/dest"
}

We will only need to make changes within the "sync" section.

We will change "default.rsync" to "default.rsyncssh" to enable rsync over ssh, and we will replace the "target" variable with the "host" and "targetdir" variables:

settings = {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
        default.rsyncssh,
        source = "/source/to/copy",
        host = "remote_mirror_ip_address",
        targetdir = "/remotesync"
}

Save the file and exit.

Testing Remote Syncing

We can now restart the lsyncd service by issuing the following command:

sudo service lsyncd restart

If we ssh into our remote mirror, we should be able to see the changes in the remote "/remotesync" directory:

sudo su
ssh remote_mirror_ip_address
ls /remotesync

You should see all of the files that we added to the local "/source/to/copy" directory.

Taking it Further

The lsyncd service is a great way to synchronize files across directories or systems. Due to its Lua-based configuration files, it can be very powerful and flexible. The Lua documentation (linked earlier), and the lsync documentation are great resources that will allow you to develop more complex syncing operations.

Check out some of the other examples in "/usr/share/doc/lsyncd/examples" directory for more ideas.

By Justin Ellingwood

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products

About the author(s)

Justin Ellingwood
Justin Ellingwood
See author profile
Category:
Tutorial

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
8 Comments
Leave a comment...

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

nifty, thanks!

Thanks for this. One thing I ran into using lsyncd version 2.1.5-1 that you might want to update:

  • Restarting synchronization daemon lsyncd Warn: settings = { … } is deprecated. please use settings{ … } (without the equal sign)
alexdo
Site Moderator
Site Moderator badge
October 30, 2024

Yes, in Lsyncd 2.1.5 and later, the configuration syntax has been updated, and using settings = { ... } is deprecated. Now, the correct syntax is to use settings{ ... } without the equal sign.

Here’s how it should look in your configuration file:

settings {
   logfile = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd.status",
   inotifyMode = "CloseWrite",
}

Using settings{ ... } instead of settings = { ... } will prevent warnings and ensure compatibility with newer versions.

Oh really cool.

Is there any command that i can check the sync status?

alexdo
Site Moderator
Site Moderator badge
October 30, 2024

By default, Lsyncd logs its activities, which can be useful for monitoring sync status. Check the log file (usually at /var/log/lsyncd/lsyncd.log):

tail -f /var/log/lsyncd/lsyncd.log

This command will display real-time updates in the log, showing which files are being synced.

  • Use pgrep to Confirm Lsyncd is Running: Confirm Lsyncd is actively running with the pgrep command:
pgrep -fl lsyncd

This will show if Lsyncd is running as a process.

You can see active Lsyncd processes and their sync status by running:

ps aux | grep lsyncd

If you configured Lsyncd with a status file, you can check its sync status there.

Regards

cd /usr/share/doc/lsyncd*/examples

How do I start this using systemctl ?

alexdo
Site Moderator
Site Moderator badge
October 28, 2024

Heya,

You can run the following command to start Lsyncd using your existing configuration:

sudo systemctl start lsyncd

Enable Lsyncd to Start at Boot: To ensure that Lsyncd starts automatically when the system boots, use:

sudo systemctl enable lsyncd

Check the Status: To verify that Lsyncd is running without issues, check its status:

sudo systemctl status lsyncd

View Logs: If you want to monitor the log output for any messages or errors, check the log file specified in your configuration, usually found at /var/log/lsyncd.log:

tail -f /var/log/lsyncd.log

Restarting or Reloading: If you make any changes to your configuration file, restart Lsyncd to apply those changes:

sudo systemctl restart lsyncd

Stop the Service: If you ever need to stop Lsyncd, you can do so with:

sudo systemctl stop lsyncd

Regards

Is there any way to do bi directional syncing in case we have something like a load balanced WordPress setup?

alexdo
Site Moderator
Site Moderator badge
October 28, 2024

Yes, you can achieve bidirectional syncing in a load-balanced WordPress setup, but it requires careful planning to avoid conflicts and ensure data consistency. Here are a few approaches you might consider:

  1. Centralized Database: Have all WordPress instances connect to a single database, ensuring all changes are reflected across nodes.

  2. File Synchronization Tools:

  • Lsyncd: Configure it for bidirectional syncing but implement conflict resolution.
  • Unison: A tool that supports bidirectional syncing, with proper conflict handling.
  1. Distributed File Systems: Use solutions like GlusterFS or Ceph for shared storage and automatic data replication.

  2. Shared Storage Solutions: NFS or cloud storage (e.g., AWS S3) can centralize media uploads.

  3. WordPress Multisite: Consider this for managing multiple sites with a shared installation and database.

  4. Caching: Implement caching (e.g., Redis) to improve performance across instances.

Choose the method that best fits your needs and ensure you have a strategy for resolving conflicts.

Regards

thanks for the post before starting implementation I have some points I’d like to clarify

  1. my ssh config has permit root login set to false can I proceed ?

2.if I generate key pairs on localhost then copy to the remote host Do I have to update the key for all others client ?

alexdo
Site Moderator
Site Moderator badge
October 28, 2024

Heya,

Yes, you can proceed with the implementation even if PermitRootLogin is set to false. This setting is a good security practice as it prevents direct root login over SSH. Instead, you can log in as a regular user who has sudo privileges to perform any necessary operations.

When you generate SSH key pairs on your localhost and copy the public key to the remote host, you only need to update the key on the remote host if you want to use that key for new clients.

If you have multiple clients (e.g., different machines) that need to connect to the same remote host, you’ll need to copy the public key from each client to the remote host’s ~/.ssh/authorized_keys file. If the key changes (for example, if you generate a new key pair), you will need to update the authorized_keys file again.

Regards

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Become a contributor for community

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

DigitalOcean Documentation

Full documentation for every DigitalOcean product.

Resources for startups and SMBs

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

Get our newsletter

Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

New accounts only. By submitting your email you agree to our Privacy Policy

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.