When managing a web server or an application, there are many situations that require synchronization between directories. While a number of tools can help you accomplish this, lsyncd is a strong candidate.
This article will discuss how to configure lsyncd to mirror changes between directories on a single machine and how to mirror between remote hosts.
For this guide, we will be using an Ubuntu 12.04 VPS, but any modern distribution should function in a similar way.
Fortunately, Ubuntu includes lsyncd in its default repositories.
We can install lsyncd with the following commands:
sudo apt-get update sudo apt-get install lsyncd
This will install lsync, but it will not provide you with a default configuration. We will create one later in the article.
While lsyncd does not provide a configuration file by default, it does include some examples that we can look at to get ideas.
See the examples by checking out the files in the "/usr/share/doc/lsyncd/examples" directory:
cd /usr/share/doc/lsyncd/examples ls
lbash.lua lgforce.lua lpostcmd.lua lrsyncssh.lua lecho.lua limagemagic.lua lrsync.lua
You can look at these text files to get a sense of what can be done with configuration.
We can check out one of the more basic configurations by opening the "lrsync.lua" file:
sudo nano
---- -- User configuration file for lsyncd. -- -- Simple example for default rsync. -- settings = { statusFile = "/tmp/lsyncd.stat", statusIntervall = 1, } sync{ default.rsync, source="src", target="trg", }
The lines beginning with two dashes (--) are comments. They are not interpreted by lsyncd.
The lsync configuration file is written in the Lua programming language. You can learn more about Lua here.
We will sync two local directories in our first example. Let's create them with the following commands:
sudo mkdir -p /source/to/copy sudo mkdir /dest
We will then add some files to the first directory so that we can check if the sync is operating correctly:
cd /source/to/copy sudo touch file{1..100}
The above commands create 100 files in the "/source/to/copy" directory.
Additionally, we can create a log directory and some files for lsyncd to use:
sudo mkdir /var/log/lsyncd touch /var/log/lsyncd/lsyncd.{log,status}
Next, we can create the lsyncd configuration directory:
sudo mkdir /etc/lsyncd
We will create a configuration file inside of this directory called "lsyncd.conf.lua", with nano:
sudo nano /etc/lsyncd/lsyncd.conf.lua
The configuration file we are making will sync the "/source/to/copy" directory to the "/dest" directory.
The general settings are all configured in a section called "settings". Our section will contain a few basic settings:
settings = { logfile = "/var/log/lsyncd/lsyncd.log", statusFile = "/var/log/lsyncd/lsyncd.status" }
These options tell lsyncd to use the files that we created earlier.
The next section specifies how you want the sync operations to be performed. Since this is a local transfer, we will use regular rsync to complete the transfer.
This configuration is defined by the "default.rsync" option. This configuration means that lsyncd will wait 20 seconds or collect 1000 separate sync events and then call rsync with the changes that are needed.
According to the lsyncd manual, the rsync instance that is called is equivalent to the command:
rsync -ltsd --delete --include-from=- --exclude=* SOURCE TARGET
This means that rsync operates in the following way:
The only thing we need to do to implement these changes is to specify the sync type and the source and target directories. The sync section will look like this:
sync { default.rsync, source = "/source/to/copy", target = "/dest" }
If you want to add options to rsync to modify its behavior, you can do so by passing the "rsyncOpts" variable an array containing comma-separated strings, each representing an rsync option:
sync { default.rsync, source = "/source/to/copy", target = "/dest", rsyncOpts = {"rsync option1", "rsync option2", "rsync option3"} }
We now have enough of a configuration to test our setup. Save and close the file.
Let's go to the target directory and verify that there are no files in the "/dest" directory at this point:
cd /dest ls
The last command should return no output because the "/dest" directory should be empty.
We can start the lsyncd service by issuing the following command:
sudo service lsyncd start
Check the directory again with ls:
ls
file1 file18 file27 file36 file45 file54 file63 file72 file81 file90 file10 file19 file28 file37 file46 file55 file64 file73 file82 file91 file100 file2 file29 file38 file47 file56 file65 file74 file83 file92 file11 file20 file3 file39 file48 file57 file66 file75 file84 file93 file12 file21 file30 file4 file49 file58 file67 file76 file85 file94 file13 file22 file31 file40 file5 file59 file68 file77 file86 file95 file14 file23 file32 file41 file50 file6 file69 file78 file87 file96 file15 file24 file33 file42 file51 file60 file7 file79 file88 file97 file16 file25 file34 file43 file52 file61 file70 file8 file89 file98 file17 file26 file35 file44 file53 file62 file71 file80 file9 file99
We can see that all of the files are instantly synced.
If we add some files to the source directory, we will experience the 20 second delay that we mentioned when discussing "default.rsync":
sudo mkdir /source/to/copy/hello{1..100} ls
We won't see the files initially, but after the timeout is reached, rsync will sync the files. Check again by reissuing the "ls" command.
With a few changes to our configuration file, we can configure remote syncing.
First we need to be able to sign into the mirroring machine from the original machine through password-less ssh.
If you have not already done so, generate ssh keys on your machine with lsyncd by following this guide.
Make a key pair for your root user, since the lsyncd service will run as root. You can then copy the key file to the remote mirror machine with the following commands:
sudo su
ssh-copy-id remote_mirror_ip_address
Your key file should now allow you to log into the remote mirroring server as the root user.
Try it now so to test that it works and to create a destination directory on the remote host:
ssh remote_mirror_ip_address
We will create a directory called "/remotesync" to act as our target directory:
mkdir /remotesync
Exit out of the remote session and the local root session by typing "exit" twice:
exit exit
We can open the lsyncd configuration file again with the following command:
sudo nano /etc/lsyncd/lsyncd.conf.lua
settings = { logfile = "/var/log/lsyncd/lsyncd.log", statusFile = "/var/log/lsyncd/lsyncd.status" } sync { default.rsync, source = "/source/to/copy", target = "/dest" }
We will only need to make changes within the "sync" section.
We will change "default.rsync" to "default.rsyncssh" to enable rsync over ssh, and we will replace the "target" variable with the "host" and "targetdir" variables:
settings = {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync {
default.rsyncssh,
source = "/source/to/copy",
host = "remote_mirror_ip_address",
targetdir = "/remotesync"
}
Save the file and exit.
We can now restart the lsyncd service by issuing the following command:
sudo service lsyncd restart
If we ssh into our remote mirror, we should be able to see the changes in the remote "/remotesync" directory:
sudo su
ssh remote_mirror_ip_address
ls /remotesync
You should see all of the files that we added to the local "/source/to/copy" directory.
The lsyncd service is a great way to synchronize files across directories or systems. Due to its Lua-based configuration files, it can be very powerful and flexible. The Lua documentation (linked earlier), and the lsync documentation are great resources that will allow you to develop more complex syncing operations.
Check out some of the other examples in "/usr/share/doc/lsyncd/examples" directory for more ideas.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
nifty, thanks!
Thanks for this. One thing I ran into using lsyncd version 2.1.5-1 that you might want to update:
Yes, in Lsyncd 2.1.5 and later, the configuration syntax has been updated, and using
settings = { ... }
is deprecated. Now, the correct syntax is to usesettings{ ... }
without the equal sign.Here’s how it should look in your configuration file:
Using
settings{ ... }
instead ofsettings = { ... }
will prevent warnings and ensure compatibility with newer versions.Oh really cool.
Is there any command that i can check the sync status?
By default, Lsyncd logs its activities, which can be useful for monitoring sync status. Check the log file (usually at
/var/log/lsyncd/lsyncd.log
):This command will display real-time updates in the log, showing which files are being synced.
pgrep
to Confirm Lsyncd is Running: Confirm Lsyncd is actively running with thepgrep
command:This will show if Lsyncd is running as a process.
You can see active Lsyncd processes and their sync status by running:
If you configured Lsyncd with a status file, you can check its sync status there.
Regards
cd /usr/share/doc/lsyncd*/examples
How do I start this using systemctl ?
Heya,
You can run the following command to start Lsyncd using your existing configuration:
Enable Lsyncd to Start at Boot: To ensure that Lsyncd starts automatically when the system boots, use:
Check the Status: To verify that Lsyncd is running without issues, check its status:
View Logs: If you want to monitor the log output for any messages or errors, check the log file specified in your configuration, usually found at
/var/log/lsyncd.log
:Restarting or Reloading: If you make any changes to your configuration file, restart Lsyncd to apply those changes:
Stop the Service: If you ever need to stop Lsyncd, you can do so with:
Regards
Is there any way to do bi directional syncing in case we have something like a load balanced WordPress setup?
Yes, you can achieve bidirectional syncing in a load-balanced WordPress setup, but it requires careful planning to avoid conflicts and ensure data consistency. Here are a few approaches you might consider:
Centralized Database: Have all WordPress instances connect to a single database, ensuring all changes are reflected across nodes.
File Synchronization Tools:
Distributed File Systems: Use solutions like GlusterFS or Ceph for shared storage and automatic data replication.
Shared Storage Solutions: NFS or cloud storage (e.g., AWS S3) can centralize media uploads.
WordPress Multisite: Consider this for managing multiple sites with a shared installation and database.
Caching: Implement caching (e.g., Redis) to improve performance across instances.
Choose the method that best fits your needs and ensure you have a strategy for resolving conflicts.
Regards
thanks for the post before starting implementation I have some points I’d like to clarify
2.if I generate key pairs on localhost then copy to the remote host Do I have to update the key for all others client ?
Heya,
Yes, you can proceed with the implementation even if
PermitRootLogin
is set to false. This setting is a good security practice as it prevents direct root login over SSH. Instead, you can log in as a regular user who has sudo privileges to perform any necessary operations.When you generate SSH key pairs on your localhost and copy the public key to the remote host, you only need to update the key on the remote host if you want to use that key for new clients.
If you have multiple clients (e.g., different machines) that need to connect to the same remote host, you’ll need to copy the public key from each client to the remote host’s
~/.ssh/authorized_keys
file. If the key changes (for example, if you generate a new key pair), you will need to update theauthorized_keys
file again.Regards