How To Sandbox Processes With Systemd On Ubuntu 20.04

The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

Introduction

Sandboxing is a computer security technique that focuses on isolating a program or process from parts of a system that it does not need to interact with during normal operation. When a new program is started it has all of the abilities of the user that it runs as. These abilities are very often much more than the program needs to perform its function. This can lead to security issues when a bad actor manipulates the program to access some of its unused abilities to do something the program would not normally do.

The purpose of sandboxing is to identify exactly what abilities and resources a program needs, and then block off everything else.

The system management suite of tools systemd is used on almost all major Linux distributions to start, stop, and manage programs and processes. It has many sandboxing options that restrict how the process it starts accesses the host system, making it more secure.

The aim of this tutorial is not to create the strictest sandbox environment possible, but rather to use the recommended and easily enabled settings to make your system more secure.

In this tutorial you will run through a practical demonstration of how to use systemd’s sandboxing techniques on Ubuntu 20.04 for an efficient workflow to implement and to test these techniques. Any process that runs on a Linux system that uses systemd can be made more secure with these techniques.

Prerequisites

You will need the following to begin this guide:

• An Ubuntu 20.04 system.
• A non-root user with sudo privileges. Follow the Initial Server Setup with Ubuntu 20.04 for information on how to do this on Ubuntu 20.04.
• Some familiarity with managing processes with systemd. Refer to the Systemd Essentials: Working with Services, Units, and the Journal guide to learn the essentials.
• A basic understanding of systemd unit files will be helpful. Refer to the Understanding Systemd Units and Unit Files guide for more information.

Step 1 — Installing lighttpd

In this tutorial, we will sandbox the lighttpd web server. lighttpd was not chosen because it is any less secure than other software, but because it is a small program with a single function that is easily sandboxed. This makes it an excellent choice for a learning application.

Let’s update the system to start:

sudo apt update

Check the packages that will be upgraded on your system before typing y:

sudo apt upgrade

Then install lighttpd:

sudo apt install lighttpd

This installation process will automatically install and enable a systemd service file for lighttpd. This will make lighttpd start on a system reboot.

Now that we have lighttpd installed and running on our system we’ll get familiar with the systemd tools we will use when we start sandboxing.

Step 2 — Preparing Your System

In this step, you will get familiar with the systemd commands that you will use and prepare your system to enable you to efficiently sandbox a process.

systemd is an umbrella name for a suite of tools that each have different names. The two that you will use are systemctl and journalctl. systemctl manages processes and their service files, while journalctl interacts with the system log.

systemd uses service files to define how a process will be managed. systemd loads these files from several locations in the file system. The following command will show you the location of the active service file and display any overrides that are in use:

sudo systemctl cat process.service

You need to replace process with the process that you are working on. Here lighttpd is used:

sudo systemctl cat lighttpd.service

This is the output from the previous command:

Output
# /lib/systemd/system/lighttpd.service
[Unit]
Description=Lighttpd Daemon
After=network-online.target

[Service]
Type=simple
PIDFile=/run/lighttpd.pid
ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf
ExecStart=/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
ExecReload=/bin/kill -USR1 $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

This output shows that the service file is located at /lib/systemd/system/lighttpd.service and that there are no override options in use. Override options add to or modify the base service file. You will use overrides to sandbox lighttpd with a dedicated override file.

Override files are located at /etc/systemd/system/process.service.d/override.conf. systemd has a dedicated edit command that will create an override file at the correct location and run systemctl daemon-reload after saving and exiting the editor. The systemctl daemon-reload instructs systemd to use any new configuration you wrote.

The systemd edit command has the following form:

sudo systemctl edit process.service

When you run this command systemd will usually choose your default CLI editor, but this is not always the case and you may find yourself in vi or even ed. You can configure which editor systemd will use by setting the SYSTEMD_EDITOR shell variable.

Set this shell variable by adding a line to your ~/.bashrc file. Open this file with a text editor:

nano ~/.bashrc

And add the following line:

export SYSTEMD_EDITOR=editor

Change editor to your preferred CLI editor. Here is the line set to use the nano editor:

export SYSTEMD_EDITOR=nano

Confirm that this is set after you log out and log in again with the echo command:

echo $SYSTEMD_EDITOR

This command will print the name of the editor you set.

The SYSTEMD_EDITOR shell variable is only set in your user’s shell and not root’s shell that gets opened by sudo. To pass this variable to root’s shell invoke systemctl edit using sudo -E:

sudo -E systemctl edit process.service

The final recommendation will make debugging your sandboxing easier by showing you any errors that your changes have caused. These errors will be recorded by the system log, which is accessed with the journalctl command.

During your sandboxing, you will make many changes that break the process you are trying to sandbox. For that reason, it is a good idea to open a second terminal and dedicate it to following the system log. This will save time re-opening the system log.

Follow the system log in the second terminal by running:

sudo journalctl -f -u process.service

• -f: Follow or tail the system log so new lines are displayed immediately.
• -u process.service: Only show the log lines for the process you are sandboxing.

The following is what you need to run to print only lighttpd’s errors:

sudo journalctl -f -u lighttpd.service

Next you’ll begin editing the override.conf file and start sandboxing lighttpd.

Step 3 — Enforcing a User and Group

In this step, you will set the non-root user that lighttpd will run as.

In its default configuration, lighttpd starts running as the root user and then changes to the www-data user and group. This is a problem because while lighttpd is running as root it can do anything that root can do—which is anything.

systemd provides the ability to start and run the process as a non-root user thereby avoiding this problem.

Return to your first terminal session and begin editing the override file by running:

sudo -E systemctl edit lighttpd.service

Now, add the following lines:

[Service]
User=www-data
Group=www-data

• [Service]: Tells systemd that the following options should be applied to the [Service] section.
• User=www-data: Defines the user to start the process as.
• Group=www-data: Defines the group to start the process as.

Next, save and exit the editor and restart lighttpd with the following command:

sudo systemctl restart lighttpd.service

lighttpd will not be able to start because it was using the root authority to write a PID file to a location owned by root. The www-data user is not able to write to directories owned by root. This problem is indicated in the system log that will appear in your second terminal session:

journalctl error message
Aug 29 11:37:35 systemd lighttpd[7097]: 2020-08-29 11:37:35: (server.c.1233) opening pid-file failed: /run/lighttpd.pid Permission denied

Resolving this issue follows the process of sandboxing which is:

1. Implement a sandbox restriction.
2. Restart the process and check for errors.
3. Fix any errors.

Next you’ll resolve the PID file issue while still enforcing the user and group restrictions you set in this section.

Step 4 — Managing the PID File

A PID file is a file that contains the PID or Process Identification Number of a running process. Long-running programs like lighttpd use them to manage their own processes. The problem that you encountered in the previous section was that lighttpd was unable to write its PID file to /run/lighttpd.pid, because /run/ is owned by root.

systemd has the RuntimeDirectory option for this problem, which you will use to give lighttpd a location that it can write its PID file to.

The RuntimeDirectory option allows you to specify a directory under /run/ that will be created with the user and group you set in Step 3 when systemd starts lighttpd. lighttpd will be able to write its PID into this directory without needing root’s authority.

First, open and edit the override file with the same command that you used in Step 3:

sudo -E systemctl edit lighttpd.service

Next, add the following line under the two lines that you already added to the override file:

RuntimeDirectory=lighttpd

Save and exit the editor.

You do not add the full path to the directory with the RuntimeDirectory, only the name of the directory under /run/. In this case, the directory that systemd will create is /run/lighttpd/.

You now need to configure lighttpd to write its PID file into the new directory /run/lighttpd/ instead of /run/.

Open lighttpd’s configuration file with a text editor:

sudo nano /etc/lighttpd/lighttpd.conf

Change the following line:

server.pid-file             = "/run/lighttpd.pid"

To:

server.pid-file             = "/run/lighttpd/lighttpd.pid"

Save and exit the editor.

Now, restart lighttpd:

sudo systemctl restart lighttpd.service

It won’t start because it is unable to do something that needs one of root’s capabilities. Next you’ll resolve this new issue.

Step 5 — Borrowing root’s Capabilities

The following line in the system log explains the issue that stopped lighttpd starting:

journalctl error message
Aug 29 12:07:22 systemd lighttpd[7220]: 2020-08-29 12:07:22: (network.c.311) can't bind to socket: 0.0.0.0:80 Permission denied

Only root can open a network port below number 1024. lighttpd is trying to open the HTTP port 80, but it is being denied because the www-data user cannot do that.

The issue is resolved by giving the lighttpd process a small part of root’s power—that is, to open ports below 1024.

root’s “power” is divided into abilities called “capabilities”. The root user has every capability and can therefore do anything. Breaking root’s power up into capabilities means that they can be given individually to non-root processes. This allows that process to do something that would have required a full root user, but a normal user can now do with one of root’s capabilities.

The systemd option to give a process one or more of root’s capabilities is the AmbientCapabilities option.

Open the override file:

sudo -E systemctl edit lighttpd.service

Then add the following line under the lines you already added:

AmbientCapabilities=CAP_NET_BIND_SERVICE

The CAP_NET_BIND_SERVICE capability allows a process to open ports under 1024.

Save and exit the file.

lighttpd will now be able to start.

You now have a working lighttpd web server that you have made more secure than its default configuration. There are more sandboxing options provided by systemd that you can use to make your target process even more secure. We will explore some of these in the following sections.

In the next step, you will restrict what lighttpd can access in the file system.

Step 6 — Locking Down the Filesystem

The lighttpd process runs as the www-data user and so can access any file on the system that www-data has permission to read and write to. In the case of www-data that isn’t very much, but still more than lighttpd needs.

The first and easiest sandbox setting is the ProtectHome option. This option stops the process from reading or writing to anything under /home/. lighttpd does not need access to anything under /home/ so implementing this will protect all of your private files without affecting lighttpd.

Open the override file:

sudo -E systemctl edit lighttpd.service

Then add the following line at the bottom of the file:

ProtectHome=true

Save and exit the editor then restart lighttpd to check that it is working as you expect with the following command:

sudo systemctl restart lighttpd.service

You have protected /home/, but that still leaves the rest of the file system. This is taken care of with the ProtectSystem option, which stops a process from writing to parts of the file system.

The ProtectSystem option has three settings that offer increasing levels of protection. They are as follows:

• true: Sets the following directories to read only:
  • /usr/
  • /boot/
  • /efi/
• full: Sets the following directories to read only:
  • /usr/
  • /boot/
  • /efi/
  • /etc/
• strict: Sets the following directories to read only:
  • Entire file system

A higher level of protection is more secure so set the ProtectSystem option to strict by adding the following line to the override file:

ProtectSystem=strict

Save and exit the editor and restart lighttpd with the following command:

sudo systemctl restart lighttpd.service

lighttpd will not be able to start because it needs to write its log files to /var/log/lighttpd/ and the strict setting forbids that. The following line in the system log shows the problem:

journalctl error message
Aug 29 12:44:41 systemd lighttpd[7417]: 2020-08-29 12:44:41: (server.c.752) opening errorlog '/var/log/lighttpd/error.log' failed: Read-only file system

This issue was anticipated by systemd with the LogsDirectory option. It takes the name of a directory under /var/log/ that the process is permitted to write its logs into.

Open the override file again in your first terminal session:

sudo -E systemctl edit lighttpd.service

The lighttpd log directory is /var/log/lighttpd/ so add the following line to the bottom of the override file:

LogsDirectory=lighttpd

Save and exit the editor and restart lighttpd:

sudo systemctl restart lighttpd.service

lighttpd will now be able to start and run.

In the next step, you will limit how the lighttpd process can interact with the rest of the system by restricting the system calls it is allowed to make.

Step 7 — Restricting System Calls

A system call is how a program requests something from the kernel. The number of system calls is quite large and includes actions like reading, writing, and deleting files, hardware related tasks like mounting a file system, spawning process, rebooting, and many more.

systemd has created groups of system calls that processes, like lighttpd, typically use and which exclude calls that they do not. The blocked system calls are things like mounting a file system and rebooting the system, which lighttpd never needs to do.

First, open the override file:

sudo -E systemctl edit lighttpd.service

Add the following line to the bottom of the file to use the SystemCallFilter option to set the @system-service group:

SystemCallFilter=@system-service

Save and exit the editor and restart lighttpd:

sudo systemctl restart lighttpd.service

In the next section, you will apply the remaining recommended sandboxing options.

Step 8 — Implementing Further Options

The systemd documentation recommends the following options are enabled for long-running, networked processes like lighttpd. These settings are all optional, but each one makes the process you are sandboxing more secure and should be used if you can.

You should enable these options one at a time and restart your process after each one. If you add them all at once debugging a problem will be much harder.

The recommended options following are accompanied by a brief description of what they do. Add these lines to your override file under the lines you have already added:

NoNewPrivileges=true

This option stops the sandboxed process and any of its children from obtaining new privileges.

ProtectKernelTunables=true

This option stops the process from changing any kernel variables.

ProtectKernelModules=true

This option stops the process from loading or unloading kernel modules.

ProtectKernelLogs=true

This option stops the process from reading and writing directly to the kernel log. It must use the system log application to record any log messages.

ProtectControlGroups=true

This option stops the process from modifying the system control groups.

MemoryDenyWriteExecute=true

This option stops the process from modifying any code that is running in the system’s memory.

RestrictSUIDSGID=true

This option stops the process from setting the set-user-ID (SUID) or set-group-ID (SGID) on files or directories. This ability can be abused to elevate privileges.

KeyringMode=private

This option stops the process from accessing the kernel keyring of other processes that are running as the same user.

ProtectClock=true

This option stops the process from changing the hardware and software system clocks.

RestrictRealtime=true

This option stops the process from enabling real-time scheduling that can be abused to overload the CPU.

PrivateDevices=true

This option stops the process from accessing physical devices attached to the system such as storage devices or USB devices.

PrivateTmp=true

This option forces the process to use private /tmp/ and /var/tmp/ directories. This stops the process from being able to read other program’s temporary files that are stored in those shared system directories.

ProtectHostname=true

This option stops the process from changing the system’s hostname.

The process that you have sandboxed is now much more secure than it was in its default configuration. You can now take these techniques and use them for any other processes you need to secure on your Linux system.

Conclusion

In this article, you made the lighttpd program more secure by using the systemd sandboxing options. You can use these techniques with any process that systemd manages allowing you to continue to improve the security of your system.

The entire list of sandboxing and other security options are found in systemd’s online documenation. Also, check out further security topics on the DigitalOcean Community.