How to Keep Ubuntu 20.04 Servers Updated

Introduction

In this tutorial, you’ll explore some best practices for keeping your Ubuntu 20.04 server stack up to date. Just as with network security hardening, there are many steps you can take to ensure your servers will continue to work securely without future intervention.

There are a number of tools and configurations that you can apply to most Ubuntu servers beyond what is configured for you automatically. If you’re doing your own server administration, it can be quite disruptive and error-prone to patch all of your environments manually.

This tutorial will cover:

• Testing graceful reboots following best practices for application management, to minimize any complications from maintenance updates
• Configuring automatic updates for most of the packages and libraries running on your machine
• Live kernel patching, and other best practices around kernel updates

Prerequisites

• An Ubuntu 20.04 server and a non-root user with sudo privileges. You can learn more about how to set up a user with these privileges in our Initial Server Setup with Ubuntu 20.04 guide.

Step 1 – Following Best Practices for Application Management

A fundamental part of configuring your server for automatic upgrades is ensuring that all of the applications running on the server are able to restart correctly after unplanned downtime or a reboot. Linux package managers are designed to run non-disruptively in the background so that necessary maintenance does not come with any additional overhead. Despite this, one of the most common reasons for not having a good update strategy in place is being concerned about how your server will behave after being restarted.

Wherever possible, the applications in your stack should be managed by your server’s init system, which on most modern Linux distributions including Ubuntu is systemd. Systemd provides the systemctl command for interacting with your running services and automatically restarting them as needed. Virtually all software that is installed via your package manager and designed to run in the background should automatically provide a systemd service and a configuration unit file as a best practice.

When running your own software, or software deployed from Git repositories, it is not a bad idea to write your own unit files in order to integrate with systemd. As a lightweight alternative, you may prefer to use a tool like supervisor. You can also use your system’s cron scheduler with the @reboot syntax.

After your configuration is in place, make sure to test it through reboots. You can reboot by running sudo shutdown now -r, which will cleanly stop your running processes and reboot immediately. You can also specify a time in hh:mm, or a number of minutes from now, instead of now in order to schedule a reboot in the future. Production deployments should generally not require your attention after any unplanned outages, and all necessary services and endpoints should come back up automatically.

Now that you’ve ensured that your environment won’t have any trouble persisting through maintenance reboots, in the next step, you’ll learn how to schedule automatic updates.

Step 2 – Configuring Unattended Upgrades

Ubuntu’s package manager, apt, has a well-established workflow for performing a full system upgrade. First, you run apt update to refresh your package listings, and then, you run apt upgrade without specifying a package in order to upgrade every package on the system. This workflow may be slightly different if you have any version conflicts from third-party packages or if you are intentionally keeping some packages unupgraded, but the core commands are the same.

Ubuntu provides a unique tool called unattended-upgrades in order to automatically retrieve and install security patches and other essential upgrades for your server. Most Ubuntu servers come with this tool automatically installed and configured, but you can install it with the following apt commands:

sudo apt update
sudo apt install unattended-upgrades

After installation, you can check to ensure that the unattended-upgrades service is running using systemctl:

sudo systemctl status unattended-upgrades.service

Output
● unattended-upgrades.service - Unattended Upgrades Shutdown
     Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-02-14 17:51:49 UTC; 3h 4min ago
       Docs: man:unattended-upgrade(8)
   Main PID: 829 (unattended-upgr)
      Tasks: 2 (limit: 1137)
     Memory: 10.6M
     CGroup: /system.slice/unattended-upgrades.service

The default configuration of unattended-upgrades will automatically retrieve bug fix and security updates for most of the packages included in the Ubuntu repositories. However, if you are using older versions of some packages in order to avoid upstream changes, or if your server uses third party package repositories in addition to Ubuntu’s, you can further configure the unattended-upgrades service.

Its configuration is stored in /etc/apt/apt.conf.d/50unattended-upgrades. Open this file using nano or your favorite text editor:

sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

The file is well-annotated, and you can see many lines of code comments (beginning with //) explaining its functionality. The first configuration block handles which packages will be automatically updated, matching a template for Ubuntu package repository names. Files in the core repository and in the -security repository will be updated by default, but the lines containing the -updates, -proposed, and -backports repositories are commented out by default.

These repositories are disabled by default because they are more likely to contain breaking changes to your installed packages. In order to enable them for unattended upgrades manually, you can delete the // comment symbols from these lines.

// Automatically upgrade packages from these (origin:archive) pairs
//
// Note that in Ubuntu security updates may pull in new dependencies
// from non-security sources (e.g. chromium). By allowing the release
// pocket these get automatically pulled in.
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
        // Extended Security Maintenance; doesn't necessarily exist for
        // every release and this system may not have it installed, but if
        // available, the policy for updates is such that unattended-upgrades
        // should also install from here by default.
        "${distro_id}ESMApps:${distro_codename}-apps-security";
        "${distro_id}ESM:${distro_codename}-infra-security";
//      "${distro_id}:${distro_codename}-updates";
//      "${distro_id}:${distro_codename}-proposed";
//      "${distro_id}:${distro_codename}-backports";
};
…

Further down the file, there are a number of options with true / false configuration toggles. For example, there is a toggle to automatically reboot following the installation of packages that need a reboot in order to take effect. You could enable this option by removing the // comment symbol and changing false to true. However, doing so will cause your server to become unavailable at unpredictable intervals. If you enable this option be sure your applications or users can tolerate downtime.

// Automatically reboot *WITHOUT CONFIRMATION* if
//  the file /var/run/reboot-required is found after the upgrade
//Unattended-Upgrade::Automatic-Reboot "false";

Save and close the file when you are done editing it. If you are using nano, press Ctrl+X, then when prompted, Y and then Enter.

If you made changes to the configuration, reload the unattended-upgrades service in order for it to take effect:

sudo systemctl reload unattended-upgrades.service

You should now have solutions in place to ensure all of the packages on your server receive essential security updates without any additional intervention. In the last step, you’ll learn how to keep your kernel updated, and how best to handle server reboots when they are necessary.

Step 3 – Updating and Livepatching the Kernel

Less often than other packages, you will need to update your system’s kernel. The Linux kernel contains (almost) all running hardware drivers and is responsible for most low-level system interactions. Kernel updates are usually only necessary if there is a high-profile vulnerability to address, if you need to make use of a publicized new kernel feature, or if your kernel has become so old that there is a greater risk of accumulated bugs and vulnerabilities that you may not be aware of.

There is no universal method of automatically scheduling Linux kernel updates. This is because kernel updates have historically required a full system reboot, and scheduling reboots is impossible without making assumptions about your environment. Many servers are expected to provide as close to 24/7 availability as possible, and a reboot can take an unknown amount of time, or require manual intervention.

If you are willing to tolerate some downtime, updating your kernel is straightforward: your unattended apt updates can be configured to install and prepare new kernels along with other packages, and after rebooting, your server should automatically use the new kernel. Most production deployments require additional complexity around rebooting like this to ensure service availability. For example, you might use a load balancer to automatically redirect traffic to servers that can provide identical functionality in a horizontally scaled deployment while they are individually rebooted in sequence, in order to avoid any visible downtime.

Enabling Livepatch to Ensure Server Uptime During Kernel Updates

To avoid downtime during kernel upgrades, you can use a feature of the Linux kernel called live patching. This feature makes it possible to implement kernel updates without rebooting. There are two major maintainers for kernel live patches: Canonical, who provides their own Livepatch Service for Ubuntu, and KernelCare who support Ubuntu in addition to most other major Linux distributions. Both require registration to use, and only Canonical’s service is free for individual use.

You can register for a Livepatch key at https://auth.livepatch.canonical.com/. After enrolling, you can install the canonical-livepatch snap package. Snap is another Ubuntu package manager that runs alongside apt.

sudo snap install canonical-livepatch

You can enable canonical-livepatch with a one-line command using the key from their website:

sudo canonical-livepatch enable your-key

The output should contain the message Successfully enabled device. The service should run in the background from now on without any further intervention, and you can check its status using canonical-livepatch status:

sudo canonical-livepatch status

Output
last check: 55 seconds ago
kernel: 5.4.0-26.30-generic
server check-in: succeeded
patch state: ✓ all applicable livepatch modules inserted
patch version: 84.1
tier: updates (Free usage; This machine beta tests new patches.)
machine id: d56589e7fa994005a266d4caf9b9dcf7

You have now configured automatic kernel updates for your server, meaning it should no longer be necessary to reboot in order to maintain a secure and up-to-date environment.

Conclusion

In this tutorial, you explored multiple strategies to keep your Ubuntu servers updated automatically. You also learned some of the nuances of package repositories, kernel updates, and handling server reboots. These are all important fundamentals of DevOps and of working with servers more broadly, and nearly all production configurations build on these core concepts.

Next, you may want to learn more about Package Management in Ubuntu. If you would like to learn more about the Snap package format, visit our tutorial How To Package and Publish a Snap Application on Ubuntu 18.04 .

sudo systemctl reload unattended-upgrades.service

sudo systemctl restart unattended-upgrades.service