How To Configure HAProxy Logging with Rsyslog on CentOS 8 [Quickstart]

Introduction

HAProxy, which stands for High Availability Proxy, is a widely used TCP and HTTP-based proxy server that runs on Linux, Solaris, and FreeBSD. It is used to load balance applications by distributing requests between multiple servers, and to ensure that applications are highly available for users.

By default on many CentOS 8 installations, HAProxy is not configured to write its log output to a file. This quickstart tutorial will explain how to configure HAProxy logging with Rsyslog by using a Unix domain socket for reliability, speed, and security.

Prerequisites

To complete this tutorial, you will need a server running CentOS 8. This server should have a non-root user with administrative privileges. To set this up, follow the Initial Server Setup guide for CentOS 8 tutorial.

Step 1 — Installing and Enabling HAProxy

To install HAProxy, run the following dnf command:

sudo dnf install haproxy

When you are prompted Is this ok [y/N]: enter y and press RETURN to install the package.

Once you have installed HAProxy, enable and start it using the systemctl command:

sudo systemctl enable haproxy.service

You should receive the following output:

Output
Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

With the HAProxy service now enabled, you can start it up to ensure that it runs with a default configuration on your server:

sudo systemctl start haproxy.service

Next examine HAProxy’s status to make sure it is running:

sudo systemctl status haproxy.service

You should receive output like the following. Note the highlighted active (running) portion of the output. If your server shows the same highlighted section then HAProxy is running correctly on your server and you can proceed with configuring logging.

Output
● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2020-09-09 21:16:39 UTC; 4min 39s ago
   Process: 21340 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
   Main PID: 21341 (haproxy)
      Tasks: 2 (limit: 2881)
     Memory: 2.7M
     CGroup: /system.slice/haproxy.service
             ├─21341 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
. . .

If your output is different, or the status shows something like Active: failed, then follow the troubleshooting steps in the How to Troubleshoot Common HAProxy Errors tutorial to determine what is preventing HAProxy from starting correctly.

Once you have confirmed that HAProxy is enabled and running, you can continue to the next step, which is configuring HAProxy’s logging directives.

Step 2 — Configuring HAProxy Logging Directives

To configure HAProxy’s logging directives, open /etc/haproxy/haproxy.cfg in vi or your preferred editor:

sudo vi /etc/haproxy/haproxy.cfg

Press i to switch to INSERT mode, then find the line log 127.0.0.1 local2 and comment it out by adding a # character to the beginning of the line, as highlighted in the following example:

. . .
# 2) configure local2 events to go to the /var/log/haproxy.log
#   file. A line like the following can be added to
#   /etc/sysconfig/syslog
#
#    local2.*                       /var/log/haproxy.log
#
    #log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
. . .

Now add a line directly after the commented out line with the following contents:

    log         /dev/log local0

The entire section of /etc/haproxy/haproxy.cfg that you edited should contain the following lines:

. . .
#    local2.*                       /var/log/haproxy.log
#
     #log         127.0.0.1 local2
     log         /dev/log local0

     chroot      /var/lib/haproxy
     pidfile     /var/run/haproxy.pid
. . .

The chroot line is important, because it restricts the HAProxy process to accessing files in the /var/lib/haproxy directory only. The log /dev/log local0 line will create a file inside that directory that Rsyslog will use to collect log entries from.

Once you are finished editing the log lines in /etc/haproxy/haproxy.cfg, save and close the file by pressing ESC, typing :wq, and pressing ENTER.

The last step that you need to complete in this section is to create the /var/lib/haproxy/dev directory since it does not exist by default.

Create the directory using the mkdir command and then restart HAProxy:

sudo mkdir /var/lib/haproxy/dev
sudo systemctl restart haproxy.service

You have now configured HAProxy to send its logs to a Unix domain socket that resides in /var/lib/haproxy/dev/log. In the next step, you will configure Rsyslog to create and access the socket.

Step 3 — Configuring Rsyslog to Collect HAProxy Logs

Rsyslog’s default configuration on CentOS 8 does not handle HAProxy logs. To collect logs from the HAProxy service, open a new file /etc/rsyslog.d/99-haproxy.conf using vi or your preferred editor:

sudo vi /etc/rsyslog.d/99-haproxy.conf

Press i to switch to INSERT mode, then paste the following lines into the file:

$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
  /var/log/haproxy.log
  stop
}

The $AddUnixListenSocket directive tells Rsyslog to create a Unix domain socket in the specified location, in this case /var/lib/haproxy/dev/log. The :programname, startswith, "haproxy" section specifies the file where Rsyslog will write the log entries to that it collects from the socket.

Once you are finished editing /etc/rsyslog.d/99-haproxy.conf, save and close the file by pressing ESC, typing :wq, and pressing ENTER.

You have now configured Rsyslog to read log entries from the Unix domain socket in /var/lib/haproxy/dev/log and write them to a log file in /var/log/haproxy.log.

However, before restarting Rsyslog you will need to determine if SELinux is enforcing access control on your CentOS 8 system.

To check SELinux’s current policy, run the following:

getenforce

You will receive one of the following outputs:

• Enforcing - In this mode, SELinux is enforcing access controls on your system. You will need to complete the following optional Step 4 — Configuring SELinux section.
• Permissive - In this case, SELinux logs all access attempts to its log file, but does not enforce access controls on your system.
• Disabled - If SELinux is disabled, then it is not logging or enforcing any access control policies on your system.

If the getenforce command returned either Permissive or Disabled, then you can restart Rsyslog with the following command:

sudo systemctl restart rsyslog

Once you restart Rsyslog, you will be able to view logs in the /var/log/haproxy.log file that you configured in /etc/rsyslog.d/99-haproxy.conf. Proceed to Step 5 — Testing HAProxy Logging to make sure that everything is working as expected.

Otherwise, if your system is running SELinux in Enforcing mode, then the next section of this tutorial explains how to add a module to allow Rsyslog and HAProxy to communicate with each other over their shared Unix domain socket.

Step 4 — (Optional) Configuring SELinux

If your CentOS 8 system is configured with SELinux in Enforcing mode, then you will need to allow Rsyslog access to HAProxy’s chroot directory. Allowing this access will let Rsyslog create the Unix domain socket that HAproxy will send its logs to.

If you are not familiar with SELinux, this tutorial series An Introduction to SELinux on CentOS 7 will help you learn how to manage and interact with SELinux. Although it is written for CentOS 7, the principles and commands in the series are equally applicable to CentOS 8.

To enable Rsyslog and HAProxy access to their shared socket, the first task is to create a Type Enforcement policy file. Open a new file called rsyslog-haproxy.te in vi or your preferred editor:

vi rsyslog-haproxy.te

Press i to switch to INSERT mode, then paste the following lines into the file:

module rsyslog-haproxy 1.0;

require {
    type syslogd_t;
    type haproxy_var_lib_t;
    class dir { add_name remove_name search write };
    class sock_file { create setattr unlink };
}

#============= syslogd_t ==============
allow syslogd_t haproxy_var_lib_t:dir { add_name remove_name search write };
allow syslogd_t haproxy_var_lib_t:sock_file { create setattr unlink };

The first line defines the module name and version. The require portion tells the SELinux module loader about the types and classes that are required for the policy to be loaded as a module. The last two lines are the rules that allow Rsyslog access to HAProxy’s chroot and socket file respectively.

When you are done editing the file, save and close it by pressing ESC, typing :wq, and pressing ENTER.

Next, run the following command to install the checkpolicy package, which contains the checkmodule utility that you will use to turn the Type Enforcement file into an SELinux module.

sudo dnf install checkpolicy

Now that you have the checkmodule tool installed, the next step is to compile the module and then load it into SELinux. Run the following to compile the Type Enforcement file into an SELinux module:

checkmodule -M -m rsyslog-haproxy.te -o rsyslog-haproxy.mod

Next, run semodule_package to generate a complete policy package that SELinux can load into the Linux kernel:

semodule_package -o rsyslog-haproxy.pp -m rsyslog-haproxy.mod

The final step is to load the package that you generated into the Linux kernel using the semodule command:

sudo semodule -i rsyslog-haproxy.pp

Adding the module may take a few seconds. Once the command completes you can confirm that the module is loaded into the kernel by running the semodule command:

sudo semodule -l |grep rsyslog-haproxy

You should receive output like the following:

Output
rsyslog-haproxy

Once the module is loaded you can restart Rsyslog with the following command:

sudo systemctl restart rsyslog

You have now defined, compiled, and loaded an SELinux policy that will allow HAProxy and Rsyslog to communicate over their shared socket.

In the next step you will test that everything works by making an HTTP request to HAProxy and examining its new log file.

Step 5 — Testing HAProxy Logging

Now that you have configured HAProxy, Rsyslog, and optionally SELinux, you can test that logging to /var/log/haproxy.log is working.

By default the haproxy package ships with a configuration file that creates an HTTP listener socket on port 5000. The configuration points to a non-existent backend server, so any request to the port will result in an HTTP 503 error.

To check for a 503 error in your /var/log/haproxy.log file, first generate an HTTP request using curl like this:

curl -si http://127.0.0.1:5000

You should receive output like the following:

Output
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

Now examine /var/log/haproxy.log for any HTTP 503 responses using the grep command:

sudo grep -E 'NOSRV.+503' /var/log/haproxy.log

You should receive a line (or multiple lines) like the following:

Output
Sep 9 21:32:22 centos-s-1vcpu-1gb-nyc3-01 haproxy[4451]: 127.0.0.1:56024 [9/Sep/2020:21:32:22.098] main app/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

This line corresponds to the curl request that you made, which means that Rsyslog and HAProxy are configured to use their shared socket correctly.

Conclusion

In this quickstart tutorial, you configured HAProxy to log to a Unix domain socket. You also set up Rsyslog to create and read from the socket so that the two programs can communicate with each other without opening up any TCP/IP ports on your system. Finally, you optionally compiled, packaged, and loaded an SELinux policy to allow Rsyslog and HAProxy shared access to their socket.