Common HAProxy Errors

Introduction

There are three main commands, and a common log location that you can use to get started troubleshooting HAProxy errors. Generally when you are troubleshooting HAProxy, you will use these commands in the order indicated here, and then examine the log file for specific diagnostic data.

The commands and log that you will commonly use to troubleshoot HAProxy across most Linux distributions are:

• systemctl - Used to control and interact with Linux services via the systemd service manager.
• journalctl - Used to query and view the logs that are generated by systemd.
• haproxy - When troubleshooting, this command is used to check HAProxy’s configuration.
• /var/log/haproxy.log - This file contains log entries from HAProxy itself detailing TCP and HTTP traffic that is being handled by the server.

These commands, how to use them, and HAProxy’s logs where you can find additional information about errors are described in further detail in the following sections.

systemctl Commands for HAProxy

To troubleshoot common HAProxy errors using the systemd service manager, the first step is to inspect the state of the HAProxy processes on your system. The following systemctl commands will query systemd for the state of HAProxy’s processes on most Linux distributions.

sudo systemctl status haproxy.service -l --no-pager

The -l flag will ensure that output is not truncated or ellipsized. The --no-pager flag will make sure that output will go directly to your terminal without requiring any interaction on your part to view it. If you omit the --no-pager flag you will be able to scroll through the output using arrow keys, or the page up and down keys. To quit from the pager use the q key. You should receive output like this:

Output
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-08-20 19:30:11 UTC; 5s ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
 Main PID: 488 (haproxy)
    Tasks: 2 (limit: 2344)
. . .
Aug 19 21:31:46 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

Your output may be slightly different depending on which Linux distribution you are using, but in any case, make a note of the Active line in the output. If your HAProxy server does not show active (running) as highlighted in the example output but you expect it should, there may be an error. Typically if there is a problem, you will have a line like the following in your output (note the highlighted failed portion):

Example Error Output
   Active: failed (Result: exit-code) since Thu 2020-08-20 19:32:26 UTC; 6s ago

If there is a problem with your HAProxy process or configuration you can troubleshoot it further using the journalctl command.

journalctl Commands for HAProxy

To inspect the systemd logs for HAProxy, you can use the journalctl command. The systemd logs for HAProxy will usually indicate whether there is a problem with starting or managing the HAProxy process.

These logs are separate from HAProxy’s request and error logs. journalctl displays logs from systemd that describe the HAProxy service itself, from startup to shutdown, along with any process errors that may be encountered along the way.

sudo journalctl -u haproxy.service --since today --no-pager

The --since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors. You should receive output like the following (there may be a few extra lines between the Starting and Started lines depending on your Linux distribution):

Output
Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Starting HAProxy Load Balancer...
. . .
Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

If there is an error, you will have a line in the output that is similar to the following, with the main difference between Linux distributions being the highlighted yourhostname portion:

Example Error Output
Aug 20 19:32:25 yourhostname systemd[1]: Failed to start HAProxy Load Balancer.

If your HAProxy server has errors in the journalctl logs like the previous example, then the next step to troubleshoot possible issues is investigating HAProxy’s configuration using the haproxy command line tool.

Troubleshooting with haproxy

To troubleshoot HAProxy configuration issues, use the haproxy -c command. The tool will parse your HAProxy files and detect any errors or missing settings before attempting to start the server.

Run the command like this on Ubuntu, Debian, CentOS, and Fedora based distributions. Be sure to change the path to the configuration file if you are using a different filename or location:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

A working HAProxy configuration will result in output like the following:

Output
Configuration file is valid

If there is an error in your HAProxy configuration, like a typo or misplaced directive, haproxy -c will detect it and attempt to notify you about the problem.

For example, attempting to use the bind directive in haproxy.cfg in the wrong location will result in messages like the following:

Example Error Output
[ALERT] 232/194354 (199) : parsing [/etc/haproxy/haproxy.cfg:13] : unknown keyword 'bind' in 'global' section
[ALERT] 232/194354 (199) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 232/194354 (199) : Fatal errors found in configuration.

In this example the bind directive is misplaced inside a global configuration section, so HAProxy generates the unknown keyword error. The message also includes a line number 13, so that you can edit the file and fix or remove the erroneous line without having to search through the file.

Learning how to use haproxy -c to detect and fix errors is useful when you are troubleshooting an existing error, or before you reload HAProxy with an edited configuration that may contain errors.

HAProxy Log Files

HAProxy log files are a very helpful resource for troubleshooting. Generally, any error that you receive in a browser or other HTTP client will have a corresponding entry in HAProxy’s logs. Sometimes HAProxy will also output errors related to configuration and other debugging information to its log files.

On Ubuntu and Debian based Linux distributions, the haproxy package includes scripts that configure log output in /var/log/haproxy.log.

On CentOS, Fedora, and other RedHat-derived Linux distributions, haproxy does not output to a log file by default. To log HAProxy output logs to /var/log/haproxy.log, follow this quickstart tutorial, How To Configure HAProxy Logging with Rsyslog on CentOS 8.

When you are troubleshooting HAProxy using its log file, examine /var/log/haproxy.log for errors using a tool like tail or less. For example, to view the last two lines of the log using tail, run the following command:

sudo tail -n 2 /var/log/haproxy.log

An example error will resemble something like the following lines, regardless of which Linux distribution you are using to run your HAProxy server:

Log Examples
Aug 20 19:36:21 d6cdd0c71489 haproxy[19202]: [ALERT] 258/134605 (19202) : Proxy 'app', server 'app1' [/etc/haproxy/haproxy.cfg:88] verify is enabled by default but no CA file specified. If you're running on a LAN where you're certain to trust the server's certificate, please set an explicit 'verify none' statement on the 'server' line, or use 'ssl-server-verify none' in the global section to disable server-side verifications by default.
Aug 20 19:36:22 d6cdd0c71489 haproxy[4451]: 203.0.113.1:54428 [20/Aug/2020:19:36:22.288] main app/<NOSRV> 0/-1/-1/-1/1 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

These example lines are just for illustration purposes. If you are diagnosing errors with your HAProxy server, chances are the lines in your logs will have different contents than these. Some lines will include success responses and other non-critical diagnostic entries.

Regardless of your Linux distribution, the format of the lines in your HAProxy logs will include any HTTP status codes that are returned to clients, along with requesting IPs and the status of backend servers.

Once you have an idea of what might be causing problems with your HAProxy server you can continue researching and troubleshooting the issue. The HTTP status code and text description are especially useful, since they give you explicit and specific terms that you can use to narrow down the range of possible causes of a problem.

Conclusion

Troubleshooting HAProxy errors can range from diagnosing errors with the service itself, to locating misconfigured options for modules, or to examining customized access control rules in detail. This introduction to diagnosing issues with HAProxy explained how to use a number of utilities to help narrow down the possible causes of errors. Usually, you will use these utilities in the same order, although you can always skip some, or start directly with examining logs if you have a general idea of what the problem might be.

However, as a general sequence for troubleshooting, it helps to be methodical and use these tools in the order described. Start troubleshooting with systemctl to examine the state of the HAProxy server. If you need more information, examine the systemd logs for HAProxy using the journalctl command. If the issue is still not apparent after checking journalctl, testing HAProxy’s configuration using haproxy -c -f /etc/haproxy/haproxy.cfg is the next step. Finally, for in-depth troubleshooting, examining HAProxy’s log files will usually indicate a specific error, with helpful diagnostic messages and error codes.

The rest of the tutorials in this series will examine some common errors that you may encounter when using HAProxy in more detail.

Introduction

An HAProxy cannot bind socket error message is generated when there is another process listening on the same interface and TCP port combination that HAProxy is configured to use, or when HAProxy attempts to use an IP address that is not assigned to a network interface. Both error conditions derive from the underlying operating system’s network stack.

In the first case, when there is another process that is already using an interface and port that HAProxy is attempting to bind to, the underlying error on Linux is EADDRINUSE. The issue is that only a single process can be bound to an IP address and port combination at any given time.

In the second case, when HAProxy is attempting to use an IP address that is not assigned to an interface on the system, the underlying error on Linux is EADDRNOTAVAIL. The issue here is that an IP socket cannot be created using an address that is not available to the operating system.

However, both underlying errors generate the same HAProxy error message, so troubleshooting a cannot bind socket error requires examining the list of currently used sockets and IP addresses on a Linx system.

To detect a cannot bind socket error message, you will need to examine systemctl and journalctl output to determine the IP address and port combination that are causing the error. Then you can inspect other running processes and network interfaces and decide how to resolve the issue, whether it is by switching servers, changing the IP address or port that HAProxy uses, or any combination of these options.

Troubleshooting with systemctl

Following the troubleshooting steps from the How to Troubleshoot Common HAProxy Errors tutorial at the beginning of this series, the first step when you are troubleshooting an cannot bind socket error message is to check HAProxy’s status with systemctl.

The output from systemctl status will in many cases contain all the diagnostic information that you need to resolve the error. It may include the IP address that HAProxy is using, as well as the port that it is attempting to bind to. The output will also indicate how long HAProxy has been unable to start so that you can determine how long the issue has been affecting HAProxy.

On CentOS, Fedora and RedHat-derived systems, use this systemctl command to examine HAProxy’s status:

sudo systemctl status haproxy.service -l --no-pager

The -l flag will ensure that systemctl outputs the entire contents of a line, instead of substituting in ellipses (…) for long lines. The --no-pager flag will output the entire log to your screen without invoking a tool like less that only shows a screen of content at a time.

Since you are troubleshooting a cannot bind socket error message, you should receive output that is similar to the following:

Output
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2020-08-19 14:57:05 UTC; 3s ago
  Process: 138738 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE (code=exited, status=1/FAILURE)
  Process: 138736 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
 Main PID: 138738 (code=exited, status=1/FAILURE)

Aug 19 14:57:05 92214d8ff5e2 systemd[1]: Starting HAProxy Load Balancer...
Aug 19 14:57:05 92214d8ff5e2 haproxy[138738]: [ALERT] 231/145705 (138738) : Starting frontend main: cannot bind socket [0.0.0.0:80]
. . .
Aug 19 14:57:05 92214d8ff5e2 systemd[1]: Failed to start HAProxy Load Balancer.

This example systemctl output includes some highlighted lines from the systemd journal that describes the error. These lines give you all the information about the error that you need to troubleshoot it further. Specifically, the line cannot bind socket [0.0.0.0:80] describes the socket that HAProxy is trying to use (0.0.0.0:80), so you can skip the following journalctl steps and instead proceed to the Troubleshooting with ss and ps Utilities section at the end of this tutorial. The other highlighted line indicates the status of the HAProxy process, which in the case of a cannot bind socket error will show Failed to start HAProxy Load Balancer.

If your systemctl output does not give specific information about the IP address and port or ports that are causing the error (if you are using Ubuntu or Debian then this applies), then you will need to examine journalctl output from the systemd logs. The following section explains how to use journalctl to troubleshoot a cannot bind socket error.

Troubleshooting Using journalctl Logs

If your systemctl output does not include specifics about a cannot bind socket error, you should proceed with using the journalctl command to examine systemd logs for HAProxy.

On Ubuntu and Debian-derived systems, run the following command:

sudo journalctl -u haproxy.service --since today --no-pager

On CentOS, Fedora, and RedHat-derived systems, use this command to inspect the logs:

sudo journalctl -u haproxy.service --since today --no-pager

The --since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors.

If HAProxy is unable to bind to a port that is in use, search through the output for lines that are similar to the following log entries, specifically lines that contain the cannot bind socket error message as highlighted in this example:

Output
-- Logs begin at Wed 2020-08-19 19:38:12 UTC, end at Wed 2020-08-19 19:53:53 UTC. --
. . .
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: Starting HAProxy Load Balancer...
Aug 19 19:39:21 92214d8ff5e2 haproxy[135]: [ALERT] 231/193921 (135) : Starting frontend main: cannot bind socket [0.0.0.0:80]
Aug 19 19:39:21 92214d8ff5e2 haproxy[135]: [ALERT] 231/193921 (135) : Starting frontend main: cannot bind socket [:::80]
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: haproxy.service: Main process exited, code=exited, status=1/FAILURE
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Aug 19 19:39:21 92214d8ff5e2 systemd[1]: Failed to start HAProxy Load Balancer.
. . .

The first highlighted line of output indicates that HAProxy cannot bind to port 80 on all available IPv4 interfaces (denoted by the 0.0.0.0 IP address). Depending on your system’s configuration, the IP addresses may be different and only show individual IPs.

If you are using HAProxy with IPv6, then the output may also include a line like the second one that is highlighted with an IPv6 specific interface and port error, in this case :::80. The first two :: characters indicate all available IPv6 interfaces, while the trailing :80 indicates the port.

Even though your own system may have different conflicting interfaces and ports, the errors will be similar to the output shown here. With this output from journalctl you will be able to diagnose the issue using ss, ps, and ip commands in the following section of this tutorial.

Troubleshooting with ss and ps Utilities

To troubleshoot a cannot bind socket error you need to determine what other process is listening on the IP address and port that HAProxy is attempting to use, or if the IP address is available to HAProxy.

For example, if another server like Nginx is configured to listen on port 8080 on all available IPv4 network interfaces, the full socket would be 0.0.0.0:8080. If HAProxy is also configured to use 0.0.0.0:8080 then the operating system will throw an EADDRINUSE error, and HAProxy will show a cannot bind socket error message, since it cannot claim the socket for itself.

In the previous journalctl section, something was already bound to all the available IPv4 addresses (denoted by 0.0.0.0:80). Most modern Linux distributions include a utility called ss which can be used to gather information about the state of a system’s network sockets.

The following command will determine the name of the process that is already bound to an IPv4 interface on port 80. Ensure that you substitute the port from the error message if it is different from 80 in the following command:

sudo ss -4 -tlnp | grep 80

The flags to the ss command alter its default output in the following ways:

• -4 restricts ss to only display IPv4-related socket information.
• -t restricts the output to tcp sockets only.
• -l displays all listening sockets with the -4 and -t restrictions taken into account.
• -n ensures that port numbers are displayed, as opposed to protocol names like ‘httporhttps`. This is important since HAProxy may be attempting to bind to a non-standard port and a service name can be confusing as opposed to the actual port number.
• -p outputs information about the process that is bound to a port.
• | grep 80 limits the output to lines that contain the characters 80 so there are fewer lines that you have to examine

With all of those flags, you should receive output like the following:

Output
LISTEN   0         511                 0.0.0.0:80               0.0.0.0:*        users:(("nginx",pid=40,fd=6))

The first three fields are not important when troubleshooting a cannot bind socket error so they can be ignored. The important fields are the fourth (0.0.0.0:80), which matches the journalctl error that you discovered earlier, along with the last users:(("nginx",pid=40,fd=6)), specifically the pid=40 portion.

If you have a cannot bind socket error that is related to an IPv6 interface, repeat the ss invocation, this time using the -6 flag to restrict the interfaces to the IPv6 network stack like this:

sudo ss -6 -tlnp |grep 80

The -6 flag limits the ip command to IPv6 interfaces. If HAProxy is unable to bind to an IPv6 socket, you should have output like the following:

Output
LISTEN   0         511                    [::]:80                  [::]:*        users:(("nginx",pid=40,fd=7))

Again, substitute the port number in question from your journalctl output if it is different from the highlighted 80 given here.

In both these cases of IPv4 and IPv6 errors, the ss output indicates that there is a program with process ID 40 (the pid=40 in the output) that is bound to the 0.0.0.0:80 and [::]:80 interfaces respectively. This process is preventing HAProxy from starting since it already owns the port. To determine the name of the program, use the ps utility like this, substituting the process ID from your output in place of the highlighted 40 value in this example:

sudo ps -p 40

You will receive output that is similar to the following:

Output
PID TTY          TIME CMD
40 ?        00:00:00 nginx

The highlighted nginx in the output is the name of the process that is listening on the interfaces. Now that you have the name of the program that is preventing HAProxy from starting, you can decide how to resolve the error. You could stop the nginx process, reconfigure nginx to listen on a different interface and port, or reconfigure HAProxy to avoid the port collision.

It is important to note that the process may be different from nginx and the port and IP addresses may not always be 0.0.0.0 or [::] if you are diagnosing a cannot bind socket error. Oftentimes, different web servers and proxies will be in use on the same server. Each may be attempting to bind to different IPv4 ports and IPv6 interfaces to handle different web traffic. For example, a server that is configured with HAProxy listening on the IPv4 loopback address (also referred to as localhost) on port 8080 will show ss output like this:

Output
LISTEN   0         2000              127.0.0.1:8080            0.0.0.0:*       users:(("haproxy",pid=545,fd=7))

It is important to combine systemctl output, or journalctl output that indicates specific IP addresses and ports, with diagnostic data from ss, and then ps to narrow down the process that is causing HAProxy to fail to start.

Sometimes when you are troubleshooting a cannot bind socket error message with ss and ps there will not be any output at all, which means that the error may not be caused by a socket conflict. The next section of this tutorial explains how to troubleshoot a cannot bind socket error using the ip utility.

Troubleshooting with the ip Utility

The previous section explained how an EADDRINUSE operating system error could cause a cannot bind socket error message. However, if you have examined ss and ps output and there is no socket conflict on your system, the issue may be caused by an EADDRNOTAVAIL operating system error instead. In this case HAProxy may be trying to bind to a socket that is not available to your operating system.

To determine whether a cannot bind socket error is caused by an EADDRNOTAVAIL, examine both the IPv4 and IPv6 network interfaces on your system using the ip command.

sudo ip -4 -c address show

• -4 restricts ip to only display IPv4-related interface information.
• -c adds color coding to the output so that it is easier to parse visually.
• address show displays the IP address for an interface, with the -4 and -c flags taken into account.

You should receive output that looks similar to the following on any Linux distribution that includes the ip tool:

Output
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 203.0.113.1/24 brd 203.0.113.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.0.2.1/24 brd 192.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 198.51.100.1/24 brd 198.51.100.255 scope global eth1
       valid_lft forever preferred_lft forever

Make a note of your IP addresses that correspond to the highlighted examples in this output. Your IP addresses and network interfaces will be different than the examples shown here. You may have more or fewer interfaces, and each may have more or fewer addresses assigned to them. The important part is to note the IP addresses from ip.

To examine IPv6 addresses that are assigned to your system, use the ip command with the -6 flag like this:

sudo ip -6 -c address show

You should receive output like the following:

Output
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2604:a880:400:d1::3d3:6001/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a4ff:aaff:fec9:24f8/64 scope link
       valid_lft forever preferred_lft forever

Again note the highlighted values in this example output and look for the corresponding IPv6 addresses in your output.

Once you have a list of addresses that are assigned to your system, you can try to find a matching IP address that corresponds to the cannot bind socket [x.x.x.x:80] error. If there is no IP address that matches, then HAProxy is likely configured to use an IP address that is not available to your system and the cannot bind socket error is being caused by the operating system throwing an EADDRNOTAVAIL error.

To resolve the error you will need to edit your /etc/haproxy/haproxy.cfg file and change the bind address or addresses to an IP address that is available to your system based on the output of the ip command.

For example, if /etc/haproxy/haproxy.cfg contained a bind line like the following using 198.51.100.123 as the IP address, but your system has 198.51.100.1 assigned based on the example output above, you will need to edit the bind line.

Following this hypothetical example, this haproxy.cfg snippet shows the invalid IP address:

. . .
frontend main
        bind 198.51.100.123:80

A correct bind line that matches the IP address in the example ip output would look like this:

. . .
frontend main
        bind 198.51.100.1:80

Once you have edited /etc/haproxy/haproxy.cfg with the correct IP address, restart it using the systemctl command:

sudo systemctl restart haproxy.service

Now examine HAProxy’s status and make sure that the output shows an active (running) line:

sudo systemctl status haproxy.service

Output
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-08-19 21:31:46 UTC; 17h ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
. . .
Aug 19 21:31:46 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

If you have resolved the cannot bind socket error your output should be similar to this example output. The highlighted lines that show HAProxy is active, and that the process was started successfully.

Conclusion

In this tutorial you learned how to troubleshoot an HAProxy cannot bind socket error message on both IPv4 and IPv6 interfaces. You learned how to use systemctl to examine the status of the HAProxy server and try to find error messages. You also learned how to use journalctl to examine the systemd logs for specific information about a cannot bind socket error.

With the appropriate error messages from the systemd logs, you then learned about the ss utility and how to use it to examine the state of a system’s network sockets. After that you learned how to combine process ID information from ss with the ps utility to find the name of the process that is causing HAProxy to be unable to start.

Finally, in the case of a cannot bind socket error that is related to an unavailable IPv4 or IPv6 address, you learned how to use the ip utility to examine available network interfaces on your system.

Introduction

In this tutorial you will learn how to troubleshoot and fix an HAProxy Setting tune.ssl.default-dh-param to 1024 by default warning message using the methods described in the How to Troubleshoot Common HAProxy Errors tutorial at the beginning of this series. One you confirm that your server is generating the warning message, you will learn how to fix it by setting HAProxy’s ssl-dh-param-file configuration option to use a custom dhparams.pem file.

You may encounter an HAProxy Setting tune.ssl.default-dh-param to 1024 by default warning message when your HAProxy server is configured with an SSL/TLS certificate and the tune.ssl.default-dh-param parameter is not set in HAProxy’s haproxy.cfg configuration file. When the parameter is not set, HAProxy will default to a value of 1024 bits for the Diffie-Hellman key agreement portion of the TLS handshake, which is considered insecure.

Your HAProxy server may trigger this warning without you knowing about it. You can configure an SSL/TLS certificate and HAProxy will run, but the server may be warning you about the issue in the background. If you aren’t sure whether your HAProxy server is defaulting to a 1024 bit Diffie-Hellman parameter, you can use a tool like the SSL Labs Server Test to check for you. If you receive a report that has a line This server supports weak Diffie-Hellman (DH) key exchange parameters, then your HAProxy server is affected.

If you have already determined that your HAProxy server is generating a warning about the tune.ssl.default-dh-param setting and you would like to skip troubleshooting, the Resolving a Secure tune.ssl.default-dh-param Warning section at the end of this tutorial explains how to fix the issue.

Checking for a tune.ssl.default-dh-param Warning Using systemctl

Following the troubleshooting steps from the How to Troubleshoot Common HAProxy Errors tutorial at the beginning of this series, the first step when you are troubleshooting a Setting tune.ssl.default-dh-param to 1024 by default warning message is to check HAProxy’s status with systemctl.

The output from systemctl status may contain all the diagnostic information that you need to resolve the error. However, it is important to examine the current status of the haproxy service to ensure that it is running and that any services that rely on it are still able to function while you resolve the warning.

Use this systemctl command to examine HAProxy’s status on any Linux distribution:

sudo systemctl status haproxy.service -l --no-pager

The -l flag will ensure that systemctl outputs the entire contents of a line, instead of substituting in ellipses (…) for long lines. The --no-pager flag will output the entire log to your screen without invoking a tool like less that only shows a screen of content at a time.

Since you are troubleshooting a Setting tune.ssl.default-dh-param to 1024 by default warning message, you should receive output that is similar to the following:

Output
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-10-06 14:31:39 UTC; 2min 31s ago
  Process: 71406 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
 Main PID: 71407 (haproxy)
. . .

Oct 06 14:31:39 bb9fb4c53743 systemd[1]: Starting HAProxy Load Balancer...
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]: [WARNING] 279/143139 (71407) : parsing [/etc/haproxy/haproxy.cfg:69] : 'bind *:5000' :
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]:   unable to load default 1024 bits DH parameter for certificate '/etc/haproxy/fullchain.pem'.
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]:   , SSL library will use an automatically generated DH parameter.
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]: [WARNING] 279/143139 (71407) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]: Proxy main started.
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]: Proxy static started.
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]: Proxy app started.
Oct 06 14:31:39 bb9fb4c53743 systemd[1]: Started HAProxy Load Balancer.

There are two important highlighted sections to note in this output. The first is the active (running) line, which indicates that HAProxy is available and running. If your server shows a line like Active: failed, then you will need to troubleshoot HAProxy using the methods described in the How to Troubleshoot Common HAProxy Errors tutorial at the beginning of this series.

The second set of highlighted lines are from the systemd journal and contain the tune.ssl.default-dh-param warning. These lines give you all the information about the warning, how it is triggered, the steps taken by HAProxy, and how to resolve it.

If your systemctl output includes lines with the tune.ssl.default-dh-param warning, then skip to the Resolving a tune.ssl.default-dh-param Warning section at the end of this tutorial to learn how to configure HAProxy with a more secure value.

Otherwise if your systemctl output does not give specific information about the warning, but you know that your server is using weak Diffie-Hellman parameters, the next section of this tutorial will guide you through using journalctl logs to locate the warning message.

Checking for a tune.ssl.default-dh-param Warning Using journalctl

If your systemctl output does not include a tune.ssl.default-dh-param warning but you know that your server is affected, you should proceed with using the journalctl command to examine systemd logs for HAProxy.

Run the following command on any Linux distribution to inspect the systemd journal’s logs for the haproxy service:

sudo journalctl -u haproxy.service -l --no-pager | grep tune.ssl.default-dh-param

The | grep tune.ssl.default-dh-param portion of the command uses a pipe (|) to send the output from journalctl to the grep command as input. Using the pipe means that only matching lines from the journal will be shown on your screen.

If the journal includes a tune.ssl.default-dh-param warning, then you will receive output like the following:

Output
Oct 06 14:31:39 bb9fb4c53743 haproxy[71407]: [WARNING] 279/143139 (71407) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

Since this output includes the warning you can skip to the Setting a Secure tune.ssl.default-dh-param Value section at the end of this tutorial to learn how to configure HAProxy with a more secure value.

If you do not receive any output after running the journaltcl command, the next section will demonstrate how you can check for a tune.ssl.default-dh-param warning using HAProxy’s built-in configuration checking tool.

Checking for a tune.ssl.default-dh-param Warning Using haproxy -c or Log Files

HAProxy includes a command that can examine and validate its configuration files. You can use the command to check for syntax errors or invalid settings without restarting HAProxy and risking downtime for your services. Additionally, if your HAProxy server is configured to log output to a file or the system logs, you can examine the log files to check for a tune.ssl.default-dh-param warning.

To check for a tune.ssl.default-dh-param warning using HAProxy itself, run the following command on any Linux distribution. If your HAProxy configuration file is in a different location than the highlighted /etc/haproxy/haproxy.cfg path, be sure to substitute in the correct path to your file:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

If your server is configured with SSL/TLS and does not have a tune.ssl.default-dh-param set, then you will receive warning output similar to the following:

Output
[WARNING] 279/150829 (71512) : parsing [/etc/haproxy/haproxy.cfg:69] : 'bind *:5000' :
  unable to load default 1024 bits DH parameter for certificate '/etc/haproxy/fullchain.pem'.
  , SSL library will use an automatically generated DH parameter.
[WARNING] 279/150829 (71512) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Configuration file is valid

The highlighted line contains the tune.ssl.default-dh-param warning.

To check for the warning message using HAProxy’s logs, you can search for lines that match the tune.ssl.default-dh-param string using the grep command. Run the following command to check for warnings:

sudo grep tune.ssl.default-dh-param /var/log/haproxy.log

Note that if you have configured HAproxy with a different log location, be sure to substitute in your path in place of the highlighted /var/log/haproxy.log file.

If your server is generating the warning, then you will receive output like the following:

Output
Oct  6 14:31:39 bb9fb4c53743 haproxy[71407]: [WARNING] 279/143139 (71407) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

Now that you have determined whether your server is generating a tune.ssl.default-dh-param warning, you can proceed to the next step in this tutorial, which explains two methods that you can use to resolve the issue.

Resolving a tune.ssl.default-dh-param Warning

You can resolve a tune.ssl.default-dh-param warning in HAProxy with a few different settings. One option is to set the tune.ssl.default-dh-param value to something larger than 1024 explicitly in /etc/haproxy/haproxy.cfg per the warning message. However, the HAProxy documentation recommends specifying custom Diffie-Hellman parameters since that approach is more secure, so we’ll use that method instead.

First you’ll generate a dhparams.pem file using the openssl utility. Once the file is created, you’ll add it to your global HAProxy configuration section so that any frontend blocks will inherit the setting.

To generate the custom DH parameters, run the following:

sudo openssl dhparam -out /etc/haproxy/dhparams.pem 2048

You will receive output like the following:

Output
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
..............

Once the command completes, configure HAProxy to use the custom DH parameters file. Open /etc/haproxy/haproxy.cfg in vi or your preferred editor.

sudo vi /etc/haproxy/haproxy.cfg

Find the section in the file like the following:

Output
. . .
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
    #ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

Now add a line after the ssl-default-server-ciphers PROFILE=SYSTEM line like this:

ssl-dh-param-file /etc/haproxy/dhparams.pem

The entire section should resemble the following, including the new highlighted ssl-dh-param-file line that you added:

/etc/haproxy/haproxy.cfg
. . .
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
    #ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM
    ssl-dh-param-file /etc/haproxy/dhparams.pem

When you are done editing the file, save and close it by pressing ESC, typing :wq, and pressing ENTER.

Test the configuration using the haproxy -c command to ensure that you have added the correct setting and that HAProxy can access the custom dhparams.pem file:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

You should receive output stating that the configuration is valid like the following:

Output
Configuration file is valid

If your configuration is valid, restart HAProxy so that it uses the new Diffie-Hellman parameters file:

sudo systemctl restart haproxy.service

You have now configured HAProxy with a 2048 bit set of custom Diffie-Hellman parameters that all frontends will be able to use. You have also suppressed the tune.ssl.default-dh-param warnings.

Conclusion

In this tutorial, you learned how to troubleshoot an HAProxy Setting tune.ssl.default-dh-param to 1024 by default warning message. You explored four different methods to examine HAProxy’s configuration to find the message. First you used systemctl and journalctl to examine the status of the HAProxy server and try to find the message in the systemd logs. Then you examined the HAProxy configuration file using the built in haproxy -c configuration check. Finally you checked for log entries using the grep command on HAProxy’s log file.

After you confirmed that your server was generating a Setting tune.ssl.default-dh-param to 1024 by default warning message, you generated a custom Diffie-Hellman parameters file and configured HAProxy to use it by setting the ssl-dh-param-file configuration directive.

Now that you have that configuration in place, your HAProxy server will default to using the more secure Diffie-Hellman parameters, and the tune.ssl.default-dh-param warning message will no longer be in your logs.

Introduction

HAProxy, which stands for High Availability Proxy, is a widely used TCP and HTTP-based proxy server that runs on Linux, Solaris, and FreeBSD. It is used to load balance applications by distributing requests between multiple servers, and to ensure that applications are highly available for users.

By default on many CentOS 8 installations, HAProxy is not configured to write its log output to a file. This quickstart tutorial will explain how to configure HAProxy logging with Rsyslog by using a Unix domain socket for reliability, speed, and security.

Prerequisites

To complete this tutorial, you will need a server running CentOS 8. This server should have a non-root user with administrative privileges. To set this up, follow the Initial Server Setup guide for CentOS 8 tutorial.

Step 1 — Installing and Enabling HAProxy

To install HAProxy, run the following dnf command:

sudo dnf install haproxy

When you are prompted Is this ok [y/N]: enter y and press RETURN to install the package.

Once you have installed HAProxy, enable and start it using the systemctl command:

sudo systemctl enable haproxy.service

You should receive the following output:

Output
Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

With the HAProxy service now enabled, you can start it up to ensure that it runs with a default configuration on your server:

sudo systemctl start haproxy.service

Next examine HAProxy’s status to make sure it is running:

sudo systemctl status haproxy.service

You should receive output like the following. Note the highlighted active (running) portion of the output. If your server shows the same highlighted section then HAProxy is running correctly on your server and you can proceed with configuring logging.

Output
● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2020-09-09 21:16:39 UTC; 4min 39s ago
   Process: 21340 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
   Main PID: 21341 (haproxy)
      Tasks: 2 (limit: 2881)
     Memory: 2.7M
     CGroup: /system.slice/haproxy.service
             ├─21341 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
. . .

If your output is different, or the status shows something like Active: failed, then follow the troubleshooting steps in the How to Troubleshoot Common HAProxy Errors tutorial to determine what is preventing HAProxy from starting correctly.

Once you have confirmed that HAProxy is enabled and running, you can continue to the next step, which is configuring HAProxy’s logging directives.

Step 2 — Configuring HAProxy Logging Directives

To configure HAProxy’s logging directives, open /etc/haproxy/haproxy.cfg in vi or your preferred editor:

sudo vi /etc/haproxy/haproxy.cfg

Press i to switch to INSERT mode, then find the line log 127.0.0.1 local2 and comment it out by adding a # character to the beginning of the line, as highlighted in the following example:

. . .
# 2) configure local2 events to go to the /var/log/haproxy.log
#   file. A line like the following can be added to
#   /etc/sysconfig/syslog
#
#    local2.*                       /var/log/haproxy.log
#
    #log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
. . .

Now add a line directly after the commented out line with the following contents:

    log         /dev/log local0

The entire section of /etc/haproxy/haproxy.cfg that you edited should contain the following lines:

. . .
#    local2.*                       /var/log/haproxy.log
#
     #log         127.0.0.1 local2
     log         /dev/log local0

     chroot      /var/lib/haproxy
     pidfile     /var/run/haproxy.pid
. . .

The chroot line is important, because it restricts the HAProxy process to accessing files in the /var/lib/haproxy directory only. The log /dev/log local0 line will create a file inside that directory that Rsyslog will use to collect log entries from.

Once you are finished editing the log lines in /etc/haproxy/haproxy.cfg, save and close the file by pressing ESC, typing :wq, and pressing ENTER.

The last step that you need to complete in this section is to create the /var/lib/haproxy/dev directory since it does not exist by default.

Create the directory using the mkdir command and then restart HAProxy:

sudo mkdir /var/lib/haproxy/dev
sudo systemctl restart haproxy.service

You have now configured HAProxy to send its logs to a Unix domain socket that resides in /var/lib/haproxy/dev/log. In the next step, you will configure Rsyslog to create and access the socket.

Step 3 — Configuring Rsyslog to Collect HAProxy Logs

Rsyslog’s default configuration on CentOS 8 does not handle HAProxy logs. To collect logs from the HAProxy service, open a new file /etc/rsyslog.d/99-haproxy.conf using vi or your preferred editor:

sudo vi /etc/rsyslog.d/99-haproxy.conf

Press i to switch to INSERT mode, then paste the following lines into the file:

$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
  /var/log/haproxy.log
  stop
}

The $AddUnixListenSocket directive tells Rsyslog to create a Unix domain socket in the specified location, in this case /var/lib/haproxy/dev/log. The :programname, startswith, "haproxy" section specifies the file where Rsyslog will write the log entries to that it collects from the socket.

Once you are finished editing /etc/rsyslog.d/99-haproxy.conf, save and close the file by pressing ESC, typing :wq, and pressing ENTER.

You have now configured Rsyslog to read log entries from the Unix domain socket in /var/lib/haproxy/dev/log and write them to a log file in /var/log/haproxy.log.

However, before restarting Rsyslog you will need to determine if SELinux is enforcing access control on your CentOS 8 system.

To check SELinux’s current policy, run the following:

getenforce

You will receive one of the following outputs:

• Enforcing - In this mode, SELinux is enforcing access controls on your system. You will need to complete the following optional Step 4 — Configuring SELinux section.
• Permissive - In this case, SELinux logs all access attempts to its log file, but does not enforce access controls on your system.
• Disabled - If SELinux is disabled, then it is not logging or enforcing any access control policies on your system.

If the getenforce command returned either Permissive or Disabled, then you can restart Rsyslog with the following command:

sudo systemctl restart rsyslog

Once you restart Rsyslog, you will be able to view logs in the /var/log/haproxy.log file that you configured in /etc/rsyslog.d/99-haproxy.conf. Proceed to Step 5 — Testing HAProxy Logging to make sure that everything is working as expected.

Otherwise, if your system is running SELinux in Enforcing mode, then the next section of this tutorial explains how to add a module to allow Rsyslog and HAProxy to communicate with each other over their shared Unix domain socket.

Step 4 — (Optional) Configuring SELinux

If your CentOS 8 system is configured with SELinux in Enforcing mode, then you will need to allow Rsyslog access to HAProxy’s chroot directory. Allowing this access will let Rsyslog create the Unix domain socket that HAproxy will send its logs to.

If you are not familiar with SELinux, this tutorial series An Introduction to SELinux on CentOS 7 will help you learn how to manage and interact with SELinux. Although it is written for CentOS 7, the principles and commands in the series are equally applicable to CentOS 8.

To enable Rsyslog and HAProxy access to their shared socket, the first task is to create a Type Enforcement policy file. Open a new file called rsyslog-haproxy.te in vi or your preferred editor:

vi rsyslog-haproxy.te

Press i to switch to INSERT mode, then paste the following lines into the file:

module rsyslog-haproxy 1.0;

require {
    type syslogd_t;
    type haproxy_var_lib_t;
    class dir { add_name remove_name search write };
    class sock_file { create setattr unlink };
}

#============= syslogd_t ==============
allow syslogd_t haproxy_var_lib_t:dir { add_name remove_name search write };
allow syslogd_t haproxy_var_lib_t:sock_file { create setattr unlink };

The first line defines the module name and version. The require portion tells the SELinux module loader about the types and classes that are required for the policy to be loaded as a module. The last two lines are the rules that allow Rsyslog access to HAProxy’s chroot and socket file respectively.

When you are done editing the file, save and close it by pressing ESC, typing :wq, and pressing ENTER.

Next, run the following command to install the checkpolicy package, which contains the checkmodule utility that you will use to turn the Type Enforcement file into an SELinux module.

sudo dnf install checkpolicy

Now that you have the checkmodule tool installed, the next step is to compile the module and then load it into SELinux. Run the following to compile the Type Enforcement file into an SELinux module:

checkmodule -M -m rsyslog-haproxy.te -o rsyslog-haproxy.mod

Next, run semodule_package to generate a complete policy package that SELinux can load into the Linux kernel:

semodule_package -o rsyslog-haproxy.pp -m rsyslog-haproxy.mod

The final step is to load the package that you generated into the Linux kernel using the semodule command:

sudo semodule -i rsyslog-haproxy.pp

Adding the module may take a few seconds. Once the command completes you can confirm that the module is loaded into the kernel by running the semodule command:

sudo semodule -l |grep rsyslog-haproxy

You should receive output like the following:

Output
rsyslog-haproxy

Once the module is loaded you can restart Rsyslog with the following command:

sudo systemctl restart rsyslog

You have now defined, compiled, and loaded an SELinux policy that will allow HAProxy and Rsyslog to communicate over their shared socket.

In the next step you will test that everything works by making an HTTP request to HAProxy and examining its new log file.

Step 5 — Testing HAProxy Logging

Now that you have configured HAProxy, Rsyslog, and optionally SELinux, you can test that logging to /var/log/haproxy.log is working.

By default the haproxy package ships with a configuration file that creates an HTTP listener socket on port 5000. The configuration points to a non-existent backend server, so any request to the port will result in an HTTP 503 error.

To check for a 503 error in your /var/log/haproxy.log file, first generate an HTTP request using curl like this:

curl -si http://127.0.0.1:5000

You should receive output like the following:

Output
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

Now examine /var/log/haproxy.log for any HTTP 503 responses using the grep command:

sudo grep -E 'NOSRV.+503' /var/log/haproxy.log

You should receive a line (or multiple lines) like the following:

Output
Sep 9 21:32:22 centos-s-1vcpu-1gb-nyc3-01 haproxy[4451]: 127.0.0.1:56024 [9/Sep/2020:21:32:22.098] main app/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

This line corresponds to the curl request that you made, which means that Rsyslog and HAProxy are configured to use their shared socket correctly.

Conclusion

In this quickstart tutorial, you configured HAProxy to log to a Unix domain socket. You also set up Rsyslog to create and read from the socket so that the two programs can communicate with each other without opening up any TCP/IP ports on your system. Finally, you optionally compiled, packaged, and loaded an SELinux policy to allow Rsyslog and HAProxy shared access to their socket.