How To Troubleshoot Common HAProxy Errors

Introduction

There are three main commands, and a common log location that you can use to get started troubleshooting HAProxy errors. Generally when you are troubleshooting HAProxy, you will use these commands in the order indicated here, and then examine the log file for specific diagnostic data.

The commands and log that you will commonly use to troubleshoot HAProxy across most Linux distributions are:

• systemctl - Used to control and interact with Linux services via the systemd service manager.
• journalctl - Used to query and view the logs that are generated by systemd.
• haproxy - When troubleshooting, this command is used to check HAProxy’s configuration.
• /var/log/haproxy.log - This file contains log entries from HAProxy itself detailing TCP and HTTP traffic that is being handled by the server.

These commands, how to use them, and HAProxy’s logs where you can find additional information about errors are described in further detail in the following sections.

systemctl Commands for HAProxy

To troubleshoot common HAProxy errors using the systemd service manager, the first step is to inspect the state of the HAProxy processes on your system. The following systemctl commands will query systemd for the state of HAProxy’s processes on most Linux distributions.

sudo systemctl status haproxy.service -l --no-pager

The -l flag will ensure that output is not truncated or ellipsized. The --no-pager flag will make sure that output will go directly to your terminal without requiring any interaction on your part to view it. If you omit the --no-pager flag you will be able to scroll through the output using arrow keys, or the page up and down keys. To quit from the pager use the q key. You should receive output like this:

Output
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-08-20 19:30:11 UTC; 5s ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
 Main PID: 488 (haproxy)
    Tasks: 2 (limit: 2344)
. . .
Aug 19 21:31:46 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

Your output may be slightly different depending on which Linux distribution you are using, but in any case, make a note of the Active line in the output. If your HAProxy server does not show active (running) as highlighted in the example output but you expect it should, there may be an error. Typically if there is a problem, you will have a line like the following in your output (note the highlighted failed portion):

Example Error Output
   Active: failed (Result: exit-code) since Thu 2020-08-20 19:32:26 UTC; 6s ago

If there is a problem with your HAProxy process or configuration you can troubleshoot it further using the journalctl command.

journalctl Commands for HAProxy

To inspect the systemd logs for HAProxy, you can use the journalctl command. The systemd logs for HAProxy will usually indicate whether there is a problem with starting or managing the HAProxy process.

These logs are separate from HAProxy’s request and error logs. journalctl displays logs from systemd that describe the HAProxy service itself, from startup to shutdown, along with any process errors that may be encountered along the way.

sudo journalctl -u haproxy.service --since today --no-pager

The --since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors. You should receive output like the following (there may be a few extra lines between the Starting and Started lines depending on your Linux distribution):

Output
Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Starting HAProxy Load Balancer...
. . .
Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

If there is an error, you will have a line in the output that is similar to the following, with the main difference between Linux distributions being the highlighted yourhostname portion:

Example Error Output
Aug 20 19:32:25 yourhostname systemd[1]: Failed to start HAProxy Load Balancer.

If your HAProxy server has errors in the journalctl logs like the previous example, then the next step to troubleshoot possible issues is investigating HAProxy’s configuration using the haproxy command line tool.

Troubleshooting with haproxy

To troubleshoot HAProxy configuration issues, use the haproxy -c command. The tool will parse your HAProxy files and detect any errors or missing settings before attempting to start the server.

Run the command like this on Ubuntu, Debian, CentOS, and Fedora based distributions. Be sure to change the path to the configuration file if you are using a different filename or location:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

A working HAProxy configuration will result in output like the following:

Output
Configuration file is valid

If there is an error in your HAProxy configuration, like a typo or misplaced directive, haproxy -c will detect it and attempt to notify you about the problem.

For example, attempting to use the bind directive in haproxy.cfg in the wrong location will result in messages like the following:

Example Error Output
[ALERT] 232/194354 (199) : parsing [/etc/haproxy/haproxy.cfg:13] : unknown keyword 'bind' in 'global' section
[ALERT] 232/194354 (199) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 232/194354 (199) : Fatal errors found in configuration.

In this example the bind directive is misplaced inside a global configuration section, so HAProxy generates the unknown keyword error. The message also includes a line number 13, so that you can edit the file and fix or remove the erroneous line without having to search through the file.

Learning how to use haproxy -c to detect and fix errors is useful when you are troubleshooting an existing error, or before you reload HAProxy with an edited configuration that may contain errors.

HAProxy Log Files

HAProxy log files are a very helpful resource for troubleshooting. Generally, any error that you receive in a browser or other HTTP client will have a corresponding entry in HAProxy’s logs. Sometimes HAProxy will also output errors related to configuration and other debugging information to its log files.

On Ubuntu and Debian based Linux distributions, the haproxy package includes scripts that configure log output in /var/log/haproxy.log.

On CentOS, Fedora, and other RedHat-derived Linux distributions, haproxy does not output to a log file by default. To log HAProxy output logs to /var/log/haproxy.log, follow this quickstart tutorial, How To Configure HAProxy Logging with Rsyslog on CentOS 8.

When you are troubleshooting HAProxy using its log file, examine /var/log/haproxy.log for errors using a tool like tail or less. For example, to view the last two lines of the log using tail, run the following command:

sudo tail -n 2 /var/log/haproxy.log

An example error will resemble something like the following lines, regardless of which Linux distribution you are using to run your HAProxy server:

Log Examples
Aug 20 19:36:21 d6cdd0c71489 haproxy[19202]: [ALERT] 258/134605 (19202) : Proxy 'app', server 'app1' [/etc/haproxy/haproxy.cfg:88] verify is enabled by default but no CA file specified. If you're running on a LAN where you're certain to trust the server's certificate, please set an explicit 'verify none' statement on the 'server' line, or use 'ssl-server-verify none' in the global section to disable server-side verifications by default.
Aug 20 19:36:22 d6cdd0c71489 haproxy[4451]: 203.0.113.1:54428 [20/Aug/2020:19:36:22.288] main app/<NOSRV> 0/-1/-1/-1/1 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

These example lines are just for illustration purposes. If you are diagnosing errors with your HAProxy server, chances are the lines in your logs will have different contents than these. Some lines will include success responses and other non-critical diagnostic entries.

Regardless of your Linux distribution, the format of the lines in your HAProxy logs will include any HTTP status codes that are returned to clients, along with requesting IPs and the status of backend servers.

Once you have an idea of what might be causing problems with your HAProxy server you can continue researching and troubleshooting the issue. The HTTP status code and text description are especially useful, since they give you explicit and specific terms that you can use to narrow down the range of possible causes of a problem.

Conclusion

Troubleshooting HAProxy errors can range from diagnosing errors with the service itself, to locating misconfigured options for modules, or to examining customized access control rules in detail. This introduction to diagnosing issues with HAProxy explained how to use a number of utilities to help narrow down the possible causes of errors. Usually, you will use these utilities in the same order, although you can always skip some, or start directly with examining logs if you have a general idea of what the problem might be.

However, as a general sequence for troubleshooting, it helps to be methodical and use these tools in the order described. Start troubleshooting with systemctl to examine the state of the HAProxy server. If you need more information, examine the systemd logs for HAProxy using the journalctl command. If the issue is still not apparent after checking journalctl, testing HAProxy’s configuration using haproxy -c -f /etc/haproxy/haproxy.cfg is the next step. Finally, for in-depth troubleshooting, examining HAProxy’s log files will usually indicate a specific error, with helpful diagnostic messages and error codes.

The rest of the tutorials in this series will examine some common errors that you may encounter when using HAProxy in more detail.