How To Monitor Nagios Alerts with Alerta on Ubuntu 16.04

Introduction

Alerta is a web application used to consolidate and de-duplicate alerts from multiple monitoring systems and visualize them on a single screen. Alerta can integrate with many well-known monitoring tools like Nagios, Zabbix, Sensu, InfluxData Kapacitor, and many others.

In this tutorial you’ll set up Alerta and configure it to display notifications from Nagios, the popular open-source monitoring system.

Prerequisites

To follow this tutorial, you will need:

• Two Ubuntu 16.04 servers set up by following the Ubuntu 16.04 initial server setup guide, including a sudo non-root user and a firewall.
• On the first Ubuntu server, which is where you’ll run Nagios, install the following components:
  • Apache, MySQL, and PHP, by following the tutorial How To Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu 16.04.
  • Nagios 4, installed by following the tutorial How To Install Nagios 4 and Monitor Your Servers on Ubuntu and its prerequisites to configure it.
• On the second Ubuntu server, which is where we’ll install Alerta in this tutorial, install the following components:
  • Nginx, installed by following the tutorial How To Install Nginx on Ubuntu 16.04.
  • MongoDB, installed by following the tutorial How to Install MongoDB on Ubuntu 16.04.
  • Alerta, installed by following steps 1 through 6 in the tutorial How To Monitor Zabbix Alerts with Alerta on Ubuntu 16.04.

Step 1 — Installing the Nagios-to-Alerta Gateway

You can extend Nagios’ functionality with Nagios Event Broker (NEB) modules. NEB is Nagios’ event integration mechanism, and NEB modules are shared libraries that let you integrate other services with Nagios. In this step, we’ll install the Nagios to Alerta Gateway, the NEB module that will send notifications to Alerta.

Log into your Nagios server as your non-root user:

ssh sammy@your_nagios_server_ip

The Nagios to Alerta Gateway does not have preconfigured system packages, so you’ll have to build it from source. To do that, you’ll need to install some development tools and files. You’ll also need Git installed so you can fetch the source code from GitHub.

sudo apt-get install -y git curl gcc make libcurl4-openssl-dev

With the prerequisites installed, use Git to clone the source code from the project’s GitHub repository:

git clone https://github.com/alerta/nagios-alerta.git

Then change to the new nagios-alerta directory:

cd nagios-alerta

Then compile the nagios-alerta module using make:

make nagios4

You’ll see the following output:

Output
cd ./src && make nagios4
make[1]: Entering directory '/home/sammy/nagios-alerta/src'
gcc -fPIC -g -O2 -DHAVE_CONFIG_H -I../include -I../include/nagios4 -lcurl -o alerta-neb.o alerta-neb.c -shared  -lcurl
make[1]: Leaving directory '/home/sammy/nagios-alerta/src'

If you see something different, ensure you have all of the prerequisites installed.

Now run the installation task:

sudo make install

You’ll see this output, indicating the module was installed in /usr/lib/nagios:

Output
cd ./src && make install
make[1]: Entering directory '/home/sammy/nagios-alerta/src'
[ -d /usr/lib/nagios ] || mkdir /usr/lib/nagios
install -m 0644 alerta-neb.o /usr/lib/nagios
make[1]: Leaving directory '/home/sammy/nagios-alerta/src'

With the module installed, we can configure Nagios to use this new module.

Step 2 — Configure Nagios-to-Alerta Gateway

Let’s configure Nagios to send notification messages to Alerta.

First, enable the newly installed Alerta broker module in the Nagios main configuration file. Open the Nagios configuration file in your editor:

sudo vi /usr/local/nagios/etc/nagios.cfg

Find the section which contains the broker_module directives:

...
# EVENT BROKER MODULE(S)
# This directive is used to specify an event broker module that should
# by loaded by Nagios at startup.  Use multiple directives if you want
# to load more than one module.  Arguments that should be passed to
# the module at startup are separated from the module path by a space.
#
[...]
#broker_module=/somewhere/module1.o
#broker_module=/somewhere/module2.o arg1 arg2=3 debug=0
...

To configure the Alerta module, you need to provide two mandatory arguments:

• URL: The address which is used to communicate with the Alerta API. You configured this in Step 3 of the tutorial How To Monitor Zabbix Alerts with Alerta on Ubuntu 16.04.
• key: The API key you created in step 4 of the tutorial How To Monitor Zabbix Alerts with Alerta on Ubuntu 16.04. You need this to authenticate with Alerta and post events.

Add this line to the file to configure the Alerta integration:

...
broker_module=/usr/lib/nagios/alerta-neb.o http://your_alerta_server_ip/api key=ALERTA_API_KEY
...

There are some additional optional arguments you can specify as well:

• env: This specifies the environment name. The default environment name is Production.
• hard_only: Forwards results in Hard state only. You can find more info about Nagios State Types in the Nagios documentation. Set this to 1 to enable this mode.
• debug: - enable debug mode for the module. Set this to 1 to enable this mode.

To specify all of these options, use this line instead:

...
broker_module=/usr/lib/nagios/alerta-neb.o http://your_alerta_server_ip/api key=ALERTA_API_KEY env=Production hard_only=1 debug=1
...

Save the file and exit the editor.

In order to identify alerts by environment and service name, you’ll need to set up environment and service names using Nagios Custom Object Variables. To do this, use the _Environment and _Service variables in your configuration. Let’s configure those now.

Open the default Nagios host object configuration file, which you’ll find in the /usr/local/nagios/etc/objects/ directory:

sudo vi /usr/local/nagios/etc/objects/localhost.cfg

We’ll mark all alerts with this host as Production alerts, and we’ll call the default service Nagios. Find the following host definition:

...
define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

...

Add the _Environment and _Service values to the configuration:

...
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        _Environment            Production
        _Service                Nagios
        }
...

Now mark all the events associated with a lack of space on the system partitio as System alerts. Locate this section of the file which defines how to check for free space:

...
define service{
        use                             local-service         ; Name of service template to use
        host_name                       localhost
        service_description             Root Partition
        check_command                   check_local_disk!20%!10%!/
        }
...

Modify it to associate it with the System service:

...
define service{
        use                             local-service         ; Name of service template to use
        host_name                       localhost
        service_description             Root Partition
        check_command                   check_local_disk!20%!10%!/
        _Service                        System
        }
...

Save the file and exit the editor. Restart Nagios to apply these new settings:

sudo systemctl restart nagios.service

Check the Nagios log file to ensure that the service is running properly:

tail /usr/local/nagios/var/nagios.log

You’ll see the following output:

Output
...
[1505804481] [alerta] Initialising Nagios-Alerta Gateway module, v3.5.0
[1505804481] [alerta] debug is on
[1505804481] [alerta] states=Hard (only)
[1505804481] [alerta] Forward service checks, host checks and downtime to http://your_alerta_server_ip/api
[1505804481] Event broker module '/usr/lib/nagios/alerta-neb.o' initialized successfully.
[1505804481] Successfully launched command file worker with pid 25416

Now Nagios will send a notification as soon as any system or service goes off. Let’s generate a test event.

Step 3 — Generating a Test Alert to Verify Nagios-Alerta Integration

Let’s generate a test alert to ensure everything is connected. By default, Nagios keeps track of the amount of free disk space on your server. We’ll create a temporary file that’s large enough to trigger Nagios’ file system usage alert.

First, determine how much free space you have on the Nagios server. You can use the df command to find out:

df -h

You’ll see output like the following:

Output
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/vda1        20G  3.1G   16G   17% /

Look at the amount of free space available. In this case, the free space is 16GB. Your free space may differ.

Use the fallocate command to create a file that takes up more than 80% of the available disk space, which should be enough to trigger the alert:

fallocate -l 14G /tmp/temp.img

Within a few minutes, Nagios will trigger an alert about the amount of free disk space and will send the notification message to Alerta. You will see this new notification in the Alerta dashboard:

Now that you know the alerts are working, delete the temporary file you created so you can reclaim your disk space:

rm -f /tmp/temp.img

After a minute Nagios will send the recovery message. The alert will then disappear from the main Alerta dashboard, but you can view all closed events by selecting Closed.

You can click on the event row to view more details.

Conclusion

In this tutorial, you configured Nagios to send notifications to another server running Alerta.

Alerta gives you a convenient place to track alerts from many systems. For example, if some parts of your infrastructure use Nagios and others use Zabbix, you can merge notifications from both systems into one panel.