LEMP Stack Monitoring with Monit on Ubuntu 14.04

Introduction

Monit is a free and open source service monitoring application which can perform various event-based actions. Monit can send email notifications, restart a service or application, or take other responsive actions.

This tutorial will build on a basic LEMP stack (Linux, Nginx, MySQL, PHP). Monit will be incorporated to monitor all services in the stack and alert the root user of any adverse conditions.

An optional external Monit server can also be used for remote monitoring of a web application or other services.

Prerequisites

• Before we begin, you will first need to set up a Ubuntu 14.04 Droplet
• You will need a standard user account which has sudo privileges
• This tutorial adds Monit to an existing LEMP stack. For a tutorial on how to create an initial LEMP stack, see How To Install Linux, nginx, MySQL, PHP (LEMP) stack on Ubuntu 14.04
• Optional: If you want to monitor a remote website, DNS, or mail server, you should have that server set up with a publicly accessible domain or IP address (more in Step 6)

Step 1 — Configure Email Delivery for Monit Notifications

Part of system monitoring typically involves email notifications for alerts. As such, proper email delivery must be in place in order for Monit to send email notifications. A typical Monit alert email will look similar to this:

From: monit@example.com 
To: root@yourserver.com

Resource limit matched Service example.com

        Date:        Mon, 22 Dec 2014 03:04:06
        Action:      alert
        Host:        example.com
        Description: cpu user usage of 79.8% matches resource limit [cpu user usage>70.0%]

Your faithful employee,
Monit

This tutorial will set up Monit to send you an email every time an alert is triggered.

Note: Monit’s notifications will likely go to your spam folder by default. Reverse DNS (known as a PTR record) must be properly configured to ensure the highest chance mail will be delivered successfully. Your Droplet’s hostname must match its fully qualified domain name (FQDN), so, for example, they could both be hostname.example.com. To edit the PTR record of a DigitalOcean Droplet, access the DigitalOcean Control Panel. Navigate to Settings and choose the Rename tab. Enter the new hostname and click Rename.

This guide assumes you do not have a preexisting mail transfer agent (MTA), so we will install Postfix. A local installation of Postfix allows the system to send notification emails to an external mail provider such as Gmail or Yahoo.

To begin installing Postfix as your MTA, first update the system’s repository source list.

sudo apt-get update

Then install the Postfix and GNU Mailutils packages from Ubuntu’s repositories.

sudo apt-get install postfix mailutils

Near the end of the installation, you will be prompted to select a server configuration type as shown in the screenshot below. Choose Internet Site.

When prompted for the System Mail Name, use the fully qualified domain name (FQDN) of your Droplet. Note: The System Mail Name can also be changed later in /etc/mailname.

Next, open the file /etc/aliases for editing. This guide will use Nano but you can use whichever text editor you prefer.

sudo nano /etc/aliases

Here we will add a personal email address where we’ll receive Monit’s notification emails. These mail notifications will come from our LEMP server’s root user.

postmaster: root
root: myemail@gmail.com

It is also possible to add multiple destinations if desired:

root: username, itstaff@mycompany.com, otherperson@other.com

Save your changes and exit Nano. Then run the following to update the aliases file:

sudo newaliases

A test message can be sent from your Droplet to check mail delivery. Please check spam folders if the test message is not first seen in your inbox.

echo test | mail -s "test message from my VPS" root

Step 2 — Install and Configure Monit

Monit is also available in the Ubuntu package repositories. For a brief reference guide on Monit, please see this tutorial.

Monit can be installed on your LEMP server with:

sudo apt-get install monit

On Ubuntu 14.04, the Monit configuration files are located in /etc/monit/ and the main Monit configuration file is /etc/monit/monitrc.

To open monitrc in Nano for editing:

sudo nano /etc/monit/monitrc

Uncomment the following lines and change them to match what’s shown below:

set mailserver localhost	#Use localhost for email alert delivery.

set mail-format {
      from: monit@$HOST
   subject: monit alert --  $EVENT $SERVICE
   message: $EVENT Service $SERVICE
                 Date:        $DATE
                 Action:      $ACTION
                 Host:        $HOST
                 Description: $DESCRIPTION

            Your faithful employee,
            Monit
}

set alert root@localhost not on { instance, action }	#Set email address to receive alerts. This guide uses root mail.

Still in the monitrc file, now uncomment the following lines and change example.com to match your server’s domain or IP address.

check system example.com
    if loadavg (1min) > 4 then alert
    if loadavg (5min) > 2 then alert
    if memory usage > 75% then alert
    if swap usage > 25% then alert
    if cpu usage (user) > 70% then alert
    if cpu usage (system) > 30% then alert
    if cpu usage (wait) > 20% then alert

We’ll also add this entry at the end of the file:

check filesystem rootfs with path /	#Alert if low on disk space.
    if space usage > 90% then alert

Save your changes and exit Nano.

Step 3 — Configure Service Monitoring for LEMP Services in Monit

On Ubuntu 14.04, Monit configurations can be specified directly in the /etc/monit/monitrc file or via individual files in /etc/monit/conf.d/. In this tutorial, individual files will be created under the /etc/monit/conf.d/ directory.

First we will provide Monit with the means to manage a service. For the sake of simplicity in this tutorial, we will place all process monitoring into a single file located at /etc/monit/conf.d/lemp-services. Using the following entries, Monit will watch Nginx, MySQL and PHP-FPM, and restart these services if they are abnormally stopped for any reason.

We can create the working file with Nano:

sudo nano /etc/monit/conf.d/lemp-services

Add the following entries for the services in our LEMP stack:

check process nginx with pidfile /var/run/nginx.pid
    group www-data
    start program = "/etc/init.d/nginx start"
    stop program = "/etc/init.d/nginx stop"
    
check process mysql with pidfile /var/run/mysqld/mysqld.pid
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"
        
check process php5-fpm with pidfile /var/run/php5-fpm.pid
    start program = "/etc/init.d/php5-fpm start"
    stop program = "/etc/init.d/php5-fpm stop"

Then save your changes.

Step 4 — Add Actions for Restarting Unhealthy LEMP Services

Now that Monit is able to manage select services, actions can be added to restart services as desired. For example, Monit has the ability to monitor TCP connections. If the server is no longer proving HTTP connections, Monit can restart PHP-FPM or Nginx to automatically resolve the issue.

To build on our existing configuration, we’ll now further edit /etc/monit/conf.d/lemp-services. The additions we’ll make below are shown in red and we’ll tell Monit to restart Nginx and PHP-FPM if HTTP connections are no longer available. Additionally, we will have Monit restart MySQL if the socket is unavailable.

Note: Make sure to use your Droplet’s domain or IP address where you see example.com in the first and third entries.

check process nginx with pidfile /var/run/nginx.pid
    group www-data
    start program = "/etc/init.d/nginx start"
    stop program = "/etc/init.d/nginx stop"
    if failed host example.com port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout
    
check process mysql with pidfile /var/run/mysqld/mysqld.pid
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"
    if failed unixsocket /var/run/mysqld/mysqld.sock then restart
    if 5 restarts within 5 cycles then timeout
    
check process php5-fpm with pidfile /var/run/php5-fpm.pid
    start program = "/etc/init.d/php5-fpm start"
    stop program = "/etc/init.d/php5-fpm stop"
    if failed host example.com port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout

Save your changes and close Nano. Then restart Monit to apply the configuration changes you’ve made so far.

sudo service monit restart

Step 5 (Optional) — Monitor Logs for Errors and Keywords

Monit can also monitor logs for specific keywords and then perform an action or send an alert. This is helpful in cases where a web application is having trouble or when a team requires notification of a particular traceback or event from logs.

Below is an Nginx log example with a timeout error which Monit can monitor for, and alert of:

2014/12/22 11:03:54 [error] 21913#0: *202571 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2600:3c01::f03c:91ff:fe6e:5a91, server: example.com, request: "GET /wp-admin/admin-ajax.php?action=wordfence_doScan&isFork=1&cronKey=40cb51ccsdfsf322fs35 HTTP/1.0", upstream: "fastcgi://unix:/var/run/example.com.sock", host: "example.com"

Building on our existing configuration, open your LEMP services configuration file again in Nano.

sudo nano /etc/monit/conf.d/lemp-services

Add the following entry. This will send a notification when any timeouts occur from Nginx communicating with PHP-FPM.

check file nginx-error with path /var/log/nginx/error.log
    if match "^timed out" then alert

Save your change and close Nano. Then restart Monit for the change to take effect:

sudo service monit restart

Step 6 (Optional) — Use Monit to Monitor a Remote Website and Other Services

In addition to using Monit locally, Monit can watch a variety of external services and connections. In this example, we’ll use the local instance of Monit we’ve already set up, and add some new monitoring configurations for external services.

It is preferable to have an external Monit system in an entirely different data center for out-of-band purposes. If a web application is based in New York, having a small external Monit server in San Francisco would be ideal.

Below are examples of external Monit checks which can be implemented on a second host running Monit. These examples would be placed in the external server’s /etc/monit/conf.d/lemp-external file to remotely check our LEMP stack at remote-example.com.

Use Nano to create this configuration file:

sudo nano /etc/monit/conf.d/lemp-external

Monitor ICMP response and HTTP & HTTPS connectivity:

# ICMP check
check host remote-example.com with address remote-example.com
    if failed icmp type echo
        for 5 times within 5 cycles
        then alert

# HTTP check
    if failed 
          port 80 protocol http 
       for 5 times within 5 cycles
       then alert        

# HTTPS check
    if failed 
          port 443 type tcpSSL protocol http 
       for 5 times within 5 cycles
       then alert

Monitor DNS:

check host ns1.example.com with address ns1.example.com
    if failed port 53 type udp protocol dns then alert

Monitor SMTP:

check host smtp.example.com with address smtp.example.com
    if failed port 25 type tcp protocol smtp then alert

Monitor Healthcheck URL of Web Application

For web applications, Monit can also perform a particular request on a healthcheck URL. Below is an example of a site remote-example.com with a healthcheck URL of: https://remote-example.com/healthcheck.

check host remote-example.com with address remote-example.com
    if failed 
          port 443 type tcpSSL protocol http 
       request "/healthcheck"     
       for 5 times within 5 cycles
       then alert

Step 7 — Manage Monit From Command Line

Monit provides a command line utility as well. From there, simple commands can be used to check the overall Monit status and complete useful tasks such as temporarily starting or stopping monitoring.

In order to run Monit status checks from the command line, the Monit web service must be enabled. To do this, open /etc/monit/monitrc for editing in Nano.

sudo nano /etc/monit/monitrc

Uncomment the following lines to enable the web service locally:

set httpd port 2812 and
        use address localhost
        allow localhost

Save your changes and exit Nano. Then restart Monit:

sudo service monit restart

Now it is possible to check Monit’s status from the command line.

Below are commands to temporarily disable and enable monitoring:

sudo monit unmonitor all

sudo monit monitor all

Step 8 — View Reports

Let’s take a look at the reports for all the checks we’ve been setting up.

sudo monit status

Now you’ll see output for everything you’ve configured Monit to check, including local LEMP services, and any external checks:

sudo monit status
The Monit daemon 5.6 uptime: 0m 

System 'example.com'
  status                            Running
  monitoring status                 Monitored
  load average                      [0.00] [0.01] [0.05]
  cpu                               0.5%us 0.4%sy 0.0%wa
  memory usage                      115132 kB [22.9%]
  swap usage                        0 kB [0.0%]
  data collected                    Mon, 22 Dec 2014 16:50:42

Filesystem 'rootfs'
  status                            Accessible
  monitoring status                 Monitored
  permission                        755
  uid                               0
  gid                               0
  filesystem flags                  0x1000
  block size                        4096 B
  blocks total                      5127839 [20030.6 MB]
  blocks free for non superuser     4315564 [16857.7 MB] [84.2%]
  blocks free total                 4581803 [17897.7 MB] [89.4%]
  inodes total                      1310720
  inodes free                       1184340 [90.4%]
  data collected                    Mon, 22 Dec 2014 16:50:42

Process 'nginx'
  status                            Running
  monitoring status                 Monitored
  pid                               14373
  parent pid                        1
  uptime                            28m 
  children                          4
  memory kilobytes                  1364
  memory kilobytes total            9228
  memory percent                    0.2%
  memory percent total              1.8%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  port response time                0.018s to example.com:80 [HTTP via TCP]
  data collected                    Mon, 22 Dec 2014 16:50:42

Process 'mysql'
  status                            Running
  monitoring status                 Monitored
  pid                               12882
  parent pid                        1
  uptime                            32m 
  children                          0
  memory kilobytes                  44464
  memory kilobytes total            44464
  memory percent                    8.8%
  memory percent total              8.8%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  unix socket response time         0.000s to /var/run/mysqld/mysqld.sock [DEFAULT]
  data collected                    Mon, 22 Dec 2014 16:50:42

Process 'php5-fpm'
  status                            Running
  monitoring status                 Monitored
  pid                               17033
  parent pid                        1
  uptime                            0m 
  children                          2
  memory kilobytes                  13836
  memory kilobytes total            22772
  memory percent                    2.7%
  memory percent total              4.5%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  port response time                0.018s to example.com:80 [HTTP via TCP]
  data collected                    Mon, 22 Dec 2014 16:50:42

File 'nginx-error'
  status                            Accessible
  monitoring status                 Monitored
  permission                        644
  uid                               0
  gid                               0
  timestamp                         Mon, 22 Dec 2014 16:18:21
  size                              0 B
  data collected                    Mon, 22 Dec 2014 16:50:42

Remote Host 'example.com'
  status                            Online with all services
  monitoring status                 Monitored
  icmp response time                0.021s [Echo Request]
  port response time                0.107s to example.com:443 [HTTP via TCPSSL]
  port response time                0.062s to example.com:80 [HTTP via TCP]
  data collected                    Mon, 22 Dec 2014 16:50:42
  

Use this data to check the health of your services and see useful statistics.

Troubleshooting

If any problems arise, first check Monit’s logs located at /var/log/monit.log. This will give you more information as to the nature of the problem.

Example error log entries:

[UTC Dec 22 13:59:54] error    : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 14:10:16] error    : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 15:24:19] error    : 'example.com' failed protocol test [HTTP] at INET[example.com:80] via TCP -- HTTP: Error receiving data -- Resource temporarily unavailable
[UTC Dec 22 15:57:15] error    : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 17:00:57] error    : ICMP echo response for example.com 1/3 timed out -- no response within 5 seconds
[UTC Dec 22 17:49:00] error    : 'example.com' failed, cannot open a connection to INET[example.com:443/API] via TCPSSL

Conclusion

Having completed this guide, you should now have Monit configured for monitoring a LEMP stack on Ubuntu 14.04. Monit is quite extensible and can be easily customized or expanded for monitoring all kinds of services for small and large networks.

Below are some additional links for Monit:

• mmonit.com

• Configuration Examples

check process php7.0-fpm with pidfile /var/run/php/php7.0-fpm.pid
    start program = "/etc/init.d/php7.0-fpm start"
    stop program = "/etc/init.d/php7.0-fpm stop"

check network eth0 with interface eth0 
       if failed link then alert