How To Set Up a Serf Cluster on Several Ubuntu VPS

Introduction

---

Serf is a decentralized service orchestration and service discovery tool. It is extremely fault tolerant and decentralized, with no single point of failure like other similar tools. Serf can be used to trigger any event across a cluster of systems as well as perform monitoring duties. It’s built on top of the Gossip protocol which is designed for decentralized communication. In order for a node to join a Serf cluster, the node only needs to initially know the address of one other node in the cluster. Once the node joins, all membership information is propagated throughout the cluster. The Gossip protocol makes Serf extremely easy to setup and configure.

Using multiple VPS

---

Serf is designed to run across multiple VPS and machines and is compatible with nix, windows, and Mac OS systems. This tutorial will show you how to setup Serf on two different Ubuntu servers.

In this tutorial, the servers will be named SerfNode1 and SerfNode2. You will need to know the IP address of each server; the following IP addresses are used to represent each VPS in this tutorial. [Wherever you see these IP addresses in the tutorial, you will replace them with your own IP address].

SerfNode1 | 1.1.1.1

SerfNode2 | 2.2.2.2

---

Installing Serf

---

This will need to be done on both SerfNode1 and SerfNode2

Download the latest Serf package:

wget https://dl.bintray.com/mitchellh/serf/0.3.0_linux_amd64.zip

Install the unzip tool to unzip the package:

apt-get install unzip

Unzip the Serf package:

unzip 0.3.0_linux_amd64.zip

Add Serf to the binaries directory so that it can be executed from anywhere:

mv serf /usr/local/bin

Creating a Serf cluster

---

Start the first Serf node on SerfNode1:

serf agent -node=**SerfNode1** -bind=1.1.1.1:7496 

You should see something similar to the below output:

==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
    Node name: '**SerfNode1**'
    Bind addr: '1.1.1.1:7496'
     RPC addr: '127.0.0.1:7373'
    Encrypted: false
     Snapshot: false
      Profile: lan

==> Log data will now stream in as it occurs:

    2014/01/18 21:57:57 [INFO] Serf agent starting
    2014/01/18 21:57:57 [WARN] Binding to public address without encryption!
    2014/01/18 21:57:57 [INFO] serf: EventMemberJoin: **SerfNode1** 1.1.1.1
    2014/01/18 21:57:58 [INFO] agent: Received event: member-join
    

Note: The node parameter specifies the node’s name and the bind represents the IP address and port to bind on.

On SerfNode2 we will start the Serf agent in the background:

serf agent -node=**SerfNode2** -bind=2.2.2.2:7497 -rpc-addr=127.0.0.1:7373 &

Note: The ‘&’ tells the command to execute in the background

Tell SerfNode2 to join SerfNode1:

serf join 1.1.1.1:7496

You should see output similar to the following:

...
    2014/01/18 22:03:04 [INFO] serf: EventMemberJoin: **SerfNode2** 2.2.2.2
    2014/01/18 22:03:05 [INFO] agent: Received event: member-join1922
...

Awesome! You now have a small working Serf cluster. In order to setup additional servers you simply repeat the process we did on SerfNode2. In order to join a Serf cluster you just need to instruct the VPS to join one other Serf agent already in the cluster. The Gossip protocol automatically notifies all other Serf agents in the cluster of the new VPS.

Events and Event Handling

---

Another reason Serf is so awesome is how easy the event handling is. Let’s first send an event to the cluster.

Sending a simple user event

---

On SerfNode2, execute the following command:

serf event hello

On SerfNode1, you should see something similar to this:

2014/01/16 15:48:05 [INFO] agent: Received event: user-event: hello

Woot! We just sent our first event to the cluster. Ok, that is cool and all, but this event doesn’t really do much.

Creating a custom event handler

---

Now we will configure some custom event handling. Serf can trigger custom events across a cluster in order to initiate things such as deployments, security updates, system configuration, etc. Any event on a Linux machine that can be scripted, Serf can trigger it.

Let’s start with a simple example.

On SerfNode1, stop the Serf agent by pressing Ctrl + C. It should give you the output below:

    2014/01/16 15:58:54 [INFO] agent: requesting serf shutdown
    2014/01/16 15:58:54 [WARN] Shutdown without a Leave
    2014/01/16 15:58:54 [INFO] agent: shutdown complete

Now we will create a custom event script that writes “written to file” to a text file within the /usr/src directory. When the user sends the ‘write’ event it will execute this script.

First let’s create our event handler. The event handler can be any executable file – in our case we will be using a bash file.

Switch to the /usr/src directory:

cd /usr/src

Open up nano:

nano handler.sh

Use the following script for the event handler:

 #!/bin/bash
if [ "${SERF_USER_EVENT}" = "write" ]; then
        echo "written to file" >> test.txt
fi

echo "${SERF_USER_EVENT}"

Note: The ${SERF_USER_EVENT} is the name of the evnt that we are sending. Notice how you can use if statements to setup different events.

Press Ctrl + X to exit nano

Press Y to save

Hit enter

Make the script executable:

chmod +x handler.sh

Now we will restart the Serf agent, but this time we will use the event handler we just created:

serf agent -log-level=debug -event-handler=./handler.sh -node=**SerfNode1** -bind=1.1.1.1:7496

(OPTIONAL STEP)

Each Serf agent can have its own event handler. If you wanted to have a custom event handler for SerfNode2, you would simply do the same event handler creation process you did for SerfNode1, or you could just copy the event handler script over to the SerfNode2 server’s /usr/src directory and execute the following commands:

On SerfNode2, leave the Serf cluster:

serf leave

Navigate to the /usr/src directory:

cd /usr/src

Start the Serf agent in the background with the custom event handler:

serf agent -log-level=debug -event-handler=./handler.sh -node=**SerfNode2** -bind=2.2.2.2:7496 &

Testing the event handler

---

On SerfNode2, rejoin SerfNode1:

serf join 1.1.1.1:7496

Execute the following command:

serf event write 

On SerfNode1 switch to the /usr/src directoy:

cd /usr/src

Now you should see the test.txt file in the directoy. This file was created when we triggered the Serf event from SerfNode2. Pretty nifty eh? Now let’s do something a little more advanced.

Setting up free memory monitoring

---

We are going to setup a custom event handler that will log the free memory on a cluster of servers to a central server.

On SerfNode1, press Ctrl+C to leave the Serf cluster.

Make sure you are in the /usr/src directory:

cd /usr/src

Open up the handler.sh script:

nano handler.sh

Change the script to the following:

 #!/bin/bash
if [ "${SERF_USER_EVENT}" = "mem" ]; then
   serf event memresponse "$(awk '/MemTotal/ {printf( "%.2f\n", $2 / 1024 ) }'              /proc/meminfo) MB from $(wget -qO- http://ipecho.net/plain ; echo) at $(date)"
fi

Press Ctrl + X to exit nano

Press Y to save

Hit enter

This script will trigger a Serf event that will return the free memory on the virtual server with the following format:

490 MB from 1.1.1.1 at Sun Jan 19 00:37:22 EST 2014

Now we need a way to log this on a VPS

On SerfNode2, leave the Serf cluster:

serf leave

Make sure you are in the /usr/src directory:

cd /usr/src

Create a handler script:

nano handler.sh

Use the following for the script:

 #!/bin/bash
if [ "${SERF_USER_EVENT}" = "memresponse" ]; then
    cat >> mem.txt
    echo "\n" >> mem.txt
fi

Press Ctrl + X to exit nano

Press Y to save

Hit enter

Make the script executable:

chmod +x handler.sh

Start the agent on SerfNode1:

serf agent -log-level=debug -event-handler=./handler.sh -node=**SerfNode1** -bind=1.1.1.1:7496

Start the agent on SerfNode2:

serf agent -log-level=debug -event-handler=./handler.sh -node=**SerfNode2** -bind=2.2.2.2:7496 &

On SerfNode2, rejoin SerfNode12:

serf join 1.1.1.1:7496

Trigger the ‘mem’ event:

serf event mem

Check the mem.txt file:

nano mem.txt

Now you have a functioning memory monitoring tool that can be distributed across multiple virtual servers.

Serf events in detail

---

Below are some of the variables that come in handy when creating custom event handling scripts. These are taken directly from the <a href=“http://serfdom.io”>Serf website.</a>

• SERF_EVENT is the event type that is occurring. This will be one of member-join, member-leave, member-failed, or user.

• SERF_SELF_NAME is the name of the node that is executing the event handler.

• SERF_SELF_ROLE is the role of the node that is executing the event handler.

• SERF_USER_EVENT is the name of the user event type if SERF_EVENT is “user”.

• SERF_USER_LTIME is the LamportTime of the user event if SERF_EVENT is “user”.

When an event is triggered, the following is the layout of the event command:

serf event [SERF_EVENT_NAME] [PAYLOAD]

• The payload is anything following the event name. The payload is interpreted as stdin by the script.

• When a custom user event is used, the SERF_USER_EVENT variable should be used instead of the SERF_EVENT variable.

Conclusion

---

Serf is a great way to trigger events across a cluster of machines. It is simple, lightweight, and fault tolerant. In addition to these great features, it is extremely decentralized and has no single point of failure. Some example use cases are: system configuration, deployments, security updates, message broadcasting, and server monitoring. Serf is also extremely customizable and can be adapted to be a solution to a wide range of problems.

More information and documentation on Serf can be found <a href=“http://www.serfdom.io”>here</a>.

<div class=“author”>Submitted by: <a href=“http://blog.opendev.io”>Cooper Thompson</a></div>