How To Set Up a Production Elasticsearch Cluster on Ubuntu 14.04

This tutorial assumes that your servers are using a VPN like the one described here: How To Use Ansible and Tinc VPN to Secure Your Server Infrastructure. This will provide private network functionality regardless of the physical network that your servers are using.

If you are using a shared private network, you must use a VPN to protect Elasticsearch from unauthorized access. Each server must be on the same private network because Elasticsearch doesn’t have security built into its HTTP interface. The private network must not be shared with any computers you don’t trust.

We will refer to your servers’ VPN IP addresses as vpn_ip. We will also assume that they all have a VPN interface that is named “tun0”, as described in the tutorial linked above.

You will want to restrict outside access to your Elasticsearch instance, so outsiders can’t access your data or shut down your Elasticsearch cluster through the HTTP API. In other words, you must configure Elasticsearch such that it only allows access to servers on your private network (VPN). To do this, we need to configure each node to bind to the VPN IP address, vpn_ip, or interface, “tun0”.

Find the line that specifies network.host, uncomment it, and replace its value with the respective server’s VPN IP address (e.g. 10.0.0.1 for node01) or interface name. Because our VPN interface is named “tun0” on all of our servers, we can configure all of our servers with the same line:

network.host: [_tun0_, _local_]

Note the addition of “_local_”, which configures Elasticsearch to also listen on all loopback devices. This will allow you to use the Elasticsearch HTTP API locally, from each server, by sending requests to localhost. If you do not include this, Elasticsearch will only respond to requests to the VPN IP address.

Next, set the name of your cluster, which will allow your Elasticsearch nodes to join and form the cluster. You will want to use a descriptive name that is unique (within your network).

Find the line that specifies cluster.name, uncomment it, and replace its value with the your desired cluster name. In this tutorial, we will name our cluster “production”:

cluster.name: production

Next, we will set the name of each node. This should be a descriptive name that is unique within the cluster.

Find the line that specifies node.name, uncomment it, and replace its value with your desired node name. In this tutorial, we will set each node name to the hostname of server by using the ${HOSTNAME} environment variable:

node.name: ${HOSTNAME}

If you prefer, you may name your nodes manually, but make sure that you specify unique names. You may also leave node.name commented out, if you don’t mind having your nodes named randomly.

Next, you will need to configure an initial list of nodes that will be contacted to discover and form a cluster. This is necessary in a unicast network.

Find the line that specifies discovery.zen.ping.unicast.hosts and uncomment it. Replace its value with an array of strings of the VPN IP addresses or hostnames (that resolve to the VPN IP addresses) of all of the other nodes.

For example, if you have three servers node01, node02, and node03 with respective VPN IP addresses of 10.0.0.1, 10.0.0.2, and 10.0.0.3, you could use this line:

discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]

Alternatively, if all of your servers are configured with name-based resolution of their VPN IP addresses (via DNS or /etc/hosts), you could use this line:

discovery.zen.ping.unicast.hosts: ["node01", "node02", "node03"]

Your servers are now configured to form a basic Elasticsearch cluster. There are more settings that you will want to update, but we’ll get to those after we verify that the cluster is working.

Save and exit elasticsearch.yml.

Now start Elasticsearch:

sudo service elasticsearch restart

Then run this command to start Elasticsearch on boot up:

sudo update-rc.d elasticsearch defaults 95 10

Be sure to repeat these steps (Configure Elasticsearch Cluster) on all of your Elasticsearch servers.

To verify that mlockall is working on all of your Elasticsearch nodes, run this command from any node:

curl http://localhost:9200/_nodes/process?pretty

Each node should have a line that says "mlockall" : true, which indicates that memory locking is enabled and working:

Nodes process output:
...
  "nodes" : {
    "kQgZZUXATkSpduZxNwHfYQ" : {
      "name" : "es03",
      "transport_address" : "10.0.0.3:9300",
      "host" : "10.0.0.3",
      "ip" : "10.0.0.3",
      "version" : "2.2.0",
      "build" : "8ff36d1",
      "http_address" : "10.0.0.3:9200",
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 1650,
        "mlockall" : true
      }
...

If mlockall is false for any of your nodes, review the node’s settings and restart Elasticsearch. A common reason for Elasticsearch failing to start is that ES_HEAP_SIZE is set too high.

First, find the process ID (PID) of your Elasticsearch process. An easy way to do this is to use the ps command to list all of the processes that belong to the elasticsearch user:

ps -u elasticsearch

You should see output that looks like this. The number in the first column is the PID of your Elasticsearch (java) process:

Output:
  PID TTY          TIME CMD
11708 ?        00:00:10 java

Then run this command to show the open file limits for the Elasticsearch process (replace the highlighted number with your own PID from the previous step):

cat /proc/11708/limits | grep 'Max open files'

Output
Max open files            65535                65535                files

The numbers in the second and third columns indicate the soft and hard limits, respectively, as 64k (65535). This is OK for many setups, but you may want to increase this setting.

To increase the maximum number of open file descriptors in Elasticsearch, you just need to change a single setting.

Open the /etc/default/elasticsearch file for editing:

sudo vi /etc/default/elasticsearch

Find MAX_OPEN_FILES, uncomment it, and set it to the limit you desire. For example, if you want a limit of 128k descriptors, change it to 131070:

MAX_OPEN_FILES=131070

Save and exit.

Now restart Elasticsearch to put the changes into place:

sudo service elasticsearch restart

Then follow the previous subsection to verify that the limits have been increased.

Be sure to repeat this step on any of your Elasticsearch servers that require higher file descriptor limits.

Before configuring dedicated master nodes, ensure that your cluster will have at least 3 master-eligible nodes. This is important to avoid a split-brain situation, which can cause inconsistencies in your data in the event of a network failure.

To configure a dedicated master node, edit the node’s Elasticsearch configuration:

sudo vi /etc/elasticsearch/elasticsearch.yml

Add the two following lines:

node.master: true 
node.data: false

The first line, node.master: true, specifies that the node is master-eligible and is actually the default setting. The second line, node.data: false, restricts the node from becoming a data node.

Save and exit.

Now restart the Elasticsearch node to put the change into effect:

sudo service elasticsearch restart

Be sure to repeat this step on your other dedicated master nodes.

You can query the cluster to see which nodes are configured as dedicated master nodes with this command: curl -XGET 'http://localhost:9200/_cluster/state?pretty'. Any node with data: false and master: true are dedicated master nodes.

To configure a dedicated data node—a data node that is not master-eligible—edit the node’s Elasticsearch configuration:

sudo vi /etc/elasticsearch/elasticsearch.yml

Add the two following lines:

node.master: false 
node.data: true

The first line, node.master: false, specifies that the node is not master-eligible. The second line, node.data: true, is the default setting which allows the node to be a data node.

Save and exit.

Now restart the Elasticsearch node to put the change into effect:

sudo service elasticsearch restart

Be sure to repeat this step on your other dedicated data nodes.

You can query the cluster to see which nodes are configured as dedicated data nodes with this command: curl -XGET 'http://localhost:9200/_cluster/state?pretty'. Any node that lists master: false and does not list data: false are dedicated data nodes.

When running an Elasticsearch cluster, it is important to set the minimum number of master-eligible nodes that need to be running for the cluster to function normally, which is sometimes referred to as quorum. This is to ensure data consistency in the event that one or more nodes lose connectivity to the rest of the cluster, preventing what is known as a “split-brain” situation.

To calculate the number of minimum master nodes your cluster should have, calculate n / 2 + 1, where n is the total number of “master-eligible” nodes in your healthy cluster, then round the result down to the nearest integer. For example, for a 3-node cluster, the quorum is 2.

The minimum master nodes setting can be set dynamically, through the Elasticsearch HTTP API. To do so, run this command on any node (replace the highlighted number with your quorum):

curl -XPUT localhost:9200/_cluster/settings?pretty -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 2
    }
}'

Output:
{
  "acknowledged" : true,
  "persistent" : {
    "discovery" : {
      "zen" : {
        "minimum_master_nodes" : "2"
      }
    }
  },
  "transient" : { }
}

If you want to check this setting later, you can run this command:

curl -XGET localhost:9200/_cluster/settings?pretty