How To Set Up a CoreOS Cluster on DigitalOcean

Introduction

If you are planning on using CoreOS in your infrastructure, the first thing you will want to set up is a CoreOS cluster. In order for CoreOS machines to form a cluster, their etcd2 instances must be connected. In this tutorial, we will give step-by-step instructions to quickly create a 3-node CoreOS cluster on DigitalOcean.

Prerequisites

If you are unfamiliar with the components that CoreOS is built on (docker, etcd2, and fleet) it is highly recommended that you read An Introduction to CoreOS System Components. You will want to pay particular attention to the section that covers etcd2, since that component is essential to the cluster discovery process.

SSH Keys

Every CoreOS server that you create will need to have at least one SSH public key installed during its creation process. The key(s) will be installed to the core user’s authorized keys file, and you will need the corresponding private key(s) to log in to your CoreOS server.

If you do not already have any SSH keys associated with your DigitalOcean account, do so now by following steps 1-3 of this tutorial: How To Use SSH Keys with DigitalOcean Droplets. Then you will want to add your private key to your SSH agent on your client machine by running the following command:

ssh-add

For more about this step, see this article.

DigitalOcean Personal Access Token

If you are planning on using the DigitalOcean API to create your CoreOS machines, refer to this tutorial for information on how to generate and use a Personal Access Token with write permissions.

Now that you have the prerequisites out of the way, let’s start building our CoreOS cluster!

Generate a New Discovery URL

The first step to setting up a new CoreOS cluster is generating a new discovery URL, a unique address that stores peer CoreOS addresses and metadata. The easiest way to do this is to use https://discovery.etcd.io, a free discovery service. A new discovery URL can be generated by visiting https://discovery.etcd.io/new in a web browser or by running the following curl command:

curl -w "\n" "https://discovery.etcd.io/new?size=3"

Either method will return a fresh, unique discovery URL that looks something like the following (the highlighted part will be a unique token):

https://discovery.etcd.io/5c1574906b3502aa9d8dc43c1b185775

You will use your resulting discovery URL to create your new CoreOS cluster. The same discovery URL must be specified in the etcd2 section of the cloud-config of each server that you want to add to a particular CoreOS cluster.

Now that we have a discovery URL, let’s look at how to create cloud-config file that uses it.

Write a Cloud-Config File

CoreOS uses a file called cloud-config which allows you to declaratively customize network configuration, systemd units, and other OS-level items. This file is written in YAML format, which uses indentation to denote data hierarchy. The cloud-config file is processed when a machine is booted, and provides a way to configure your machines with etcd2 settings that will allow them to discover the cluster that they should join.

We will cover how to write a minimal cloud-config to get a working CoreOS cluster up and running. For a full list of items that can be configured with cloud-config, check out the official documentation. They also provide a helpful tool that can check your cloud-config file’s syntax, Cloud-Config Validator.

Minimal Cloud-Config

As mentioned earlier, the peer addresses of each CoreOS machine in a cluster is stored with the discovery URL. Therefore, each machine in a cluster must use the same discovery URL and pass in its own IP address where its etcd2 service can be reached. These are specified in cloud-config under the etcd2 section, and are shown in the code block below.

You will also need to specify a units section, which will start the etcd2 and fleet services that are necessary for a working CoreOS cluster.

Here is a basic cloud-config file that can be used with your CoreOS machines to make a new cluster (substitute the value of discovery with the discovery URL that you generated earlier):

#cloud-config

coreos:
  etcd2:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new:
    discovery: https://discovery.etcd.io/<$><discovery_token><$>
    # multi-region deployments, multi-cloud deployments, and Droplets without
    # private networking need to use $public_ipv4:
    advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
    initial-advertise-peer-urls: http://$private_ipv4:2380
    # listen on the official ports 2379, 2380 and one legacy port 4001:
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://$private_ipv4:2380
  fleet:
    public-ip: $private_ipv4   # used for fleetctl ssh command
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start

This cloud-config script can be used to set up a basic CoreOS that can be used for testing purposes; unfortunately, it is not very secure. For a more serious setup, you should set up a secure CoreOS cluster by following this tutorial: How To Secure Your CoreOS Cluster with TLS/SSL and Firewall Rules.

Create CoreOS Cluster

Now that you know what your cloud-config file for each machine in your new CoreOS cluster will consist of, let’s create your CoreOS cluster. Because Droplets can be created through the DigitalOcean Control Panel or API, we will show you how to create your CoreOS cluster using both methods.

DigitalOcean Control Panel

First, visit the DigitalOcean Control Panel then click the Create Droplet button.

Next, select CoreOS as your Linux distribution, then select which channel you want to use (Stable, Beta, or Alpha).

Then select your desired Droplet size. A smaller size is fine if you’re doing basic testing.

Next, select your preferred datacenter region.

Under the Select additional options header, select Private Networking and User Data. Copy and paste your cloud-config script into the User Data text field. It should look something like this:

Next, select at least one SSH key that you want to use to log in to your Droplets.

Under the Finalize and create section, create at least three Droplets and specify their hostnames. In our example, we’ll call them coreos-01, coreos-02, and coreos-03:

Lastly, click the Create button to create the Droplets that will form your CoreOS cluster.

To learn more about the Droplet creation process, using the DigitalOcean Control Panel, refer to this guide.

DigitalOcean API

If you use the DigitalOcean API to create your CoreOS Droplets, you can specify your cloud-config via the user_data parameter in your Droplet creation POST request–just paste the whole script in there.

Let us assume that we want to create three 1 GB Droplets named coreos-01, coreos-02, and coreos-03 with private networking, in the NYC3 data center, using the CoreOS Stable channel image, and the cloud-config file shown earlier. Here is an example of the curl command you would run to create it using the DigitalOcean API:

curl -X POST "https://api.digitalocean.com/v2/droplets" \
      -d'{"names":["coreos-01","coreos-02","coreos-03"],"region":"nyc3","size":"1GB","private_networking":true,"image":"coreos-stable","user_data":
"#cloud-config

coreos:
  etcd2:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new:
    discovery: https://discovery.etcd.io/<$><discovery_token><$>
    # multi-region deployments, multi-cloud deployments, and Droplets without
    # private networking need to use $public_ipv4:
    advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
    initial-advertise-peer-urls: http://$private_ipv4:2380
    # listen on the official ports 2379, 2380 and one legacy port 4001:
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://$private_ipv4:2380
  fleet:
    public-ip: $private_ipv4   # used for fleetctl ssh command
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start",
      "ssh_keys":[ <SSH Key ID(s)> ]}' \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json"

You must substitute your SSH Key ID(s) or fingerprint(s) for <SSH Key ID(s)>, and make sure $TOKEN is set to one of your read/write DigitalOcean Personal Access Tokens.

After running this command with the appropriate substitutions, your 3-node CoreOS cluster will be created.

For more information about using the API, please refer to this tutorial.

Verify Cluster

To verify that our 3-machine cluster has formed properly, we must SSH to one of the cluster members.

Log into the coreos-01 machine as the core user via SSH, and use the -A option to forward your SSH agent. Remember to substitute the public IP address:

ssh -A core@coreos-01_public_IP

At the command prompt, enter this fleetctl command to show all the members of the cluster:

fleetctl list-machines

You should see a list of all of the online machines in the cluster, identifiable by their respective peer-addr IP addresses. Here is an example of the output:

MACHINE		IP		METADATA
59b2fffd...	10.131.29.141	-
853b0df3...	10.131.63.121	-
cd64a2e3...	10.131.63.120	-

If you see all of the machines that you created, all of them are aware of each other via etcd2, and your cluster has formed properly!

Adding New Machines

If you would like to add new machines to an existing CoreOS cluster, simply create a new Droplet using the same cloud-config (and discovery URL). Your new CoreOS machine will automatically join the existing cluster.

If you forgot which discovery URL you used, you may look it up on one of the members of the cluster. Use the following grep command on one of your existing machines:

grep DISCOVERY /run/systemd/system/etcd2.service.d/20-cloudinit.conf

You will see a line the contains the original discovery URL, like the following:

Environment="ETCD_DISCOVERY=https://discovery.etcd.io/575302f03f4fb2db82e81ea2abca55e9"

Conclusion

Your basic CoreOS cluster is set up, and now you can move on to testing with it! If you are looking to set up a secure CoreOS cluster, follow this tutorial: How To Secure Your CoreOS Cluster with TLS/SSL and Firewall Rules.

The rest of the tutorials in this series will show you more about CoreOS, and how to use docker containers and service discovery with your CoreOS cluster.

listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001

#cloud-config

coreos:
  etcd2:
    discovery: "Discovery key here"
    advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
    initial-advertise-peer-urls: http://$private_ipv4:2380
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://$private_ipv4:2380
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start

coreos:
  units:
    - name: iptables-restore.service
      enable: true
write_files:
  - path: /var/lib/iptables/rules-save
    permissions: 0644
    owner: root:root
    content: |
      *filter
      :INPUT DROP [0:0]
      :FORWARD DROP [0:0]
      :OUTPUT ACCEPT [0:0]
      -A INPUT -i lo -j ACCEPT
      -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
      -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
      -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
      -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
      COMMIT