How To Manage Your Cluster with Chef and Knife on Ubuntu

Introduction

Chef is a configuration management system – it’s designed to give you a repeatable set of recipes for building your infrastructure, allowing you to automate, version, and test your infrastructure in much the same way as your application code.

This is the sixth tutorial in the Getting Started Managing Your Infrastructure Using Chef series. In this guide, we’ll assume that you’ve completed the other five tutorials, and so you’ve got a Chef server, workstation, and one or more nodes up and running.

Our Goal

knife is a command line tool packaged with Chef. You’ve likely already used knife to create and manage Chef cookbooks, data bags, or roles. When you issue a command with knife, you usually type something along the lines of:

knife cookbook create

The example command above uses the cookbook knife subcommand. This guide will introduce you to some new knife subcommands for issuing commands and getting information about your Chef cluster.

We’ll cover:

• knife status
• knife ssh
• knife node

Prerequisites

This tutorial assumes you’ve followed along until the fifth guide, How To Use the DigitalOcean Plugin for Knife to Manage Droplets in Chef, in the Getting Started Managing Your Infrastructure Using Chef series.

Create Example Roles and Servers

If you don’t have an established Chef cluster, or you’d like to follow this guide closely and see the same output, we can set up some example roles and servers.

First, on your workstation, change to your chef-repo directory:

cd ~/chef-repo

We’re going to add a backend role to our existing web_server role. For now it’ll be blank, but later on you could turn this into a database or application server.

nano roles/backend.rb

Add this content to the backend.rb file:

name "backend"
description "Backend for application servers"

Then upload the new role to your Chef server.

knife role from file roles/backend.rb

Once you’ve done that, we can create some sample nodes using the DigitalOcean Knife plugin.

(Note: This plugin became unmaintained as of October 2014. It’s up to you whether you want to start using it.)

knife digital_ocean droplet create --server-name frontend01 --image 6918990 --location 4 --size 63 --ssh-keys 22222 --bootstrap --run-list "role[web_server]"

knife digital_ocean droplet create --server-name frontend02 --image 6918990 --location 4 --size 63 --ssh-keys 22222 --bootstrap --run-list "role[web_server]"

knife digital_ocean droplet create --server-name backend01 --image 6918990 --location 4 --size 63 --ssh-keys 22222 --bootstrap --run-list "role[backend]"

knife digital_ocean droplet create --server-name backend02 --image 6918990 --location 4 --size 63 --ssh-keys 22222 --bootstrap --run-list "role[backend]"

Note: If your domain names for the various hosts in your Chef cluster don’t resolve externally, and you’re making them connect to each other with edited /etc/hosts files, the provisioning will not work, as the new server instance will be given the default /etc/hosts file.

You may still have some nodes active that you’ve created during the Chef tutorial series. That means some of the commands we’ll run during this tutorial will have extra lines or information. If you’d like to follow along extactly, you can always remove these nodes by clicking Nodes > Delete from your Chef server’s web interface.

Showing Status with knife status

The status subcommand is designed to show status information about your nodes. To use knife status, just change to your chef-repo directory and type:

knife status

You’ll see a list of the nodes your Chef server knows about, including the times of their last chef-client runs, node names, fully-qualified domain names, IP addresses, and platforms.

3 minutes ago, frontend01, fe1.yourdomain.com, 111.111.111.111, ubuntu 14.04.
3 minutes ago, frontend02, fe2.yourdomain.com, 222.222.222.222, ubuntu 14.04.
20 hours ago,  backend01, be1.yourdomain.com, 333.333.333.333, ubuntu 14.04.
3 minutes ago, backend02, be2.youdomain.com, 333.333.333.333, ubuntu 14.04.

We can immediately see from this that we need to take a closer look at backend01 - it hasn’t succesfully run chef-client (“checked in” in Chef parlance) in about 20 hours.

If you’ve got a node in a similar situation, check your chef-client error logs, or use Reports > Run History from Chef server’s web interface.

Issuing Commands with knife ssh

We can use knife ssh to issue commands to our nodes (or some subset of nodes) at once. For example, we can use knife ssh to restart Nginx on all our nodes with the role frontend.

You’ll need a user authorized to SSH into the server (i.e., you can run ssh yourusername@fe1.yourdomain.com and get a shell). If you don’t have SSH keys set up, you can use the -P option to prompt for a password.

knife ssh "role:web_server" "service nginx restart" -x yourusername -a ipaddress

You should get output that looks like:

111.111.111.111  * Restarting nginx nginx      [ OK ] 
222.222.222.222  * Restarting nginx nginx      [ OK ] 

We just ran a command on all our frontend servers by issuing a single knife command. Let’s pull apart the arguments and learn a bit more about how knife ssh works.

knife ssh "web_server" "service nginx restart" -x yourusername -a ipaddress

The first argument to knife ssh is a Chef search query - usually you’ll want something like role:YOUR_ROLE_NAME, but you can also search via many other attributes (and combine them with Boolean operators). For instance, to run a command on only nodes with ubuntu in their platform attribute, you could run:

knife ssh "platform:ubuntu*" "service nginx restart" -x yourusername -a ipaddress

The * in the above is a wildcard character. It will match zero or more characters in the attribute value. In this case, it will allow us to match the value ubuntu 14.04 as displayed in the knife status output we looked at earlier.

knife ssh "role:web_server" "service nginx restart" -x yourusername -a ipaddress

The second argument to knife ssh is the command that you want to run on the servers (that match the search query). It can be almost anything you would type into a shell. You can even join commands with a semicolon (;).

knife ssh "role:web_server" "uptime;date" -x yourusername -a ipaddress

Which would output something like:

111.111.111.111  12:53:36 up 2 days, 15:25,  1 user,  load average: 0.08, 0.03, 0.05
111.111.111.111 Wed Oct 22 12:53:36 UTC 2014
222.222.222.222  12:53:30 up 2 days, 15:21,  1 user,  load average: 0.00, 0.01, 0.05
222.222.222.222 Wed Oct 22 12:53:30 UTC 2014

The -x argument we’ve already covered - it’s the SSH username to use for logging in.

The -a argument specifies which node attribute to use to as the address for SSH. By default, it’s the FQDN of your node (remember, you can find this with the knife status command shown earlier), so if you can resolve your servers by visiting http://fe1.yourdomain.com, then you can omit the -a option.

Interactive knife ssh

knife ssh also has the ability to put you into an interactive shell, where you can issue a series of commands and see the results very quickly. You can start an interactive knife ssh shell by using interactive in place of your SSH command.

knife ssh "role:web_server" interactive -x yourusername -a ipaddress

This will show us:

Connected to 111.111.111.111 and 222.222.222.222

To run a command on a list of servers, do:
  on SERVER1 SERVER2 SERVER3; COMMAND
  Example: on latte foamy; echo foobar

To exit interactive mode, use 'quit!'

knife-ssh>

You can issue commands to all servers in your search result by simply typing the command and pressing Enter.

knife-ssh> uptime
111.111.111.111  18:43:55 up 2 days, 21:16,  1 user,  load average: 0.01, 0.03, 0.05
222.222.222.222  18:43:49 up 2 days, 21:11,  1 user,  load average: 0.00, 0.01, 0.05

If you’d like to refine your server list further, just use on, as the help message suggests. Note that you should replaceSERVER1 in the example command with the attribute you’ve used as -a; for us, this is the node’s IP address.

knife-ssh> on 111.111.111.111; echo hello digitalocean
111.111.111.111 hello digitalocean

One really cool use of interactive knife ssh is using it to tail server logs. For instance, if your access log for Nginx is in the default location (/var/log/nginx/access.log), you can tail the logs with the -f (follow) option, and the results will continually print to your console.

knife-ssh> tail -f /var/log/nginx/access.log

(Depending on how Nginx is set up, you may have to use sudo in front of this command.)

If you visit the IP address for one of your nodes in a web browser, you’ll see an entry appear in the Nginx access log, without having to do anything!

Managing Nodes with knife node

If you’ve followed the How To Create Simple Chef Cookbooks to Manage Infrastructure on Ubuntu guide, then you’ve already used knife node to list all nodes on your Chef server, and to edit node attributes.

knife node list
knife node edit frontend01

You can also use knife node to delete nodes.

knife node delete frontend01

Or, show more detailed node attributes:

knife node show frontend01
*******
Node Name:   frontend01
Environment: _default
FQDN:        fe01.yourdomain.com
IP:          111.111.111.111
Run List:    role[frontend]
Roles:       web_server
Recipes:     apt, nginx, apt::default, nginx::default
Platform:    ubuntu 14.04
Tags:     

You can get a full listing of your node’s attributes using the -l option:

knife node show -l frontend01

This will return a very long list of attributes, most of them populated automatically by Ohai, which is a Chef tool that passes attributes automatically to chef-client each time it’s run.

A long printed list of attributes is not neccessarily very useful to us. However, knife has us covered there too. We can specify the --format option in order to get a JSON or YAML representation of a Node’s attributes.

knife node show frontend01 --format json
*******

{
  "name": "frontend01",
  "chef_environment": "_default",
  "run_list": [
  "role[web_server]"
]
,
  "normal": {
    "tags": [

    ]
  }
}

You can also retrieve a single node attribute with -a.

knife node show frontend01 --format json -a ipaddress
*******
{
  "frontend01": {
    "ipaddress": "111.111.111.111"
  }
}

JSON or YAML output might be very useful if we wanted to build more complex scripts that involved knife, or even for displaying dashboards and metrics.

Conclusion

knife is a powerful tool, not just for creating and updating the various objects in your Chef cluster, but also for viewing and managing the state of your cluster.

With knife ssh you can write one command and have it run on many nodes simultaneously – a very powerful tool for any devops engineer.