How To Test Your Ansible Deployment with InSpec and Kitchen

The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

Introduction

InSpec is an open-source auditing and automated testing framework used to describe and test for regulatory concerns, recommendations, or requirements. It is designed to be human-readable and platform-agnostic. Developers can work with InSpec locally or using SSH, WinRM, or Docker to run testing, so it’s unnecessary to install any packages on the infrastructure that is being tested.

Although with InSpec you can run tests directly on your servers, there is a potential for human error that could cause issues in your infrastructure. To avoid this scenario, developers can use Kitchen to create a virtual machine and install an OS of their choice on the machines where tests are running. Kitchen is a test runner, or test automation tool, that allows you to test infrastructure code on one or more isolated platforms. It also supports many testing frameworks and is flexible with a driver plugin architecture for various platforms such as Vagrant, AWS, DigitalOcean, Docker, LXC containers, etc.

In this tutorial, you’ll write tests for your Ansible playbooks running on a DigitalOcean Ubuntu 18.04 Droplet. You’ll use Kitchen as the test-runner and InSpec for writing the tests. By the end of this tutorial, you’ll be able to test your Ansible playbook deployment.

Prerequisites

Before you begin with this guide, you’ll need a DigitalOcean account in addition to the following:

• A local installation of Ruby on your machine. You can install Ruby by following the tutorial for your distribution in the series: How To Install and Set Up a Local Programming Environment for Ruby.
• Chef Development Kit (ChefDK) installed on your machine.
• SSH keys set up on your machine by following Steps 1 and 2 of How To Set Up SSH Keys. To upload your SSH public key to your DigitalOcean account you can follow our How To Add SSH Keys to a DigitalOcean Account tutorial.
• A read and write DigitalOcean personal access token. Make sure you record the token in a safe place; you’ll use it later in this tutorial. This allows you to create a Droplet on DigitalOcean, which is where this tutorial will run the tests.

Step 1 — Setting Up and Initializing Kitchen

You’ve installed ChefDK as part of the prerequisites that comes packaged with kitchen. In this step, you’ll set up Kitchen to communicate with DigitalOcean.

Before initializing Kitchen, you’ll create and move into a project directory. In this tutorial, we’ll call it ansible_testing_dir.

Run the following command to create the directory:

mkdir ~/ansible_testing_dir

And then move into it:

cd ~/ansible_testing_dir

Using gem install the kitchen-digitalocean package on your local machine. This allows you to tell kitchen to use the DigitalOcean driver when running tests:

gem install kitchen-digitalocean

Within the project directory, you’ll run the kitchen init command specifying ansible_playbook as the provisioner and digitalocean as the driver when initializing Kitchen:

kitchen init --provisioner=ansible_playbook --driver=digitalocean

You’ll see the following output:

Output
create  kitchen.yml
create  chefignore
create  test/integration/default

This has created the following within your project directory:

• test/integration/default is the directory to which you’ll save your test files.

• chefignore is the file you would use to ensure certain files are not uploaded to the Chef Infra Server, but you won’t be using it in this tutorial.

• kitchen.yml is the file that describes your testing configuration: what you want to test and the target platforms.

Now, you need to export your DigitalOcean credentials as environment variables to have access to create Droplets from your CLI. First, start with your DigitalOcean access token by running the following command:

export DIGITALOCEAN_ACCESS_TOKEN="YOUR_DIGITALOCEAN_ACCESS_TOKEN"

You also need to get your SSH Key ID number; note that YOUR_DIGITALOCEAN_SSH_KEY_IDS must be the numeric ID of your SSH key, not the symbolic name. Using the DigitalOcean API, you can get the numeric ID of your keys with the following command:

curl -X GET https://api.digitalocean.com/v2/account/keys -H "Authorization: Bearer $DIGITALOCEAN_ACCESS_TOKEN"

From this command you’ll see a list of your SSH Keys and related metadata. Read through the output to find the correct key and identify the ID number within the output:

Output
...
 {"id":your-ID-number,"fingerprint":"fingerprint","public_key":"ssh-rsa your-ssh-key","name":"your-ssh-key-name"
...

curl -X GET https://api.digitalocean.com/v2/account/keys -H "Authorization: Bearer $DIGITALOCEAN_ACCESS_TOKEN" | jq

Output
{
  "ssh_keys": [
    {
      "id": YOUR_SSH_KEY_ID,
      "fingerprint": "2f:d0:16:6b",
      "public_key": "ssh-rsa AAAAB3NzaC1yc2 example@example.local",
      "name": "sannikay"
    }
  ],
}

Once you’ve identified your SSH numeric IDs, export them with the following command:

export DIGITALOCEAN_SSH_KEY_IDS="YOUR_DIGITALOCEAN_SSH_KEY_ID"

You’ve initialized kitchen and set up the environment variables for your DigitalOcean credentials. Now you’ll move on to create and run tests on your DigitalOcean Droplets directly from the command line.

Step 2 — Creating the Ansible Playbook

In this step, you’ll create a playbook and roles that set up Nginx and Node.js on the Droplet created by kitchen in the next step. Your tests will be run against the playbook to ensure the conditions specified in the playbook are met.

To begin, create a roles directory for both the Nginx and Node.js roles:

mkdir -p roles/{nginx,nodejs}/tasks

This will create a directory structure as follows:

roles
├── nginx
│   └── tasks
└── nodejs
    └── tasks

Now, create a main.yml file in the roles/nginx/tasks directory using your preferred editor:

nano roles/nginx/tasks/main.yml

In this file, create a task that sets up and starts Nginx by adding the following content:

---
- name: Update cache repositories and install Nginx
  apt:
    name: nginx
    update_cache: yes

- name: Change nginx directory permission
  file:
    path: /etc/nginx/nginx.conf
    mode: 0750

- name: start nginx
  service:
    name: nginx
    state: started

Once you’ve added the content, save and exit the file.

In roles/nginx/tasks/main.yml, you define a task that will update the cache repository of your Droplet, which is an equivalent of running the apt update command manually on a server. This task also changes the Nginx configuration file permissions and starts the Nginx service.

You are also going to create a main.yml file in roles/nodejs/tasks to define a task that sets up Node.js:

nano roles/nodejs/tasks/main.yml

Add the following tasks to this file:

---
- name: Update caches repository
  apt:
    update_cache: yes

- name: Add gpg key for NodeJS LTS
  apt_key:
    url: "https://deb.nodesource.com/gpgkey/nodesource.gpg.key"
    state: present

- name: Add the NodeJS LTS repo
  apt_repository:
    repo: "deb https://deb.nodesource.com/node_{{ NODEJS_VERSION }}.x {{ ansible_distribution_release }} main"
    state: present
    update_cache: yes

- name: Install Node.js
  apt:
    name: nodejs
    state: present

Save and exit the file when you’re finished.

In roles/nodejs/tasks/main.yml, you first define a task that will update the cache repository of your Droplet. Then with the next task you add the GPG key for Node.js that serves as a means of verifying the authenticity of the Node.js apt repository. The final two tasks add the Node.js apt repository and install Node.js.

Now you’ll define your Ansible configurations, such as variables, the order in which you want your roles to run, and super user privilege settings. To do this, you’ll create a file named playbook.yml, which serves as an entry point for Kitchen. When you run your tests, Kitchen starts from your playbook.yml file and looks for the roles to run, which are your roles/nginx/tasks/main.yml and roles/nodejs/tasks/main.yml files.

Run the following command to create playbook.yml:

nano playbook.yml

Add the following content to the file:

---
 - hosts: all
   become: true
   remote_user: ubuntu
   vars:
    NODEJS_VERSION: 8

Save and exit the file.

You’ve created the Ansible playbook roles that you’ll be running your tests against to ensure conditions specified in the playbook are met.

Step 3 — Writing Your InSpec Tests

In this step, you’ll write tests to check if Node.js is installed on your Droplet. Before writing your test, let’s look at the format of an example InSpec test. As with many test frameworks, InSpec code resembles a natural language. InSpec has two main components, the subject to examine and the subject’s expected state:

describe '<entity>' do
  it { <expectation> }
end

In block A, the keywords do and end define a block. The describe keyword is commonly known as test suites, which contain test cases. The it keyword is used for defining the test cases.

<entity> is the subject you want to examine, for example, a package name, service, file, or network port. The <expectation> specifies the desired result or expected state, for example, Nginx should be installed or should have a specific version. You can check the InSpec DSL documentation to learn more about the InSpec language.

Another example InSpec test block:

control 'Can be anything unique' do  
  impact 0.7                         
  title 'A human-readable title'     
  desc  'An optional description'
  describe '<entity>' do             
    it { <expectation> }
  end
end

The difference between block A and block B is the control block. The control block is used as a means of regulatory control, recommendation or requirement. The control block has a name; usually a unique ID, metadata such as desc, title, impact, and finally group together related describe block to implement the checks.

desc, title, and impact define metadata that fully describe the importance of the control, its purpose, with a succinct and complete description. impact defines a numeric value that ranges from 0.0 to 1.0 where 0.0 to <0.01 is classified as no impact, 0.01 to <0.4 is classified as low impact, 0.4 to <0.7 is classified as medium impact, 0.7 to <0.9 is classified as high impact, 0.9 to 1.0 is classified as critical control.

Now to implement a test. Using the syntax of block A, you’ll use InSpec’s package resource to test if Node.js is installed on the system. You’ll create a file named sample.rb in your test/integration/default directory for your tests.

Create sample.rb:

nano test/integration/default/sample.rb

Add the following to your file:

describe package('nodejs') do
  it { should be_installed }
end

Here your test is using the package resource to check Node.js is installed.

Save and exit the file when you’re finished.

To run this test, you need to edit kitchen.yml to specify the playbook you created earlier and to add to your configurations.

Open your kitchen.yml file:

nano ansible_testing_dir/kitchen.yml

Replace the content of kitchen.yml with the following:

---
driver:
  name: digitalocean

provisioner:
  name: ansible_playbook
  hosts: test-kitchen
  playbook: ./playbook.yml

verifier:
  name: inspec

platforms:
  - name: ubuntu-18
    driver_config:
      ssh_key: PATH_TO_YOUR_PRIVATE_SSH_KEY
      tags:
        - inspec-testing
      region: fra1
      size: 1gb
      private_networking: false
    verifier:
      inspec_tests:
        - test/integration/default
suites:
  - name: default

The platform options include the following:

• name: The image you’re using.

• driver_config: Your DigitalOcean Droplet configuration. You’re specifying the following options for the driver_config:

  • ssh_key: Path to YOUR_PRIVATE_SSH_KEY. Your YOUR_PRIVATE_SSH_KEY is located in the directory you specified when creating your ssh key.
  • tags: The tags associated with your Droplet.
  • region: The region where you want your Droplet to be hosted.
  • size: The memory you want your Droplet to have.

• verifier: This defines that the project contains InSpec tests.

  • The inspec_tests part specifies that the tests exist under the project test/integration/default directory.

Note that the name and region use abbreviations. You can check on the test-kitchen documentation for the abbreviations you can use.

Once you’ve added your configuration, save and exit the file.

Run the kitchen test command to run the test. This will check to see if Node.js is installed—this will purposefully fail because you don’t currently have the Node.js role in your playbook.yml file:

kitchen test

You’ll see output similar to the following:

Output: failing test results
-----> Starting Kitchen (v1.24.0)
-----> Cleaning up any prior instances of <default-ubuntu-18>
-----> Destroying <default-ubuntu-18>...
       DigitalOcean instance <145268853> destroyed.
       Finished destroying <default-ubuntu-18> (0m2.63s).
-----> Testing <default-ubuntu-18>
-----> Creating <default-ubuntu-18>...
       DigitalOcean instance <145273424> created.
       Waiting for SSH service on 138.68.97.146:22, retrying in 3 seconds
       [SSH] Established
       (ssh ready)

       Finished creating <default-ubuntu-18> (0m51.74s).
-----> Converging <default-ubuntu-18>...
$$$$$$ Running legacy converge for 'Digitalocean' Driver
-----> Installing Chef Omnibus to install busser to run tests
       PLAY [all] *********************************************************************

       TASK [Gathering Facts] *********************************************************
       ok: [localhost]

       PLAY RECAP *********************************************************************
       localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

       Downloading files from <default-ubuntu-18>
       Finished converging <default-ubuntu-18> (0m55.05s).
-----> Setting up <default-ubuntu-18>...
$$$$$$ Running legacy setup for 'Digitalocean' Driver
       Finished setting up <default-ubuntu-18> (0m0.00s).
-----> Verifying <default-ubuntu-18>...
       Loaded tests from {:path=>". ansible_testing_dir.test.integration.default"}

Profile: tests from {:path=>"ansible_testing_dir/test/integration/default"} (tests from {:path=>"ansible_testing_dir.test.integration.default"})
Version: (not specified)
Target:  ssh://root@138.68.97.146:22

  System Package nodejs
     ×  should be installed 
     expected that System Package nodejs is installed

Test Summary: 0 successful, 1 failure, 0 skipped
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Verify failed on instance <default-ubuntu-18>.  Please see .kitchen/logs/default-ubuntu-18.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

  4.54s user 1.77s system 5% cpu 2:02.33 total

The output notes that your test is failing because you don’t have Node.js installed on the Droplet you provisioned with kitchen. You’ll fix your test by adding the nodejs role to your playbook.yml file and run the test again.

Edit the playbook.yml file to include the nodejs role:

nano playbook.yml

Add the following highlighted lines to your file:

---
 - hosts: all
   become: true
   remote_user: ubuntu
   vars:
    NODEJS_VERSION: 8

   roles:
    - nodejs

Save and close the file.

Now, you’ll rerun the test using the kitchen test command:

kitchen test

You’ll see the following output:

Output
......
Target:  ssh://root@46.101.248.71:22

  System Package nodejs
     ✔  should be installed

Test Summary: 1 successful, 0 failures, 0 skipped
       Finished verifying <default-ubuntu-18> (0m4.89s).
-----> Destroying <default-ubuntu-18>...
       DigitalOcean instance <145512952> destroyed.
       Finished destroying <default-ubuntu-18> (0m2.23s).
       Finished testing <default-ubuntu-18> (2m49.78s).
-----> Kitchen is finished. (2m55.14s)
  4.86s user 1.77s system 3% cpu 2:56.58 total

Your test now passes because you have Node.js installed using the nodejs role.

Here is a summary of what Kitchen is doing in the Test Action:

• Destroys the Droplet if it exists
• Creates the Droplet
• Converges the Droplet
• Verifies the Droplet with InSpec
• Destroys the Droplet

Kitchen will abort the run on your Droplet if it encounters any issues. This means if your Ansible playbook fails, InSpec won’t run and your Droplet won’t be destroyed. This gives you a chance to inspect the state of the instance and fix any issues. The behavior of the final destroy action can be overridden if desired. Check out the CLI help for the --destroy flag by running the kitchen help test command.

You’ve written your first tests and run them against your playbook with one instance failing before fixing the issue. Next you’ll extend your test file.

Step 4 — Adding Test Cases

In this step, you’ll add more test cases to your test file to check if Nginx modules are installed on your Droplet and the configuration file has the right permissions.

Edit your sample.rb file to add more test cases:

nano test/integration/default/sample.rb

Add the following test cases to the end of the file:

. . .
control 'nginx-modules' do
  impact 1.0
  title 'NGINX modules'
  desc 'The required NGINX modules should be installed.'
  describe nginx do
    its('modules') { should include 'http_ssl' }
    its('modules') { should include 'stream_ssl' }
    its('modules') { should include 'mail_ssl' }
  end
end

control 'nginx-conf' do
  impact 1.0
  title 'NGINX configuration'
  desc 'The NGINX config file should owned by root, be writable only by owner, and not writeable or and readable by others.'
  describe file('/etc/nginx/nginx.conf') do
    it { should be_owned_by 'root' }
    it { should be_grouped_into 'root' }
    it { should_not be_readable.by('others') }
    it { should_not be_writable.by('others') }
    it { should_not be_executable.by('others') }
  end
end

These test cases check that the nginx-modules on your Droplet include http_ssl, stream_ssl, and mail_ssl. You are also checking for /etc/nginx/nginx.conf file permissions.

You are using both the it and its keywords to define your test. The keyword its is only used to access properties of the resources. For example, modules is a property of nginx.

Save and exit the file once you’ve added the test cases.

Now run the kitchen test command to test again:

kitchen test

You’ll see the following output:

Output
...
Target:  ssh://root@104.248.131.111:22

  ↺  nginx-modules: NGINX modules
     ↺  The `nginx` binary not found in the path provided.
  ×  nginx-conf: NGINX configuration (2 failed)
     ×  File /etc/nginx/nginx.conf should be owned by "root"
     expected `File /etc/nginx/nginx.conf.owned_by?("root")` to return true, got false
     ×  File /etc/nginx/nginx.conf should be grouped into "root"
     expected `File /etc/nginx/nginx.conf.grouped_into?("root")` to return true, got false
     ✔  File /etc/nginx/nginx.conf should not be readable by others
     ✔  File /etc/nginx/nginx.conf should not be writable by others
     ✔  File /etc/nginx/nginx.conf should not be executable by others

  System Package nodejs
     ✔  should be installed
Profile Summary: 0 successful controls, 1 control failure, 1 control skipped
Test Summary: 4 successful, 2 failures, 1 skipped

You’ll see that some of the tests are failing. You’re going to fix those by adding the nginx role to your playbook file and rerunning the test. In the failing test, you’re checking for nginx modules and file permissions that are currently not present on your server.

Open your playbook.yml file:

nano ansible_testing_dir/playbook.yml

Add the following highlighted line to your roles:

---
- hosts: all
  become: true
  remote_user: ubuntu
  vars:
  NODEJS_VERSION: 8

  roles:
  - nodejs
  - nginx

Save and close the file when you’re finished.

Then run your tests again:

kitchen test

You’ll see the following output:

Output
...
Target:  ssh://root@104.248.131.111:22
✔  nginx-modules: NGINX version
     ✔  Nginx Environment modules should include "http_ssl"
     ✔  Nginx Environment modules should include "stream_ssl"
     ✔  Nginx Environment modules should include "mail_ssl"
  ✔  nginx-conf: NGINX configuration
     ✔  File /etc/nginx/nginx.conf should be owned by "root"
     ✔  File /etc/nginx/nginx.conf should be grouped into "root"
     ✔  File /etc/nginx/nginx.conf should not be readable by others
     ✔  File /etc/nginx/nginx.conf should not be writable by others
     ✔  File /etc/nginx/nginx.conf should not be executable by others

  System Package nodejs
     ✔  should be installed

Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped
Test Summary: 9 successful, 0 failures, 0 skipped

After adding the nginx role to the playbook all your tests now pass. The output shows that the http_ssl, stream_ssl, and mail_ssl modules are installed on your Droplet and the right permissions are set for the configuration file.

Once you’re finished, or you no longer need your Droplet, you can destroy it by running the kitchen destroy command to delete it after running your tests:

kitchen destroy

Following this command you’ll see output similar to:

Output
-----> Starting Kitchen (v1.24.0)
-----> Destroying <default-ubuntu-18>...
       Finished destroying <default-ubuntu-18> (0m0.00s).
-----> Kitchen is finished. (0m5.07s)
  3.79s user 1.50s system 82% cpu 6.432 total

You’ve written tests for your playbook, run the tests, and fixed the failing tests to ensure all the tests are passing. You’re now set up to create a virtual environment, write tests for your Ansible Playbook, and run your test on the virtual environment using Kitchen.

Conclusion

You now have a flexible foundation for testing your Ansible deployment, which allows you to test your playbooks before running on a live server. You can also package your test into a profile. You can use profiles to share your test through Github or the Chef Supermarket and easily run it on a live server.

For more comprehensive details on InSpec and Kitchen, refer to the official InSpec documentation and the official Kitchen documentation.