How To Acquire a Let's Encrypt Certificate Using Ansible on Ubuntu 18.04

The author selected the Electronic Frontier Foundation to receive a donation as part of the Write for DOnations program.

Introduction

Modern infrastructure management is best done using automated processes and tools. Acquiring a Let’s Encrypt certificate using the standard Certbot client is quick and easy, but is generally a task that has to be done manually when commissioning servers. This is manageable for an individual server setup, but can become tedious when deploying a larger fleet.

Using a configuration management tool such as Ansible to acquire a certificate makes this task completely automatic and reproducible. If you ever have to rebuild or update your server, you can just run your Ansible playbook, rather than having to manually carry out the steps again.

In this tutorial, you’ll write an Ansible playbook to acquire a Let’s Encrypt certificate automatically for an Ansible host machine.

Prerequisites

To complete this tutorial, you will need:

• Two Ubuntu 18.04 servers set up by following the Initial Server Setup with Ubuntu 18.04, including a sudo non-root user.

The first server will be used as your Ansible server, which we will call Ansible server throughout this tutorial. This is where Ansible will run to send the commands to the host machine. Alternatively, you can use your local machine or any other machine that has your Ansible inventory configured as your Ansible server.

On your Ansible server, you’ll need:

• A correctly configured Ansible installation that is able to connect to your Ansible hosts by following How To Install and Configure Ansible on Ubuntu 18.04.

The second server will be used as your Ansible host, which we will call the host machine throughout this tutorial. This is the machine that you wish to configure and issue certificates on. This machine will also run a web server to serve the certificate issuance validation files.

On your host machine, you’ll need:

• A domain name that you are eligible to acquire a TLS certificate for, with the required DNS records configured to point to your Ansible host machine. In this particular example, the playbook will acquire a certificate valid for your-domain and www.your-domain, however it can be adjusted for other domains or subdomains if required.

• A web server that is accessible from the internet over port 80 (HTTP), for example by following steps 1, 2, and 3 of How To Install the Apache Web Server on Ubuntu 18.04. This could also be an Nginx server, or any other suitable web server software.

Once you have these ready, log in to your Ansible server as your non-root user to begin.

Step 1 — Configuring the Settings for the Let’s Encrypt Ansible Module

Ansible has a built-in module named letsencrypt, which allows you to acquire valid TLS certificates using the ACME (Automated Certificate Management Environment) protocol.

In this first step, you will add a host variables configuration file to define the configuration variables that are required to use the module.

Firstly, create the host_vars Ansible directory on your Ansible server:

sudo mkdir /etc/ansible/host_vars

Next, create a new file in the /etc/ansible/host_vars directory with the name of your Ansible host machine. In this example, you’ll use host1 as the name of the host:

sudo nano /etc/ansible/host_vars/host1

The following sample configuration includes everything you need to get started, including: the validation method and server address, an email address to receive certificate expiry reminders to, and the directories where your Let’s Encrypt keys and certificates will be saved.

Copy the sample configuration into the file:

---
acme_challenge_type: http-01
acme_directory: https://acme-v02.api.letsencrypt.org/directory
acme_version: 2
acme_email: certificate-reminders@your-domain
letsencrypt_dir: /etc/letsencrypt
letsencrypt_keys_dir: /etc/letsencrypt/keys
letsencrypt_csrs_dir: /etc/letsencrypt/csrs
letsencrypt_certs_dir: /etc/letsencrypt/certs
letsencrypt_account_key: /etc/letsencrypt/account/account.key
domain_name: your-domain

Save and close the file when you’ve finished.

Adjust the domain name and email address as required. You can use any email address—it doesn’t have to be the one on your-domain.

Some of the directory/file paths defined may not actually exist on your server yet. This is OK; the first part of the playbook will be to create these directories and assign the relevant permissions.

You’ve added the required configuration variables to your Ansible inventory file. Next, you will begin writing the playbook to acquire a certificate.

Step 2 — Creating the Let’s Encrypt Directories and Account Key

In this step, you’ll write the Ansible tasks that you’ll use to create the required Let’s Encrypt directories, assign the correct permissions, and generate a Let’s Encrypt account key.

Firstly, create a new playbook named letsencrypt-issue.yml on your Ansible server in a new directory of your choice, for example /home/user/ansible-playbooks:

cd ~
mkdir ansible-playbooks
cd ansible-playbooks
nano letsencrypt-issue.yml

Before you can start writing Ansible tasks, you’ll need to specify the hosts and associated settings. Adjust the following according to how you referred to your hosts in the prerequisite tutorial. Then add the following to the top of the file:

---
- hosts: "host1"
  tasks:

Now you can begin writing the required tasks, the first of which is to create the file system directories required to store the Let’s Encrypt files. Add the following Ansible task to the file after the previous content:

...
  - name: "Create required directories in /etc/letsencrypt"
    file:
      path: "/etc/letsencrypt/{{ item }}"
      state: directory
      owner: root
      group: root
      mode: u=rwx,g=x,o=x
    with_items:
    - account
    - certs
    - csrs
    - keys

This Ansible task will create the account, certs, csrs, and keys directories in /etc/letsencrypt, which is where the files required for acquiring certificates will be stored.

You set the owner of the directories to root and apply the permissions u=rwx,g=x,o=x so that only root has read and write access to them. This is recommended as the directories will contain private keys, certificate signing requests (CSRs), and signed certificates, which should be kept confidential.

Next, the Let’s Encrypt account key needs to be created. You’ll use this to identify yourself to the Let’s Encrypt service.

Add the following task to your playbook:

...
  - name: "Generate a Let's Encrypt account key"
    shell: "if [ ! -f {{ letsencrypt_account_key }} ]; then openssl genrsa 4096 | sudo tee {{ letsencrypt_account_key }}; fi"

The account key doesn’t need to be re-created every time you renew the certificates, so you also add a check for an existing key if [ ! -f {{ letsencrypt_account_key }} ];, to make sure that it isn’t overwritten.

You’ll continue to work in letsencrypt-issue.yml in the next step, so don’t close this file yet.

You’ve created your playbook and set up the initial configuration and tasks in order to prepare for acquiring your Let’s Encrypt certificate. Next, you will add further tasks for the private key and CSR generation.

Step 3 — Generating Your Private Key and Certificate Signing Request

In this step, you’ll write the playbook tasks to generate the required private key and certificate signing request.

The first task in this section will generate the required private key for your certificate. Add the following to the end of your playbook that you started writing in Step 2:

...
  - name: "Generate Let's Encrypt private key"
    shell: "openssl genrsa 4096 | sudo tee /etc/letsencrypt/keys/{{ domain_name }}.key"

Subdomains on the same domain will all be added to the same certificate through the use of Subject Alternate Names (SANs), so you only need to generate one private key for now.

You’ll use the next task to generate a Certificate Signing Request (CSR) for the certificate that you want to acquire. This is submitted to Let’s Encrypt in order for them to validate and issue each certificate.

Add the following to the end of the playbook:

...
  - name: "Generate Let's Encrypt CSR"
    shell: "openssl req -new -sha256 -key /etc/letsencrypt/keys/{{ domain_name }}.key -subj \"/CN={{ domain_name }}\" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf \"\n[SAN]\nsubjectAltName=DNS:{{ domain_name }},DNS:www.{{ domain_name }}\")) | sudo tee /etc/letsencrypt/csrs/{{ domain_name }}.csr"
    args:
      executable: /bin/bash

This task generates a CSR for your domain, with the www subdomain added to the certificate as a SAN.

You’ll continue to work in letsencrypt-issue.yml in the next step, so don’t close this file yet.

You’ve written the Ansible tasks to generate the private key and CSR for your certificate. Next, you’ll work on the tasks that will begin the validation and issuance process.

Step 4 — Starting the ACME Validation Process

In this step, you’ll write a task to submit the Certificate Signing Request to Let’s Encrypt using the outputted files from the task documented in Step 3. This will return some challenge files, which you’ll need to serve on your web server in order to prove ownership of the domain name and subdomain for which you’re requesting a certificate.

The following task will submit the CSR for your-domain. Add it to the end of your playbook:

...
  - name: "Begin Let's Encrypt challenges"
    letsencrypt:
      acme_directory: "{{ acme_directory }}"
      acme_version: "{{ acme_version }}"
      account_key_src: "{{ letsencrypt_account_key }}"
      account_email: "{{ acme_email }}"
      terms_agreed: 1
      challenge: "{{ acme_challenge_type }}"
      csr: "{{ letsencrypt_csrs_dir }}/{{ domain_name }}.csr"
      dest: "{{ letsencrypt_certs_dir }}/{{ domain_name }}.crt"
      fullchain_dest: "{{ letsencrypt_certs_dir }}/fullchain_{{ domain_name }}.crt"
      remaining_days: 91
    register: acme_challenge_your_domain

This task makes wide usage of the variables that you configured in Step 1. It registers a variable containing the ACME challenge files that you’ll use in the next step. You’ll need to manually adjust the name of the variable to contain your-domain, but with all . characters replaced with a _, as dots cannot be used in a variable name. For example, the variable for example.com would become acme_challenge_example_com.

You’ll continue to work in letsencrypt-issue.yml in the next step, so don’t close this file yet.

You’ve written a task to submit your CSR to Let’s Encrypt. Next, you will add a task to implement the ACME challenge files for finalization of the certificate validation process.

Step 5 — Implementing the ACME Challenge Files

In this step, you will write an Ansible task to read and implement the ACME challenge files. These files prove that you’re eligible to acquire a certificate for the requested domains and subdomains.

The ACME challenge files must be served on a web server listening on port 80, at the /.well-known/acme-challenge/ path for the domain or subdomain that you’re requesting a certificate for. For example, in order to validate the certificate request for www.your-domain, the ACME challenge file will need to be accessible over the internet at the following path: http://www.your-domain/.well-known/acme-challenge.

The method for serving these files at the required destinations will vary significantly depending on your current web server setup. However, in this guide, we will assume that you have a web server (as per the prerequisite tutorial) configured to serve files out of the /var/www/html directory. Therefore you may need to adjust the task accordingly in order to be compatible with your own web server setup.

Firstly, add the following task that creates the .well-known/acme-challenge/ directory structure required to serve the files to the end of your playbook:

...
  - name: "Create .well-known/acme-challenge directory"
    file:
      path: /var/www/html/.well-known/acme-challenge
      state: directory
      owner: root
      group: root
      mode: u=rwx,g=rx,o=rx

Make sure to adjust the path accordingly if you are using a directory other than /var/www/html to serve files with your web server.

Next, you’ll implement the ACME challenge files that were saved into the acme_challenge_your-domain variable in Step 4 with the following task:

...
  - name: "Implement http-01 challenge files"
    copy:
      content: "{{ acme_challenge_your_domain['challenge_data'][item]['http-01']['resource_value'] }}"
      dest: "/var/www/html/{{ acme_challenge_your_domain['challenge_data'][item]['http-01']['resource'] }}"
      owner: root
      group: root
      mode: u=rw,g=r,o=r
    with_items:
    - "{{ domain_name }}"
    - "www.{{ domain_name }}"

Note that you need to manually adjust the acme_challenge_your_domain variable name in the task to be set to the name of your ACME challenge variable, which is acme_challenge_ followed by your domain name, but with all . characters replaced with _. This Ansible task copies the ACME validation files from the variable into the .well-known/acme-challenge path on your web server. This will allow Let’s Encrypt to retrieve them in order to verify the ownership of the domain and your eligibility to acquire a certificate.

You’ll continue to work in letsencrypt-issue.yml in the next step, so don’t close this file yet.

You’ve written the Ansible tasks required to create the ACME validation directory and files. Next, you will complete the ACME verification process and acquire the signed certificate.

Step 6 — Acquiring Your Certificate

In this step, you’ll write a task to trigger Let’s Encrypt to verify the ACME challenge files that you submitted, which will allow you to acquire your signed certificate(s).

The following task validates the ACME challenge files that you implemented in Step 5 and saves your signed certificates to the specified paths. Add it to the end of your playbook:

...
  - name: "Complete Let's Encrypt challenges"
    letsencrypt:
      acme_directory: "{{ acme_directory }}"
      acme_version: "{{ acme_version }}"
      account_key_src: "{{ letsencrypt_account_key }}"
      account_email: "{{ acme_email }}"
      challenge: "{{ acme_challenge_type }}"
      csr: "{{ letsencrypt_csrs_dir }}/{{ domain_name }}.csr"
      dest: "{{ letsencrypt_certs_dir }}/{{ domain_name }}.crt"
      chain_dest: "{{ letsencrypt_certs_dir }}/chain_{{ domain_name }}.crt"
      fullchain_dest: "{{ letsencrypt_certs_dir }}/fullchain_{{ domain_name }}"
      data: "{{ acme_challenge_your_domain }}"

Similarly to Step 4, this task makes use of the variables that you configured in Step 1. Once the task has completed, it will save the signed certificate to the specified paths, allowing you to begin using it for your application or service.

Note that you’ll need to manually adjust the data value in the task to be set to the name of your ACME challenge variable, similarly to Step 5.

Following is the full playbook showing each of the tasks you’ve added:

- hosts: "host1"
  tasks:

  - name: "Create required directories in /etc/letsencrypt"
    file:
      path: "/etc/letsencrypt/{{ item }}"
      state: directory
      owner: root
      group: root
      mode: u=rwx,g=x,o=x
    with_items:
    - account
    - certs
    - csrs
    - keys

  - name: "Generate a Let's Encrypt account key"
    shell: "if [ ! -f {{ letsencrypt_account_key }} ]; then openssl genrsa 4096 | sudo tee {{ letsencrypt_account_key }}; fi"

  - name: "Generate Let's Encrypt private key"
    shell: "openssl genrsa 4096 | sudo tee /etc/letsencrypt/keys/{{ domain_name }}.key"

  - name: "Generate Let's Encrypt CSR"
    shell: "openssl req -new -sha256 -key /etc/letsencrypt/keys/{{ domain_name }}.key -subj \"/CN={{ domain_name }}\" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf \"\n[SAN]\nsubjectAltName=DNS:{{ domain_name }},DNS:www.{{ domain_name }}\")) | sudo tee /etc/letsencrypt/csrs/{{ domain_name }}.csr"
    args:
      executable: /bin/bash

  - name: "Begin Let's Encrypt challenges"
    letsencrypt:
      acme_directory: "{{ acme_directory }}"
      acme_version: "{{ acme_version }}"
      account_key_src: "{{ letsencrypt_account_key }}"
      account_email: "{{ acme_email }}"
      terms_agreed: 1
      challenge: "{{ acme_challenge_type }}"
      csr: "{{ letsencrypt_csrs_dir }}/{{ domain_name }}.csr"
      dest: "{{ letsencrypt_certs_dir }}/{{ domain_name }}.crt"
      fullchain_dest: "{{ letsencrypt_certs_dir }}/fullchain_{{ domain_name }}.crt"
      remaining_days: 91
    register: acme_challenge_your_domain

  - name: "Create .well-known/acme-challenge directory"
    file:
      path: /var/www/html/.well-known/acme-challenge
      state: directory
      owner: root
      group: root
      mode: u=rwx,g=rx,o=rx

  - name: "Implement http-01 challenge files"
    copy:
      content: "{{ acme_challenge_your_domain['challenge_data'][item]['http-01']['resource_value'] }}"
      dest: "/var/www/html/{{ acme_challenge_your_domain['challenge_data'][item]['http-01']['resource'] }}"
      owner: root
      group: root
      mode: u=rw,g=r,o=r
    with_items:
    - "{{ domain_name }}"
    - "www.{{ domain_name }}"

  - name: "Complete Let's Encrypt challenges"
    letsencrypt:
      acme_directory: "{{ acme_directory }}"
      acme_version: "{{ acme_version }}"
      account_key_src: "{{ letsencrypt_account_key }}"
      account_email: "{{ acme_email }}"
      challenge: "{{ acme_challenge_type }}"
      csr: "{{ letsencrypt_csrs_dir }}/{{ domain_name }}.csr"
      dest: "{{ letsencrypt_certs_dir }}/{{ domain_name }}.crt"
      chain_dest: "{{ letsencrypt_certs_dir }}/chain_{{ domain_name }}.crt"
      fullchain_dest: "{{ letsencrypt_certs_dir }}/fullchain_{{ domain_name }}"
      data: "{{ acme_challenge_your_domain }}"

Save and close your file when you’re finished.

You’ve added the task to complete the ACME challenges and acquire your signed certificate. Next, you’ll run the playbook against your Ansible host machine in order to run all of the actions.

Step 7 — Running Your Playbook

Now that you’ve written the playbook and all of the required tasks, you can run it against your Ansible host machine to issue the certificate.

From your Ansible server, you can run the playbook using the ansible-playbook command:

ansible-playbook letsencrypt-issue.yml

This will run the playbook, one task at a time. You’ll see output similar to the following:

Output
PLAY [host1] **********************************************************************************

TASK [Gathering Facts] ************************************************************************
ok: [host1]

TASK [Create required directories in /etc/letsencrypt] ****************************************
changed: [host1] => (item=account)
changed: [host1] => (item=certs)
changed: [host1] => (item=csrs)
changed: [host1] => (item=keys)

TASK [Generate a Let's Encrypt account key] ***************************************************
changed: [host1]

TASK [Generate Let's Encrypt private key] *****************************************************
changed: [host1]

TASK [Generate Let's Encrypt CSR] *************************************************************
changed: [host1]

TASK [Begin Let's Encrypt challenges] *********************************************************
changed: [host1]

TASK [Create .well-known/acme-challenge directory] ********************************************
changed: [host1]

TASK [Implement http-01 challenge files] ******************************************************
changed: [host1] => (item=your-domain)
changed: [host1] => (item=www.your-domain)

TASK [Complete Let's Encrypt challenges] ******************************************************
changed: [host1]

PLAY RECAP ************************************************************************************
host1                      : ok=9    changed=8    unreachable=0    failed=0

If any errors are encountered while the playbook is running, these will be outputted for your review.

Once the playbook has finished, your valid Let’s Encrypt certificate will be saved to the /etc/letsencrypt/certs directory on your host machine. You can then use this, along with the private key in /etc/letsencrypt/keys, to secure connections to your web server, mail server, etc.

Let’s Encrypt certificates are valid for 90 days by default. You will receive renewal reminders via email to the address that you specified in Step 1. To renew your certificate, you can run the playbook again. Make sure to double check that any services using your certificate have picked up the new one, as sometimes you may need to manually install it, move it to a particular directory, or restart the service for it to properly adopt the new certificate.

In this step, you ran your playbook which issued your valid Let’s Encrypt certificate.

Conclusion

In this article you wrote an Ansible playbook to request and acquire a valid Let’s Encrypt certificate.

As a next step, you can look into using your new playbook to issue certificates for a large fleet of servers. You could even create a central ACME validation server that can issue certificates centrally and distribute them out to web servers.

Finally, if you’d like to learn more about the ACME specification and Let’s Encrypt project, you may wish to review the following links:

• Let’s Encrypt website and documentation.

• Ansible acme_certificate Module.

• RFC8555 - Automated Certificate Management Environment (ACME).

You may also like to view some other relevant Ansible tutorials:

• How To Use Ansible: A Reference Guide.

• How To Automate Server Setup with Ansible on Ubuntu 18.04.

• How To Use Vault to Protect Sensitive Ansible Data on Ubuntu 16.04.

fatal: [test_server]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'domain.me'\n\nThe error appears to be in '/Users/name/Desktop/test_ansible/ansible/staging/roles/letsencrypt/tasks/main.yml': line 60, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: \"Implement http-01 challenge files\"\n    ^ here\n"}

- name: "Implement http-01 challenge files"
    copy:
      content: "{{ acme_challenge_domain_me['challenge_data'][item]['http-01']['resource_value'] }}"
      dest: "/var/www/{{domain_name}}/{{ acme_challenge_domain_me['challenge_data'][item]['http-01']['resource'] }}"
      owner: root
      group: root
      mode: u=rw,g=r,o=r
    with_items:
    - "{{ domain_name }}"
    - "www.{{ domain_name }}"

FAILED! => {"changed": false, "msg": "ACME directory does not seem to follow protocol ACME v2", "other": {}}