Tutorial

How To Acquire a Let's Encrypt Certificate Using DNS Validation with acme-dns-certbot on Ubuntu 18.04

How To Acquire a Let's Encrypt Certificate Using DNS Validation with acme-dns-certbot on Ubuntu 18.04

The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

Introduction

The majority of Let’s Encrypt certificates are issued using HTTP validation, which allows for the easy installation of certificates on a single server. However, HTTP validation is not always suitable for issuing certificates for use on load-balanced websites, nor can it be used to issue wildcard certificates.

DNS validation allows for certificate issuance requests to be verified using DNS records, rather than by serving content over HTTP. This means that certificates can be issued simultaneously for a cluster of web servers running behind a load balancer, or for a system that isn’t directly accessible over the internet. Wildcard certificates are also supported using DNS validation.

The acme-dns-certbot tool is used to connect Certbot to a third-party DNS server where the certificate validation records can be set automatically via an API when you request a certificate. The advantage of this is that you don’t need to integrate Certbot directly with your DNS provider account, nor do you need to grant it unrestricted access to your full DNS configuration, which is beneficial to security.

Delegated DNS zones are used in order to redirect lookups for the certificate verification records to the third-party DNS service, so once the initial setup has been completed, you can request as many certificates as you want without having to perform any manual validation.

Another key benefit of acme-dns-certbot is that it can be used to issue certificates for individual servers that may be running behind a load balancer, or are otherwise not directly accessible over HTTP. Traditional HTTP certificate validation cannot be used in these cases, unless you set the validation files on each and every server. The acme-dns-certbot tool is also useful if you want to issue a certificate for a server that isn’t accessible over the internet, such as an internal system or staging environment.

In this tutorial, you will use the acme-dns-certbot hook for Certbot to issue a Let’s Encrypt certificate using DNS validation.

Prerequisites

To complete this tutorial, you will need:

  • An Ubuntu 18.04 server set up by following the Initial Server Setup with Ubuntu 18.04, including a sudo non-root user.

  • A domain name for which you can acquire a TLS certificate, including the ability to add DNS records. In this particular example, we will use your-domain and subdomain.your-domain, as well as *.your-domain for a wildcard certificate. However this can be adjusted for other domain, subdomains, or wildcards if required.

Once you have these ready, log in to your server as your non-root user to begin.

Step 1 — Installing Certbot

In this step, you will install Certbot, which is a program used to issue and manage Let’s Encrypt certificates.

Certbot is available within the official Ubuntu Apt repositories, however, it is instead recommended to use the repository maintained by the Certbot developers, as this always has the most up-to-date version of the software.

Begin by adding the Certbot repository:

sudo apt-add-repository ppa:certbot/certbot

You’ll need to press ENTER to accept the prompt and add the new repository to your system.

Next, install the Certbot package:

sudo apt install certbot

Once the installation has completed, you can check that Certbot has been successfully installed:

certbot --version

This will output something similar to the following:

Output
certbot 0.31.0

In this step you installed Certbot. Next, you will download and install the acme-dns-certbot hook.

Step 2 — Installing acme-dns-certbot

Now that the base Certbot program has been installed, you can download and install acme-dns-certbot, which will allow Certbot to operate in DNS validation mode.

Begin by downloading a copy of the script:

Note: As a best practice please make sure to review this Github repository and the script before running it. You can also first fork this repository and then make use of the acme-dns-certbot.py script. This fork provides an extra layer of security, ensuring that the script remains under our control and is less susceptible to unverified changes.

wget https://github.com/joohoi/acme-dns-certbot-joohoi/raw/master/acme-dns-auth.py

Once the download has completed, mark the script as executable:

chmod +x acme-dns-auth.py

Then, edit the file using your favorite text editor and adjust the first line in order to force it to use Python 3:

nano acme-dns-auth.py

Add a 3 to the end of the first line:

acme-dns-certbot.py
#!/usr/bin/env python3
. . .

This is required in order to ensure that the script uses the latest supported version of Python 3, rather than the legacy Python version 2.

Once complete, save and close the file.

Finally, move the script into the Certbot Let’s Encrypt directory so that Certbot can load it:

sudo mv acme-dns-auth.py /etc/letsencrypt/

In this step, you downloaded and installed the acme-dns-certbot hook. Next, you can begin the setup process and work toward issuing your first certificate.

Step 3 — Setting Up acme-dns-certbot

In order to begin using acme-dns-certbot, you’ll need to complete an initial setup process and issue at least one certificate.

Start by running Certbot to force it to issue a certificate using DNS validation. This will run the acme-dns-certbot script and trigger the initial setup process:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d \*.your-domain -d your-domain

You use the --manual argument to disable all of the automated integration features of Certbot. In this case you’re just issuing a raw certificate, rather than automatically installing it on a service as well.

You configure Certbot to use the acme-dns-certbot hook via the --manual-auth-hook argument. You run the --preferred-challenges argument so that Certbot will give preference to DNS validation.

You must also tell Certbot to pause before attempting to validate the certificate, which you do with the --debug-challenges argument. This is to allow you to set the DNS CNAME record(s) required by acme-dns-certbot, which is covered later in this step. Without the --debug-challenges argument, Certbot wouldn’t pause, so you wouldn’t have time to make the required DNS change.

Remember to substitute each of the domain names that you wish to use using -d arguments. If you want to issue a wildcard certificate, make sure to escape the asterisk (*) with a backslash (\).

After following the standard Certbot steps, you’ll eventually be prompted with a message similar to the following:

Output
... Output from acme-dns-auth.py: Please add the following CNAME record to your main DNS zone: _acme-challenge.your-domain CNAME a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io. Waiting for verification... ...

You’ll need to add the required DNS CNAME record to the DNS configuration for your domain. This will delegate control of the _acme-challenge subdomain to the ACME DNS service, which will allow acme-dns-certbot to set the required DNS records to validate the certificate request.

If you’re using DigitalOcean as your DNS provider, you can set the DNS record within your control panel:

A screenshot of the DigitalOcean DNS control panel, showing an example of a CNAME record for ACME DNS

It is recommended to set the TTL (time-to-live) to around 300 seconds in order to help ensure that any changes to the record are propagated quickly.

Once you have configured the DNS record, return to Certbot and press ENTER to validate the certificate request and complete the issuance process.

This will take a few seconds, and you’ll then see a message confirming that the certificate has been issued:

Output
... Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/your-domain/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/your-domain/privkey.pem ...

You’ve run acme-dns-certbot for the first time, set up the required DNS records, and successfully issued a certificate. Next you’ll set up automatic renewals of your certificate.

Step 4 — Using acme-dns-certbot

In this final step, you will use acme-dns-certbot to issue more certificates and renew existing ones.

Firstly, now that you’ve successfully issued at least one certificate using acme-dns-certbot, you can continue to issue certificates for the same DNS names without having to add another DNS CNAME record. However, if you wish to acquire a certificate for a different subdomain or entirely new domain name, you will be prompted to add another CNAME record.

For example, you could issue another standalone wildcard certificate without having to perform the verification again:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d \*.your-domain

However, if you were to attempt to issue a certificate for a subdomain, you would be prompted to add a CNAME record for the subdomain:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d subdomain.your-domain

This will show an output similar to the initial setup that you carried out in Step 3:

Output
... Please add the following CNAME record to your main DNS zone: _acme-challenge.subdomain.your-domain CNAME 8450fb54-8e01-4bfe-961a-424befd05088.auth.acme-dns.io. Waiting for verification... ...

Now that you’re able to use acme-dns-certbot to issue certificates, it’s worth considering the renewal process as well.

Once your certificates are nearing expiry, Certbot can automatically renew them for you:

sudo certbot renew

The renewal process can run start-to-finish without user interaction, and will remember all of the configuration options that you specified during the initial setup.

To test that this is working without having to wait until nearer the expiry date, you can trigger a dry run. This will simulate the renewal process without making any actual changes to your configuration.

You can trigger a dry run using the standard renew command, but with the --dry-run argument:

sudo certbot renew --dry-run

This will output something similar to the following, which will provide assurance that the renewal process is functioning correctly:

Output
... Cert not due for renewal, but simulating renewal for dry run Plugins selected: Authenticator manual, Installer None Renewing an existing certificate Performing the following challenges: dns-01 challenge for your-domain dns-01 challenge for your-domain Waiting for verification... Cleaning up challenges ...

In this final step, you issued another certificate and then tested the automatic renewal process within Certbot.

Conclusion

In this article you set up Certbot with acme-dns-certbot in order to issue certificates using DNS validation. This unlocks the possibility of using wildcard certificates as well as managing a large estate of distinct web servers that may be sitting behind a load balancer.

Make sure to keep an eye on the acme-dns-certbot repository for any updates to the script, as it’s always recommended to run the latest supported version.

If you’re interested in learning more about acme-dns-certbot, you may wish to review the documentation for the acme-dns project, which is the server-side element of acme-dns-certbot:

The acme-dns software can also be self-hosted, which may be beneficial if you’re operating in high-security or complex environments.

Alternatively, you could dig into the technical details of ACME DNS validation by reviewing the relevant section of the official RFC document which outlines how the process works:

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products

About the author(s)

Jamie Scaife
Jamie ScaifeSecurity Engineer
See author profile
Category:
Tutorial

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
10 Comments
Leave a comment...

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

How can I auto renew the certification with dns verification?

Hi @Cybil,

Thanks for your comment.

By default, Certbot will automatically install a scheduled system task to renew your certificates. You can check for it using the following command:

systemctl show certbot.timer

However, if you’re using certonly mode to issue plain certificates, your web server or other application may need to be restarted/reloaded in order for it to pick up the new certificate after renewal.

You may wish to add your own cronjob to do this, however as long as your server is rebooted at least every couple of weeks, and you are monitoring for certificate validity/expiry, you should be fine.

Thanks

Does the ACMEDNS_URL needs to be changed? And if so, what should it be replaced with? Great article but I am confused on that ACMEDNS_URL part.

Hi @iwebhost4u,

The ACMEDNS_URL is used to specify which acme-dns instance/server your client points at. By default it will use the public cloud-based acme-dns instance, which is suitable for most use cases.

However, if you host your own acme-dns instance (which isn’t covered in this tutorial), you can edit the ACMEDNS_URL variable to point the client hook to it.

Thanks

Thank you, @jamieweb!

Just to be sure/clear, so if I don’t change the value of the ACMEDNS_URL and leave it as is, will I be able to get a valid certificate? I will be using --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py to generate the certificate and copy it to another server for installation. My command will be similar to the following:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d *.your-domain -d your-domain

@iwebhost4u Yes, the default value will allow you to acquire a valid certificate.

You only need to change it if you want to point your ACME client to your own acme-dns instance.

Ok, thanks a lot!

This comment has been deleted

    Hey @jamieweb thanks for writing this tutorial. However, I’m unable to get past step 3 because for each of my domains I’m getting the same error:

    Domain: subdomain.mydomain.dev
    Type:   dns
    Detail: DNS problem: SERVFAIL looking up TXT for 
    _acme-challenge.subdomain.mydomain.dev - the domain's nameservers
    may be malfunctioning
    
    Domain: mydomain.dev
    Type:   dns
    Detail: DNS problem: NXDOMAIN looking up TXT for 
    _acme-challenge.mydomain.dev - check that a DNS record exists for
    this domain
    

    I’m new to using Google domains, and have not created any TXT resource records. Is that a prerequisite? Or if the domain registration was in the past 24hrs, do I just need to wait a couple days?

    Hi @halpoins,

    Are you running Certbot with the --debug-challenges option? You’ll need to do this in order for Certbot to pause to give you time to set up the relevant CNAME record(s) for your domain.

    During the pause, a message should appear saying: “Please add the following CNAME record to your main DNS zone”.

    Please let me know if this doesn’t work and I’ll take a closer look.

    Thanks, Jamie

    I am also experiencing the same issue as above.

    Failed authorization procedure. mydomain.com (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "v=spf1 ip4:165.227.103.35 -all" found at _acme-challenge.mydomain.com
    
    IMPORTANT NOTES:
     - The following errors were reported by the server:
    
       Domain: mydomain.com
       Type:   unauthorized
       Detail: Incorrect TXT record "v=spf1 ip4:165.227.103.35 -all" found
       at _acme-challenge.tefcor.com
    
       To fix these errors, please make sure that your domain name was
       entered correctly and the DNS A/AAAA record(s) for that domain
       contain(s) the right IP address.
    

    It looks like it’s evaluating my TXT record, which is unrelated, that record is for my email server verification under the name.

    I do have the CNAME record setup as per instructions. _acme-challenge.mydomain.com for CNAME

    Should we not use TXT record instead of CNAME?

    dkim._domainkey.mydomain.com

    Hi @martin.jozef,

    The CNAME record on _acme-challenge is used to delegate control over that DNS name to the ACME DNS server, which will serve the relevant verification records.

    It looks like this CNAME isn’t working correctly, and instead your wildcard SPF record is being served. Can you check that the CNAME record is definitely set properly for the domain and all relevant subdomains?

    Thanks, Jamie

    I found it easier on Ubuntu to simply sudo apt install python3-certbot-dns-digitalocean (which depends on certbot). Then, after adding a token to the secret file per the module documentation, run:

    certbot certonly \
      --dns-digitalocean \
      --dns-digitalocean-credentials ~/.secrets/certbot/digitalocean.ini \
      -d example.com -d somethingelse.example.com
    

    That’s probably not packaged up for all operating systems, but it’s certainly quite painless on Ubuntu. :)

    There’s a tutorial in the pipeline about this. :) It only works if you’re hosting your DNS with DigitalOcean though, or have at least delegated part of your zone there.

    I have problem too. What i have to put into the DNS zone editor? CNAME name “_acme-challenge.domain” with value “hash.auth.acme-dns.io.” or CNAME name “_acme-challenge.domain” with value “hash.auth.acme-dns.io” or TXT name “_acme-challenge.domain” with value “hash.auth.acme-dns.io.” or TXT name “_acme-challenge.domain” with value “hash.auth.acme-dns.io” I’m not sure is this dot is the end of the sentence or it should exist? But in the all ways i receive: The following errors were reported by the server: Domain: stream.club-bg.org Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.domain - check that a DNS record exists for this domain

    Hi @Nicox,

    You’ll need to set a CNAME record pointing to the ACME DNS address. The dot at the end isn’t required in all DNS control panels, but there’s no harm in adding it.

    The TXT records are all managed by ACME DNS, so you don’t need to do anything with those.

    Hope this helps. :)

    Kind regards, Jamie

    The strange thing is that, the TXT worked out, but it can’t recognize the key. I add the record as TXT with name “acme-challenge.domain.” (with point at the end) with value “hash.auth.acme-dns.io.”. And in this way i received:

    IMPORTANT NOTES: The following errors were reported by the server: Domain: domain Type: unauthorized Detail: Incorrect TXT record “hash.auth.acme-dns.io.” found at _acme-challenge.domain To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

    Hi @Nicox,

    All of the TXT records are handled by the ACME DNS service, so you definitely don’t need to set any yourself, as this won’t work.

    Have you had any luck adding the CNAME record as detailed during the setup process? You’ll need to add a CNAME record for each unique subdomain, but once you’ve done that, you can issue as many certificates as you need. :)

    Kind regards, Jamie

    Hi!

    Do I need to enable SSL in nginx by adding “ssl” to the listen directive using this method as well? Do I also need to link the SSL key using ssl_certificate in the server directive?

    Edit; yes you do! Working now

    Hi @Ecaz,

    Sorry for the delay! It looks like you’ve already worked this out, but let me know if there’s anything else I can help with.

    Thanks, Jamie

    Have some issue to acme-dns-certbot, so ended up using default certbot

    certbot -d app.mydomain.com --manual --preferred-challenges dns certonly

    Output

    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator manual, Installer None
    Obtaining a new certificate
    Performing the following challenges:
    dns-01 challenge for app.mydomain.com
    
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    NOTE: The IP of this machine will be publicly logged as having requested this
    certificate. If you're running certbot in manual mode on a machine that is not
    your server, please ensure you're okay with that.
    
    Are you OK with your IP being logged?
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    (Y)es/(N)o: Y
    
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Please deploy a DNS TXT record under the name
    _acme-challenge.app.mydomain.com with the following value:
    
    xxxxxxxxx
    
    Before continuing, verify the record is deployed.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Press Enter to Continue
    Waiting for verification...
    
    

    Now, go and add the TXT record for _acme-challenge.app.mydomain.com. use https://dnschecker.org/ to validate that it works.

    Come back and press Enter.

    Cleaning up challenges
    
    IMPORTANT NOTES:
     - Congratulations! Your certificate and chain have been saved at:
    

    Thanks for the great tutorial! Verified all steps working 5/4/21.

    For what it is worth, there is comparable newer software available from the same author at https://github.com/acme-dns/acme-dns-client.

    That might be worth a try, though it is still an early version.

    Nice work!

    Join the Tech Talk
    Success! Thank you! Please check your email for further details.

    Please complete your information!

    Become a contributor for community

    Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

    DigitalOcean Documentation

    Full documentation for every DigitalOcean product.

    Resources for startups and SMBs

    The Wave has everything you need to know about building a business, from raising funding to marketing your product.

    Get our newsletter

    Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

    New accounts only. By submitting your email you agree to our Privacy Policy

    The developer cloud

    Scale up as you grow — whether you're running one virtual machine or ten thousand.

    Get started for free

    Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

    *This promotional offer applies to new accounts only.