How To Securely Manage Secrets with HashiCorp Vault on Ubuntu 20.04

Introduction

Vault is an open-source tool that provides a secure, reliable way to store and distribute secrets like API keys, access tokens, and passwords. Software like Vault can be critically important when deploying applications that require the use of secrets or sensitive data.

In this tutorial, you will:

• Install Vault and configure it as a system service
• Initialize an encrypted on-disk data store
• Store and retrieve a sensitive value securely over TLS

With some additional policies in place, you’ll be able to use Vault to securely manage sensitive data for your various applications and tools.

As with any service that manages sensitive information, you should consider reading additional documentation regarding Vault’s deployment best practices before using it in a production-like environment. For example, Vault’s production hardening guide covers topics such as policies, root tokens, and auditing.

Prerequisites

Before you begin this guide you’ll need the following:

• One Ubuntu 20.04 server set up by following the initial setup guide for Ubuntu 20.04, including a sudo non-root user and a firewall.
• A TLS certificate, which we will use to secure Vault’s HTTP API. You can get one for free by following this Certbot standalone mode tutorial for Ubuntu 20.04.

Step 1 — Installing Vault

HashiCorp provides Vault as a typical Debian/Ubuntu package, so we’ll go through the normal steps of adding their package repository to our server’s list of package sources:

First, add Hashicorp’s GPG key to your package manager, so that your system trusts their package repositories:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

Then add the repository itself to your list of package sources, so it’ll be checked for regular updates:

sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

Then install the package:

sudo apt install vault

You can now use the vault command. Try checking Vault’s version to make sure it works.

vault --version

Output
Vault v1.8.5 (647eccfe0bd5817bdd8628f3c3171402dfc8a8fc)

The Vault executable is installed on your server, so the next step is to configure it to run as a system service.

Step 2 — Configuring Vault

Installing the Vault package automatically creates a vaultvault user on your system and sets up a system service for running Vault in the background. We need to make a couple of changes to its default configuration in order to use the HTTPS certificates generated by Let’s Encrypt.

Vault’s default configuration is stored in /etc/vault.d/vault.hcl. You’ll use this file to control various options in Vault, such as where encrypted secrets are stored.

Open vault.hcl using nano or your favorite text editor.

sudo nano /etc/vault.d/vault.hcl

Find the listener "tcp" section of the file under #HTTPS listener that contains this block. If you are using nano, you can press Ctrl+W then enter listener “tcp” to find that line directly:

...
#HTTPS listener
...
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
...
}

Edit the tls_cert_file and tls_key_file lines to point to your Let’s Encrypt certificate and key files. Don’t forget to substitute in your own domain name in place of the highlighted your_domain part of each line.

listener "tcp" {
...
  tls_cert_file = "/etc/letsencrypt/live/your_domain/fullchain.pem"
  tls_key_file = "/etc/letsencrypt/live/your_domain/privkey.pem"
}

Save and close the file. If you are using nano, press Ctrl+X, then Y when prompted to save the file, and Enter to confirm.

Next, the vault system user also needs permission to read these certificates. By default, these certificates and private keys are only accessible by root. To make these available securely, we’ll create a special group called pki to access these files. We will create the group and then add the vault user to it.

sudo groupadd pki

Update the permissions on the two directories in the /etc/letsencrypt directory to allow members of the pki group to read the contents.

sudo chgrp -R pki /etc/letsencrypt/archive
sudo chgrp -R pki /etc/letsencrypt/live
sudo chmod -R g+rx /etc/letsencrypt/archive
sudo chmod -R g+rx /etc/letsencrypt/live

Then add the vault user to the pki group. This will grant Vault access to the certificates so that it can serve requests securely over HTTPS.

sudo usermod -a -G pki vault

As a final step for convenience, add a rule in /etc/hosts to direct requests to Vault to localhost.

Replace your_domain in the following command with the domain you acquired the Let’s Encrypt certificate for:

echo 127.0.0.1 your_domain.com | sudo tee -a /etc/hosts

This command appends the line 127.0.0.1 your_domain.com to /etc/hosts so that any HTTP requests that you make on your Vault server to your_domain.com ignore DNS and are sent to localhost directly.

With the Vault service set up and the Vault configuration file complete, we’re now ready to start Vault and initialize the secret store.

Step 3 — Initializing Vault

When you first start Vault, it will be uninitialized, which means that it isn’t ready to receive and store data. In this section of the tutorial, you will start the Vault server, and then initialize it with a set of secret keys that will be used to unseal (open) Vault’s secret stores.

The first time you start Vault, the backend that actually stores the encrypted secrets is uninitialized, too. Start the Vault system service to initialize the backend and start running Vault itself.

sudo systemctl start vault.service

You can run a quick check to confirm the service has started successfully.

sudo systemctl status vault.service

You should receive output similar to the following:

Output
● vault.service - "HashiCorp Vault - A tool for managing secrets"
     Loaded: loaded (/lib/systemd/system/vault.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-11-16 21:55:13 UTC; 22s ago
       Docs: https://www.vaultproject.io/docs/
   Main PID: 104207 (vault)
      Tasks: 7 (limit: 1137)
     Memory: 179.3M
     CGroup: /system.slice/vault.service
             └─104207 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

The output of that command should include several pieces of information about the running service, such as its process ID and resource usage. Ensure that the following line is included in the output, which indicates that the service is running correctly.

Output
. . .
Active: active (running)
. . .

If the service is not active, take a look at the accompanying log lines at the end of the command’s output to see Vault’s output, which can help pinpoint any issues.

Next, we’ll set an environment variable to tell the vault command how to connect to the Vault server. In Step 1 you configured Vault to only listen on the local loopback interface, so set the VAULT_ADDR environment variable to the local HTTPS endpoint.

export VAULT_ADDR=https://your_domain:8200

The vault command can now communicate with the daemon. Note that defining the actual hostname instead of simply localhost or 127.0.0.1 is necessary to properly validate the HTTPS certificate.

Confirm that the vault server is in an uninitialized state by checking its status.

vault status

The server should return some output so that you can tell it is working but not yet initialized.

Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.8.5
Storage Type file
HA Enabled false

There are two pieces of information that Vault will expose at initialization time that will not be available at any other point:

• Initial root token. This is equivalent to root permissions to your Vault deployment, which allows the management of all Vault policies, mounts, and secrets.
• Unseal keys. These are used to unseal Vault when the daemon starts, which permits the Vault daemon to decrypt the backend secret store.

More specifically, Vault’s unsealing process decrypts the backend using a key formed by key shares. When you first initialize Vault, you may choose how many unseal keys to create and how many are necessary at unseal time to successfully unseal Vault. To learn more about Vault’s sealing mechanism, you can refer to the Vault documentation.

A typical configuration for the unseal parameters would be to create three keys and require at least two of those keys at unseal time. This permits the important key shares to be separated and stored in distinct locations to ensure that compromising one is not sufficient to unseal Vault.

In other words, whenever Vault is started, at least two unseal keys will be required in order to make the service become available and ready to use. While sealed, the files that store the actual secret values will remain encrypted and inaccessible.

Initialize Vault with three unseal keys using the -key-shares=3 option, and require at least two keys to unseal the vault with the -key-threshold=2 flag::

vault operator init -key-shares=3 -key-threshold=2

You will receive output like the following:

Output
Unseal Key 1: eZcJeydRrqeSMZ1zTN+VVll9TFT2KvJy7VlnxUgtvuz5
Unseal Key 2: ntmqCKq8rgNnKT1YSLCjVmCCZBAA3NwUeqxIyRpYD4Wm
Unseal Key 3: 3FK1+Hsorh4p8/L9mki3VskaEU2eQhLqGOI/pJkTHMbx

Initial Root Token: s.hY0ieybfDqCadz7JpL88uO3x

Be sure to save each unseal token and the initial root token in a secure way. You will not be able to retrieve these keys and root token again. For example, one option would be to store one unseal key in a password manager, another on a USB drive, and another in a GPG-encrypted file.

If you examine vault status again, the Initialized status will now be set to true, and the Total Shares and Threshold values will reflect the number of key shards and minimum number of keys that you will need to unseal the vault.

vault status

Output
. . .
Initialized     true
Sealed         	true
Total Shares   	3
Threshold      	2
Unseal Progress	0/2
. . .

Note that the Unseal Progess line shows the value 0/2. Begin unsealing Vault using your newly created unseal tokens. Run the vault operator unseal command and input any of your keys when prompted:

vault operator unseal

The command will ask for an unseal token:

Output
Key (will be hidden):

After entering it, the output from the command will indicate that the unsealing is in progress, but still requires one more unsealing key before Vault is ready for use.

Output
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       3
Threshold          2
Unseal Progress    1/2
Unseal Nonce       0f3a328b-e0c6-6294-d6a2-56da49271dff
Version            1.8.5
Storage Type       file
HA Enabled         false

Notice how the Unseal Progress 1/2 line has changed in the output. Run the unseal command again.

vault operator unseal

And enter a different key than the one you already used:

Output
Key (will be hidden):

The command’s output indicates that the unseal process had completed successfully.

Output
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    3
Threshold       2
Version         1.8.5
Storage Type    file
Cluster Name    vault-cluster-3042c7bc
Cluster ID      c3e9d814-cf2a-2901-f0e4-ebc52d29e5cc
HA Enabled      false

Vault is now unsealed and ready for use. These unseal steps are necessary whenever Vault is started or restarted.

However, unsealing is a distinct process from normal interaction with Vault (such as reading and writing values), which are authenticated by tokens. In the next steps, we’ll create the necessary access tokens and policies to store secret values and read/write to specific paths in Vault.

Step 4 — Reading and Writing Secrets

There are several secret backends that you can use with Vault, but for this example we will use the kv secret backend. This backend stores simple key/value pairs in Vault. However, it is not enabled by default.

In this section of the tutorial, you will enable the kv secret backend, and then learn how to read and write secrets to it.

First, save the previously generated root token to a shell variable for ease of use.

root_token=your_root_token_here

Next, while authenticating with the root token, enable the kv backend:

VAULT_TOKEN=$root_token vault secrets enable kv

You will receive output like the following if the command is successful:

Output
Success! Enabled the kv secrets engine at: kv/

You can then verify that it’s been added to your local list of available secrets backends:

VAULT_TOKEN=$root_token vault secrets list

You should receive output like the following:

Output
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_abc1631b    per-token private secret storage
identity/     identity     identity_631fe262     identity store
kv/           kv           kv_4d5855c8           n/a
sys/          system       system_79b13f2f       system endpoints used for control, policy and debugging

Note the highlighted line that indicates the new kv backend is enabled. Now we can store some data with this backend.

VAULT_TOKEN=$root_token vault write kv/message value=mypassword

In this command, the highlighted kv/ prefix indicates that we are writing to the kv backend mounted at the kv path, and we are storing the key value at the path message with the value mypassword. We used the root token, which has superuser privileges, to write the generic secret.

Check the secret that you created using the vault read command:

VAULT_TOKEN=$root_token vault read kv/message

You should receive output like the following, with the contents of the secret that you created:

Output
Key             	Value
---             	-----
refresh_interval	768h
value           	mypassword

However, creating, reading, and otherwise interacting with Vault using the Root Token only is not secure, scalable in a team setting, and does not allow for fine-grained access control to secrets. In the next section you’ll learn how to define policies and create additional access tokens to restrict how users can interact with Vault.

Step 5 — Creating an Authorization Policy

In a real-world scenario, you may store values like API keys or passwords that external tools can consume. Although you may read the secret value again using the root token, it is illustrative to generate a less privileged token with read-only permissions to our single secret.

In this section of the tutorial you will create a Vault policy that will enforce read-only access to secrets. To create the policy you’ll need to edit a file and then load it into Vault using the vault policy command.

To get started creating a policy, open a file called policy.hcl using nano or your preferred editor:

nano policy.hcl

Populate the file with the following Vault policy, which defines read-only access to the secret path::

path "kv/message" {
     capabilities = ["read"]
}

Save and close the file, then write this policy to Vault. The following command will load the policy.hcl file into Vault and create a policy named message-readonly with the read-only capability

VAULT_TOKEN=$root_token vault policy write message-readonly policy.hcl

Next, create a token that you will use for read-only access to Vault. Add the -policy=”message-readonly” flag to the vault token create command to use the new policy that you created:

VAULT_TOKEN=$root_token vault token create -policy="message-readonly"

The output will look like this:

Output
Key                  Value
---                  -----
token                your_token_value
token_accessor       your_token_accessor
token_duration       768h0m0s
token_renewable      true
token_policies       ["default" "message-readonly"]
identity_policies    []
policies             ["default" "message-readonly"]

Save the highlighted your_token_value value to an environment variable called app_token:

app_token=your_token_value

You can use the value of app_token to access the data stored in the path kv/message (and no other values in Vault).

VAULT_TOKEN=$app_token vault read kv/message

Output
Key                     Value
---                     -----
refresh_interval        768h0m0s
value                   mypassword

You can also test that this unprivileged token cannot perform other operations, such as listing secrets in Vault.

VAULT_TOKEN=$app_token vault list kv/

Output
Error reading kv/: Error making API request.

URL: GET https://your_domain:8200/v1/secret?list=true
Code: 403. Errors:

* 1 error occurred:
    * permission denied

This verifies that the less-privileged app token cannot perform any destructive actions or access other secret values aside from those explicitly stated in its Vault policy. If you would like to continue using the read-only token, be sure to record it somewhere safe for future use.

Conclusion

In this article you installed, configured, and deployed Vault on Ubuntu 20.04. You also created a sharded key to unseal Vault, enabled the kv backend secret store, and defined a read-only policy to limit how a user can interact with Vault secrets.

Although this tutorial only demonstrated how to use an unprivileged token, the Vault documentation has more examples of additional ways to store and access secrets as well as alternative authentication methods.

These instructions demonstrated how to deploy and use some of the core features of Vault. Your needs may require other configuration changes and more complex policies. Be sure to read the Vault documentation and make appropriate configuration changes for your needs. Some production-ready changes may include:

• Generating lesser-privileged tokens for everyday use. The specific policies that these tokens should use depends on your specific use cases, but the example app_token in this tutorial illustrates how you can create limited-privilege tokens and policies.

• If you are deploying Vault as part of a service that will be used by a team, initialize Vault with unseal keys for each team member. This approach can ensure that Vault’s storage is only decrypted when more than one team member participates in the process.