How To Create Reusable Infrastructure with Terraform Modules and Templates

The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

Introduction

One of the main benefits of Infrastructure as Code (IAC) is reusing parts of the defined infrastructure. In Terraform, you can use modules to encapsulate logically connected components into one entity and customize them using input variables you define. By using modules to define your infrastructure at a high level, you can separate development, staging, and production environments by only passing in different values to the same modules, which minimizes code duplication and maximizes conciseness.

You are not limited to using only your custom modules. Terraform Registry is integrated into Terraform and lists modules and providers that you can incorporate in your project right away by defining them in the required_providers section. Referencing public modules can speed up your workflow and reduce code duplication. If you have a useful module and would like to share it with the world, you can look into publishing it on the Registry for other developers to use.

In this tutorial, you’ll explore some of the ways to define and reuse code in Terraform projects. You’ll reference modules from the Terraform Registry, separate development and production environments using modules, learn about templates and how they are used, and specify resource dependencies explicitly using the depends_on meta argument.

Prerequisites

• A DigitalOcean Personal Access Token, which you can create via the DigitalOcean control panel. You can find instructions in the DigitalOcean product documents, How to Create a Personal Access Token.
• Terraform installed on your local machine and a project set up with the DigitalOcean provider. Complete Step 1 and Step 2 of the How To Use Terraform with DigitalOcean tutorial and be sure to name the project folder terraform-reusability, instead of loadbalance. During Step 2, do not include the pvt_key variable and the SSH key resource.
• The droplet-lb module available under modules in terraform-reusability. Follow the How to Build a Custom Module tutorial and work through it until the droplet-lb module is functionally complete. (That is, until the cd ../.. command in the Creating a Module section.)
• Knowledge of Terraform project structuring approaches. For more information, see our tutorial How To Structure a Terraform Project.
• (Optional) Two separate domains whose nameservers are pointed to DigitalOcean at your registrar. Your domains must not yet be added to your DigitalOcean account. Refer to the How To Point to DigitalOcean Nameservers From Common Domain Registrars tutorial to set this up. Note that you don’t need to do this if you don’t plan on deploying the project you’ll create through this tutorial.

Separating Development and Production Environments

In this section, you’ll use modules to separate your target deployment environments. You’ll arrange these according to the structure of a more complex project. You’ll create a project with two modules: one will define the Droplets and Load Balancers, and the other will set up the DNS domain records. Afterward, you’ll write configuration for two different environments (dev and prod), which will call the same modules.

Creating the dns-records Module

As part of the prerequisites, you set up the initial project under terraform-reusability and created the droplet-lb module in its own subdirectory under modules. You’ll now set up the second module, called dns-records, containing variables, outputs, and resource definitions. From the terraform-reusability directory, create dns-records by running:

mkdir modules/dns-records

Navigate to it:

cd modules/dns-records

This module will contain the definitions for your domain and the DNS records that you’ll later point to the Load Balancers. You’ll first define the variables, which will become inputs that this module will expose. You’ll store them in a file called variables.tf. Create it for editing:

nano variables.tf

Add the following variable definitions:

variable "domain_name" {}
variable "ipv4_address" {}

Save and close the file. You’ll now define the domain and the accompanying A and CNAME records in a file named records.tf. Create and open it for editing by running:

nano records.tf

Add the following resource definitions:

resource "digitalocean_domain" "domain" {
  name = var.domain_name
}

resource "digitalocean_record" "domain_A" {
  domain = digitalocean_domain.domain.name
  type   = "A"
  name   = "@"
  value  = var.ipv4_address
}

resource "digitalocean_record" "domain_CNAME" {
  domain = digitalocean_domain.domain.name
  type   = "CNAME"
  name   = "www"
  value  = "@"
}

First, you add the domain name to your DigitalOcean account. The cloud will automatically add the three DigitalOcean nameservers as NS records. The domain name you supply to Terraform must not already be present in your DigitalOcean account, or Terraform will show an error during infrastructure creation.

Then, you define an A record for your domain, routing it (the @ as value signifies the true domain name, without subdomains) to the IP address supplied as the variable ipv4_address. The actual IP address will be passed in when you initialize an instance of the module. For the sake of completeness, the CNAME record that follows specifies that the www subdomain should also point to the same domain. Save and close the file when you’re done.

Next, you’ll define the outputs for this module. The outputs will show the FQDN (fully qualified domain name) of the created records. Create and open outputs.tf for editing:

nano outputs.tf

Add the following lines:

output "A_fqdn" {
  value = digitalocean_record.domain_A.fqdn
}

output "CNAME_fqdn" {
  value = digitalocean_record.domain_CNAME.fqdn
}

Save and close the file when you’re done.

With the variables, DNS records, and outputs defined, the last thing you’ll need to specify are the provider requirements for this module. You’ll specify that the dns-records module requires the digitalocean provider in a file called provider.tf. Create and open it for editing:

nano provider.tf

Add the following lines:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

When you’re done, save and close the file. Now that the digitalocean provider has been defined, the dns-records module is functionally complete.

Creating Different Environments

The current structure of the terraform-reusability project will look similar to this:

terraform-reusability/
├─ modules/
│  ├─ dns-records/
│  │  ├─ outputs.tf
│  │  ├─ provider.tf
│  │  ├─ records.tf
│  │  ├─ variables.tf
│  ├─ droplet-lb/
│  │  ├─ droplets.tf
│  │  ├─ lb.tf
│  │  ├─ outputs.tf
│  │  ├─ provider.tf
│  │  ├─ variables.tf
├─ provider.tf

So far, you have two modules in your project: the one you just created (dns-records) and the one you created as part of the prerequisites (droplet-lb).

To facilitate different environments, you’ll store the dev and prod environment config files under a directory called environments, which will reside in the root of the project. Both environments will call the same two modules, but with different parameter values. The advantage of this is when the modules change internally in the future, you’ll only need to update the values you are passing in.

First, navigate to the root of the project by running:

cd ../..

Then, create the dev and prod directories under environments at the same time:

mkdir -p environments/dev && mkdir environments/prod

The -p argument orders mkdir to create all directories in the given path.

Navigate to the dev directory, as you’ll first configure that environment:

cd environments/dev

You’ll store the code in a file named main.tf, so create it for editing:

nano main.tf

Add the following lines:

module "droplets" {
  source   = "../../modules/droplet-lb"

  droplet_count = 2
  group_name    = "dev"
}

module "dns" {
  source   = "../../modules/dns-records"

  domain_name   = "your_dev_domain"
  ipv4_address  = module.droplets.lb_ip
}

Here you call and configure the two modules, droplet-lb and dns-records, which will together result in the creation of two Droplets. They’re fronted by a Load Balancer, and the DNS records for the supplied domain are set up to point to that Load Balancer. Remember to replace your_dev_domain with your desired domain name for the dev environment, then save and close the file.

Next, you’ll configure the DigitalOcean provider and create a variable for it to be able to accept the personal access token you’ve created as part of the prerequisites. Open a new file, called provider.tf, for editing:

nano provider.tf

Add the following lines:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

variable "do_token" {}

provider "digitalocean" {
  token = var.do_token
}

In this code, you require the digitalocean provider to be available and to pass in the do_token variable to its instance. Save and close the file.

Initialize the configuration by running:

terraform init

You’ll receive the following output:

Output
Initializing modules...
- dns in ../../modules/dns-records
- droplets in ../../modules/droplet-lb

Initializing the backend...

Initializing provider plugins...
- Finding digitalocean/digitalocean versions matching "~> 2.0"...
- Installing digitalocean/digitalocean v2.10.1...
- Installed digitalocean/digitalocean v2.10.1 (signed by a HashiCorp partner, key ID F82037E524B9C0E8)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The configuration for the prod environment is similar. Navigate to its directory by running:

cd ../prod

Create and open main.tf for editing:

nano main.tf

Add the following lines:

module "droplets" {
  source   = "../../modules/droplet-lb"

  droplet_count = 5
  group_name    = "prod"
}

module "dns" {
  source   = "../../modules/dns-records"

  domain_name   = "your_prod_domain"
  ipv4_address  = module.droplets.lb_ip
}

The difference between this and your dev code is that there will be five Droplets deployed. Furthermore, the domain name, which you should replace with your prod domain name, will be different. Save and close the file when you’re done.

Then, copy over the provider configuration from dev:

cp ../dev/provider.tf .

Initialize this configuration as well:

terraform init

The output of this command will be the same as the previous time you ran it.

You can try planning the configuration to see what resources Terraform would create by running:

terraform plan -var "do_token=${DO_PAT}"

The output for prod will be the following:

Output
...
Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.dns.digitalocean_domain.domain will be created
  + resource "digitalocean_domain" "domain" {
      + id   = (known after apply)
      + name = "your_prod_domain"
      + urn  = (known after apply)
    }

  # module.dns.digitalocean_record.domain_A will be created
  + resource "digitalocean_record" "domain_A" {
      + domain = "your_prod_domain"
      + fqdn   = (known after apply)
      + id     = (known after apply)
      + name   = "@"
      + ttl    = (known after apply)
      + type   = "A"
      + value  = (known after apply)
    }

  # module.dns.digitalocean_record.domain_CNAME will be created
  + resource "digitalocean_record" "domain_CNAME" {
      + domain = "your_prod_domain"
      + fqdn   = (known after apply)
      + id     = (known after apply)
      + name   = "www"
      + ttl    = (known after apply)
      + type   = "CNAME"
      + value  = "@"
    }

  # module.droplets.digitalocean_droplet.droplets[0] will be created
  + resource "digitalocean_droplet" "droplets" {
...
      + name                 = "prod-0"
...
    }

  # module.droplets.digitalocean_droplet.droplets[1] will be created
  + resource "digitalocean_droplet" "droplets" {
...
      + name                 = "prod-1"
...
    }

  # module.droplets.digitalocean_droplet.droplets[2] will be created
  + resource "digitalocean_droplet" "droplets" {
...
      + name                 = "prod-2"
...
    }

  # module.droplets.digitalocean_droplet.droplets[3] will be created
  + resource "digitalocean_droplet" "droplets" {
...
      + name                 = "prod-3"
...
    }

  # module.droplets.digitalocean_droplet.droplets[4] will be created
  + resource "digitalocean_droplet" "droplets" {
...
      + name                 = "prod-4"
...
    }

  # module.droplets.digitalocean_loadbalancer.www-lb will be created
  + resource "digitalocean_loadbalancer" "www-lb" {
...
      + name                     = "lb-prod"
...

Plan: 9 to add, 0 to change, 0 to destroy.
...

This would deploy five Droplets with a Load Balancer. It would also create the prod domain you specified with the two DNS records pointing to the Load Balancer. You can try planning the configuration for the dev environment as well—you’ll note that two Droplets would be planned for deployment.

terraform apply -var "do_token=${DO_PAT}"

terraform destroy -var "do_token=${DO_PAT}"

The following demonstrates how you have structured this project:

terraform-reusability/
├─ environments/
│  ├─ dev/
│  │  ├─ main.tf
│  │  ├─ provider.tf
│  ├─ prod/
│  │  ├─ main.tf
│  │  ├─ provider.tf
├─ modules/
│  ├─ dns-records/
│  │  ├─ outputs.tf
│  │  ├─ provider.tf
│  │  ├─ records.tf
│  │  ├─ variables.tf
│  ├─ droplet-lb/
│  │  ├─ droplets.tf
│  │  ├─ lb.tf
│  │  ├─ outputs.tf
│  │  ├─ provider.tf
│  │  ├─ variables.tf
├─ provider.tf

The addition is the environments directory, which holds the code for the dev and prod environments.

The benefit of this approach is that further changes to modules automatically propagate to all areas of your project. Barring any possible customizations to module inputs, this approach is not repetitive and promotes reusability as much as possible, even across deployment environments. Overall, this reduces clutter and allows you to trace the modifications using a version-control system.

In the final two sections of this tutorial, you’ll review the depends_on meta argument and the templatefile function.

Declaring Dependencies to Build Infrastructure in Order

While planning actions, Terraform automatically tries to identify existing dependencies and builds them into its dependency graph. The main dependencies it can detect are clear references; for example, when an output value of a module is passed to a parameter on another resource. In this scenario, the module must first complete its deployment to provide the output value.

The dependencies that Terraform can’t detect are hidden—they have side effects and mutual references not inferable from the code. An example of this is when an object depends not on the existence, but on the behavior of another one, and does not access its attributes from code. To overcome this, you can use depends_on to manually specify the dependencies in an explicit way. Since Terraform 0.13, you can also use depends_on on modules to force the listed resources to be fully deployed before deploying the module itself. It’s possible to use the depends_on meta argument with every resource type. depends_on will also accept a list of other resources on which its specified resource depends.

depends_on accepts a list of references to other resources. Its syntax looks like this:

resource "resource_type" "res" {
  depends_on = [...] # List of resources

  # Parameters...
}

Remember that you should only use depends_on as a last-resort option. If used, it should be kept well documented, because the behavior that the resources depend on may not be immediately obvious.

In the previous step of this tutorial, you haven’t specified any explicit dependencies using depends_on, because the resources you’ve created have no side effects not inferable from the code. Terraform is able to detect the references made from the code you’ve written, and will schedule the resources for deployment accordingly.

Using Templates for Customization

In Terraform, templating is substituting results of expressions in appropriate places, such as when setting attribute values on resources or constructing strings. You’ve used it in the previous steps and the tutorial prerequisites to dynamically generate Droplet names and other parameter values.

When substituting values in strings, the values are specified and surrounded by ${}. Template substitution is often used in loops to facilitate customization of the created resources. It also allows for module customization by substituting inputs in resource attributes.

Terraform offers the templatefile function, which accepts two arguments: the file from the disk to read and a map of variables paired with their values. The value it returns is the contents of the file rendered with the expression substituted—just as Terraform would normally do when planning or applying the project. Because functions are not part of the dependency graph, the file cannot be dynamically generated from another part of the project.

Imagine that the contents of the template file called droplets.tmpl is as follows:

%{ for address in addresses ~}
${address}:80
%{ endfor ~}

Longer declarations must be surrounded with %{}, as is the case with the for and endfor declarations, which signify the start and end of the for loop respectively. The contents and type of the droplets variable are not known until the function is called and actual values provided, like so:

templatefile("${path.module}/droplets.tmpl", { addresses = ["192.168.0.1", "192.168.1.1"] })

This templatefile call will return the following value:

Output
192.168.0.1:80
192.168.1.1:80

This function has its use cases, but they are uncommon. For example, you could use it when part of the configuration must exist in a proprietary format, but is dependent on the rest of the values and must be generated dynamically. In the majority of cases, it’s better to specify all configuration parameters directly in Terraform code, where possible.

Conclusion

In this article, you’ve maximized code reuse in an example Terraform project. The main way is to package often-used features and configurations as a customizable module and use it whenever needed. By doing so, you do not duplicate the underlying code (which can be error prone) and enable faster turnaround times, since modifying the module is almost all you need to do to introduce changes.

You’re not limited to your own modules. As you’ve seen, Terraform Registry provides third-party modules and providers that you can incorporate in your project.

This tutorial is part of the How To Manage Infrastructure with Terraform series. The series covers a number of Terraform topics, from installing Terraform for the first time to managing complex projects.