5
Managing Environments
The development of every piece of software requires multiple environments. Generally, we use at least three environments – one for development, one for testing, and one for production. However, larger, more complex projects might require additional environments such as training and staging. Supporting multiple environments in a traditional on-premises setting can be costly to create and maintain. The beauty of the cloud is that we have virtually unlimited resources at our disposal. With efficient IaC, we can deploy environments quickly and remove them when they are no longer needed. As we only pay for what we use, we can have as many environments as needed but only pay for them when we actually use them. In this chapter, we will see how to manage multiple environments in Terraform.
This chapter describes the two main methods of managing multiple environments in Terraform – workspaces and directory structure. In particular, we will cover the following topics:
- Google resource hierarchy
- Using workspaces to manage environments
- Using a directory structure to manage environments
- Using remote states
- Using template files
Technical requirements
To follow the code in this project, we require two Google Cloud projects, one to act as the development project and the other as the production project. We can use our existing project as the development project and create a second project for the production environment, or we can create two new ones. If we create a new project, please remember to create the bucket for the state file. If we use a service account (either with a keyfile or with service account impersonation), that service account must have permission to create resources in both projects. Using Cloud Shell for this chapter is easiest as all the permission are set up if you are the owner of both projects.
The code for this chapter can be found here: https://github.com/PacktPublishing/Terraform-for-Google-Cloud-Essential-Guide/tree/main/chap05.
Google resource hierarchy
Before we discuss how to manage environments in Terraform, let’s briefly review how to set up environments in Google Cloud. Within Google Cloud, a project is a basic container to manage a cloud deployment, including resources, permissions, and billing. When you start with Google Cloud, you generally have one project, then add a second and third project. These projects are entirely independent; that is, they share no IAM permission or resource policy.
As the number of our projects increases, managing these projects, for example, billing and IAM permissions, can become unwieldy. Thus, as we grow our footprint in Google Cloud, it is recommended to set up a resource hierarchy (you can read about this more here: https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy) with an organization as the root node. This resource hierarchy allows us to organize projects into folders and subfolders. We can then manage permissions on a folder and subfolder level in addition to the project level. For example, we can set IAM permissions and policies at the folder level, which then are inherited down to the project level.
A typical resource hierarchy is shown in Figure 5.1. Organizations adopt different folder strategies, but each environment generally resides in its own Google Cloud project. Using separate projects also enables strict separation of namespaces. We can have the same server names in both development and production as they reside in separate projects.
Figure 5.1 – A typical Google Cloud resource hierarchy
For example, in the hierarchy depicted in Figure 5.1, we can provision independent resources such as VPCs and virtual machines in both projects – Dev, and Prod. These resources can even share the same name. We can then allow the development team access to the dev environment in the Dev project and the production team access to the Prod project by setting the appropriate IAM permission on the respective project. However, we can also assign a super user the required IAM permission at the Team A folder level, to inherit the permission for both projects. Thus, resource hierarchies provide an ideal way to separate different environments.
Modern software engineering practices dictate that the different environments should be identical or nearly identical except for minor configuration differences. This practice is called dev/prod parity (https://12factor.net/dev-prod-parity). This is to ensure that the application code we developed in a development environment also works in the production environment. IaC enables us to not only create these environments consistently but also provision them quickly when needed and then destroy them when they are not needed in order to save cost.
Within Terraform, there are two established methods to support environments – workspaces and directory structure.
Using workspaces to manage environments
Note
The code for this section is under the chap05/workspaces directory in the GitHub repo of this book.
Workspaces allow us to have multiple independent state files for a single configuration. Thus, we can have one state file for our development project and a second one for production. By itself, that might appear to be very limited. However, combined with the effective use of variable definitions files (.tfvars) and Google projects, this can be an effective method to manage multiple environments.
Let’s see how we can accomplish this. Using the sample code at chap05/workspaces, run terraform init and terraform apply like normal. Then, have a look at the state file using terraform state list:
$ terraform init $ terraform apply $ terraform state list
We can see the three servers and their static IP addresses in the state file.
Next, create a new workspace named prod using the terraform workspace new prod command. The terraform workspace list command shows all the current workspaces and confirms that we are now in the prod workspace.
If we run terraform state list again, Terraform returns an empty list. Terraform created a new state file, and that state is empty:
$ terraform workspace new prod $ terraform workspace list $ terraform state list
Now, run terraform apply -var project_id=<PRODJECT_ID> where <PROJECT_ID> is the ID of our second project, our production project. Specifying project_id on the command line overrides the value provided in terraform.tfvars. We can see that it creates identical resources in our production and development projects.
Note
If you run these commands outside of Cloud Shell, you need to set the proper IAM permissions of your Terraform service account in the second project.
The following is a summary of the commands:
$ terraform apply -var "project_id=<PROJECT_ID>" $ terraform state list
Now, using the web console, investigate our two projects. We can see that we have created two identical environments, one in each project. That is the power of Terraform combined with the Google Cloud resource hierarchy.
Still in the web console, go to Cloud Storage to view the backend state, which is in our development project. We can see two .tfstate files, default.tfstate and prod.tfstate. Thus, Terraform now manages two independent versions of the state file. You can switch between the two workspaces using terraform workspace select and investigate the different state files.
In general, passing variables via the command line is not very efficient. Thus, it is better to create a second variable definition file (.tfvars) and pass that on the command line. That is, we edit the prod.tfvars file to add the project ID of our production project and run terraform apply -var-file prod.tfvars .
Workspaces combined with Google projects can be a powerful tool to manage nearly identical deployments in different environments, but this approach has some limitations. Hence, many organizations use a directory structure to manage different environments.
Using a directory structure to manage environments
Note
The code for this section is under the chap05/directory-structure directory in the GitHub repo of this book.
We saw that using workspaces with Google Cloud projects is an easy way to manage nearly identical environments. All we have to do is to create an additional workspace and supply different values in our variable definitions file. However, one of the major limitations of this approach is that the environments must be nearly identical.
Using our example, let’s say we want to have two small servers in the development environment, but in the production environment, we want to have two medium and one large server. Provisioning two servers of different sizes is easy, as we can specify different server sizes in the variable definitions file. However, having a third server of a different configuration only in production is more challenging. We could define a combination of conditional expressions and a count meta-argument, as follows:
count = (terraform.workspace == "prod") ? 1 : 0
However, that can get overly complex very quickly. A better approach is to have different subdirectories for each environment and use modules to keep our code DRY. So, if we want to have two environments, our directory structure looks like the following:
. ├── dev │ ├── backend.tf │ ├── main.tf │ ├── outputs.tf │ ├── provider.tf │ ├── terraform.tfvars │ └── variables.tf ├── modules │ └── server │ ├── main.tf │ ├── outputs.tf │ ├── startup.sh │ └── variables.tf └── prod ├── backend.tf ├── main.tf ├── outputs.tf ├── provider.tf ├── terraform.tfvars └── variables.tf
We have three subdirectories, dev, prod, and modules. Both dev and prod contain the usual files, including their own backend and variable definitions files. The modules subdirectory contains the configuration files for the server module like before.
So, looking at the two main.tf files, we can see how we can define two small servers for dev and three servers of different sizes for prod:
chap05/directory-structure/dev/main.tf
module "server1" {
source = "../modules/server"
name = "${var.server_name}-1"
machine_size = "small"
environment = var.environment
}
module "server2" {
source = "../modules/server"
name = "${var.server_name}-2"
machine_size = "small"
environment = var.environment
}In prod/main.tf we change some variable settings and add one additional module declaration for the third server:
chap05/directory-structure/prod/main.tf
module "server1" {
source = "../modules/server"
name = "${var.server_name}-1"
machine_size = "medium"
environment = var.environment
}
module "server2" {
source = "../modules/server"
name = "${var.server_name}-2"
machine_size = "medium"
environment = var.environment
}
module "server3" {
source = "../modules/server"
name = "${var.server_name}-3"
machine_size = "large"
environment = var.environment
}With these files in place, we can now run Terraform in each subdirectory specifying the appropriate project ID as an argument:
$ cd dev $ terraform init $ terraform apply -var project_id=[DEV-PROJECT-ID] $ cd ../prod $ terraform init $ terraform apply -var project_id=[PROD-PROJECT-ID]
This structure allows us to have different configurations between the environments yet keep our code DRY. This approach requires the effective use of modules, whether they are stored locally or remotely in Cloud Storage or a repository. When using separate directories, we often need to share state data between the configurations in the different subdirectories. For this, Terraform provides the concept of remote states.
Using remote states
Note
The code for this section is under the chap05/remote-state directory in the GitHub repo of this book.
Using the directory structure approach provides added flexibility. It is a good approach to decompose complex configurations into smaller, more manageable parts, often called layers, when creating complex architectures. Each layer is a logical grouping of resources. For example, let’s say we want to provision a database and a set of servers that connect to the database. Furthermore, we have two teams – one responsible for the database and the second managing the servers. In addition, while the database is relatively stable and should be running at all times, we expect the servers to be re-created many times, and we might want to remove the servers over the weekend to save cost.
We can accomplish this goal by creating two layers and placing all configuration files related to the layer in separate subdirectories. That is, we create two subdirectories, cloud-sql and compute-instance. Each subdirectory has its own configuration files, including the backend and the variable definitions file. Therefore, the file structure looks as follows:
. ├── cloud-sql │ ├── backend.tf │ ├── main.tf │ ├── outputs.tf │ ├── provider.tf │ ├── terraform.tfvars │ └── variables.tf ├── compute-instance │ ├── backend.tf │ ├── main.tf │ ├── provider.tf │ ├── startup.tftpl │ ├── terraform.tfvars │ └── variables.tf
So, let’s first provision the CloudSQL database instance. One of the features of CloudSQL (or, as you might call it – a quirk) is that we cannot reuse the name for up to 1 week after the instance is deleted (https://cloud.google.com/sql/docs/mysql/create-instance#create-2nd-gen). Thus, it is common practice to attach a short, unique string to the database name. For this, we use the random_string resource in the Terraform random provider (https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string), which is a convenient way to create a random string:
chap05/cloud-sql/main.tf
resource "random_string" "this" {
length = 4
special = false
upper = false
}
resource "google_sql_database_instance" "main" {
name = "main-instance-${random_string.this.result}"
database_version = "POSTGRES_11"
region = var.region
settings {
tier = "db-f1-micro"
}
}Now, we run Terraform independently in each subdirectory, starting with cloud-sql, but this creates a problem. Since each subdirectory has its own state file, how do we share information from the CloudSQL instance? For example, in order to connect to the database, the compute instance needs to know the connection name of the database that Terraform created. But since the CloudSQL database was configured in a different subdirectory with its own state, the configuration in the compute-instance directory does not have access to any of the attributes of google_sql_database_instance.
Now, we could use the google_sql_database_instance data source (https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/sql_database_instance) using the name of the database, but there is a better way.
The terraform_remote_state data source (https://developer.hashicorp.com/terraform/language/state/remote-state-data) enables us to use information from a separate Terraform configuration. That is, in our example, we can use the state information from the google_sql_database_instance resource that is defined in the cloud-sql directory. As previously, we need to define output values to expose the information we want to share. Since we want to share the connection_name value of the database, we expose that value in the outputs.tf file:
chap05/remote-state/cloud-sql/outputs.tf
output "connection_name" {
value = google_sql_database_instance.main.connection_name
}To use that information, we use the terraform_remote_state data source in the compute-instance directory to retrieve that information. We can then use the connection name using the data.terraform_remote_state.cloud_sql.outputs.connection_name expression, as shown in the data.tf file that follows:
chap05/ remote-state/compute-instance/data.tf
data "terraform_remote_state" "cloud_sql" {
backend = "gcs"
config = {
bucket = "<PROJECT_ID>-tf-state"
prefix = "chap05/remote-state/cloud-sql"
}
}
# For Illustration only
output "connection_name" {
value = data.terraform_remote_state.cloud_sql.outputs.connection_name
}Now, let’s say we want to use the connection name in our startup file. For example, we want to set it as an environment variable that our application code can use to create a connection to the database. How can we pass that information to our startup script?
Using template files
Note
The code for this section is under the chap05/remote-state directory in the GitHub repo of this book.
Terraform provides a built-in function called templatefile (read more about it here: https://developer.hashicorp.com/terraform/language/functions/templatefile).
This function takes a file that uses Terraform expressions and evaluates them. We can use variables, resource attributes, and other expressions in the file, and Terraform evaluates those expressions and replaces them with the values in the files. Let’s see this in practice. First, we define a template file. By convention, template files use the *.tftpl extension:
chap05/compute-instance/startup.tftpl
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Hello World!</p>
<p>The CloudSQL connection name is: ${connection_name}</body></html>During runtime, Terraform evaluates the data.terraform_remote_state.cloud_sql.outputs.connection_name expression and replaces it with the value.
Managing Terraform at scale
Before we conclude this chapter, let us look at another aspect of state files. So far, our Terraform files have been manageable as we deployed only a few resources simultaneously. However, even a medium-complexity architecture requires dozens or hundreds of interdependent resources. Using modules, we can reduce this complexity of the Terraform code, but we will still provision many cloud resources with many dependencies. Furthermore, as our use of Terraform grows, we will have teams of several members developing Terraform code simultaneously. Then, managing all resources in a single state file becomes challenging. For example, the team member responsible for the networking might want to change just when another team member is running Terraform to add a virtual machine and hence is prevented from running Terraform simultaneously. Or, the database team wants to use a new feature in the latest provider version that the networking team hasn’t been able to test yet.
Thus, it makes sense to break up the state files into separate parts that can be managed independently. When using a directory structure, each subdirectory has its own state file. Having multiple state files brings several advantages with it. First, we can use state files to separate ownership. For example, by dividing all networking, and database-related resources into separate directories, each can have its independent ownership and state files. We can then assign the responsibility of those resources to the networking and database teams, respectively.
Second, we limit the blast radius. With smaller plans, we are less likely to miss destructive changes than if everything is in the same directory and a single state file.
Furthermore, providing separation allows for more granular and safer upgrade paths. As we mentioned, Terraform is a very dynamic environment, with both Terraform and the providers making frequent updates. By separating the state files into independent units, each part can manage upgrades on its own.
Now, managing Terraform at scale for small and large teams is one of the reasons why HashiCorp introduced Terraform Cloud (https://cloud.hashicorp.com/products/terraform). Terraform Cloud is a managed service that offers a wide range of features to make it easier and more secure to manage Terraform at scale.
Summary
In this chapter, we introduced the two common methods to support multiple environments in Terraform, workspaces and directory structure. Workspaces create independent state files, which we can use to manage multiple environments – while sufficient for simple deployments, directories are a more flexible way to separate environments in Google Cloud and are generally used to manage environments.
Now that we have covered the fundamentals of Terraform using simple examples, let us build more complex deployments using the techniques you have learned so far. We begin by building a traditional three-tier architecture. Then, we will use a modern serverless architecture using Cloud Run and Redis, before using the public Terraform repository to quickly provision a GKE cluster.