Remote state and locking

Why

From the Terraform basics lesson: state is the file where Terraform remembers what it has already created. That's how it knows what to change and what to leave alone. By default, state lives locally, in a file called terraform.tfstate next to the code. That works for one person and breaks for a team.

Local state creates several problems at once:

• Divergence. Everyone has their own state file. Two people apply changes, and their pictures of reality diverge: one person's Terraform doesn't know about resources the other one created.
• Secrets in git. Values in state are stored in plain text (passwords, keys), even if a variable is marked sensitive: that flag only hides the value in plan and apply output. Committing state to the repository means a leak, and not committing it means other people don't have it.
• Concurrent apply. Two people run apply at the same time and overwrite each other's work, leaving the infrastructure in an unclear state.

So state gets moved out of the local folder into shared storage.

Concepts

Backend is where Terraform stores state. By default it's local, a file on disk.

Remote backend stores state in a shared place everyone connects to. Most often it's an S3-compatible bucket, HCP Terraform (the name Terraform Cloud has had since April 2024) or a cloud equivalent. There's also the pg backend, which stores state in PostgreSQL: this lesson's labs are built on it, because you can run it on a laptop.

Lock keeps two processes from changing the same state at the same time. Before an operation that might write state, Terraform takes a lock, and releases it afterward.

How it works

Remote backend

The backend is set with a backend block inside terraform:

terraform {
  backend "s3" {
    bucket       = "company-tfstate"
    key          = "prod/terraform.tfstate"
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true
  }
}

Now state is one shared copy, sitting in a bucket (encrypt = true turns on encryption on the S3 side), not in the repository and not on someone's laptop. Every apply reads and writes the shared state, so everyone sees the same picture of reality.

The example above is for AWS. S3-compatible storage (Yandex Object Storage, MinIO) also needs its own API address in the endpoints = { s3 = "..." } parameter, and usually several skip_* parameters that turn off checks specific to AWS. The storage's documentation says which ones.

Passwords and access keys for the backend don't go in the backend block: the block lives in git. They're passed through environment variables or terraform init -backend-config=.... For example, the pg backend takes the connection string from the PG_CONN_STR variable.

How to migrate state

The backend block only says where to store state. The actual migration is done by terraform init: it notices the backend has changed and asks whether to copy the existing state to the new location:

Do you want to copy existing state to the new backend?
  ...
  Enter a value:

Answering yes migrates the state. The terraform init -migrate-state flag explicitly asks for the same thing. After the migration, the local terraform.tfstate becomes empty, and terraform.tfstate.backup remains next to it with the old copy, secrets included. Delete it by hand.

The check that everything worked: terraform plan with no changes. If plan offers to create everything again, Terraform doesn't see state in the new location.

State locking

Shared state solves divergence, but not concurrent runs: two people can still run apply at the same time. That's why the backend supports locking. The second run gets rejected:

Error: Error acquiring the state lock

Error message: ...
Lock Info:
  ID:        db0530fe-b7c1-6207-83ab-055379db2370
  Operation: OperationTypeApply
  Who:       anna@laptop
  ...

The text in Error message varies by backend.

That way only one person changes the infrastructure at any moment. They finish, the lock is released, the next one applies. It's the same idea as a lock in a database: keep two writers from overwriting each other. If waiting it out beats failing, there's the -lock-timeout=60s flag: Terraform will try to acquire the lock for that long.

How the lock is actually stored depends on the backend. In S3 it's a separate lock file next to state. In pg it's a PostgreSQL advisory lock that lives as long as the session that took it stays open.

Locking in S3

The S3 backend's locking is not enabled by default (use_lockfile defaults to false). An example without it looks like it works, apply goes through, but there's no protection against a parallel run. You find out about it on the exact day two people hit apply at the same time.

In the past, locking meant spinning up a separate DynamoDB table and pointing to it in dynamodb_table. An extra AWS service for one mutex, and S3-compatible storage simply doesn't have DynamoDB.

Starting with Terraform 1.10 and OpenTofu 1.10, locking lives right in the bucket: a lock file sits next to state. It's turned on with use_lockfile = true from the example above, nothing else needs to be deployed. In Terraform 1.11 this approach is declared the main one, and the DynamoDB parameters are marked deprecated.

Stuck lock

A process that took a lock can crash without releasing it: a dropped network connection mid-apply, a killed CI job. Then every following run gets the same error. For that case there's terraform force-unlock <ID>, where ID comes from Lock Info.

You can release someone else's lock only when you're sure its owner is dead. If it's alive and writes to state after you, state will get corrupted. The -lock=false flag turns off locking entirely, and Terraform itself warns that you shouldn't do that.

Common mistakes

• Backend without locking. S3 without use_lockfile = true (or without dynamodb_table in older versions) works, but two people can apply at once.
• State in git. terraform.tfstate and terraform.tfstate.backup go into .gitignore. sensitive doesn't encrypt the value in state.
• Secret in the backend block. A connection string with a password in backend.tf ends up in git and in .terraform/terraform.tfstate on disk. Use environment variables or -backend-config.
• Answering no to the state-copying question. Terraform starts with an empty state in the new backend and tries to create everything again on the next apply.
• force-unlock out of habit. First find out who's holding the lock and whether they're alive.

What you'll need for the tasks

• Docker 20.10+ and Terraform 1.4+ (or OpenTofu): Terraform runs on your machine, the lab environment brings up PostgreSQL as the shared backend.
• The pg backend: the conn_str and schema_name parameters, documentation https://developer.hashicorp.com/terraform/language/backend/pg.
• terraform init when the backend changes, terraform plan to check state is in place, terraform -chdir=<folder> to run from the lab folder.
• The terraform_data resource: built into Terraform, creates nothing outside it, but lives in state. No need to download providers from the internet.
• For the mini-project: modules, variables, and outputs from lesson 4, and the terraform_remote_state data source, which reads another configuration's outputs from its state: https://developer.hashicorp.com/terraform/language/state/remote-state-data.
• For the locking lab: psql inside the lab container and PostgreSQL's system views (https://www.postgresql.org/docs/16/view-pg-locks.html, https://www.postgresql.org/docs/16/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW).

What to remember

Local state breaks teamwork: everyone has their own copy, it holds secrets, and two people can apply at once. So state gets moved into a remote backend (an S3-compatible bucket, HCP Terraform, PostgreSQL), where it's one shared copy. terraform init migrates the state, and the backup copy on disk gets deleted by hand.

State locking keeps two apply runs from happening at the same time: the first one takes the lock, the second one gets rejected or waits with -lock-timeout. In the S3 backend, locking is turned on by hand: use_lockfile = true. Before Terraform 1.10 and OpenTofu 1.10, that required a separate DynamoDB table.

This closes out the iac-basics module: the idea of IaC, Terraform, GitOps, modules, and remote state. The mini-project in lab 3 brings it all together.