Terraform module versions do not get pinned by lock file

2023-Apr-25 • by David Norton

The other day, a client was experiencing a curious problem. Their Terraform project was no longer initializing in GitHub Actions. They were using a lockfile (.terraform.lock.hcl) to prevent against unexpected version updates. We've previously discussed the dangers of unpinned versions on this website.

In this particular scenario, it was throwing the error message:

│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider
│ hashicorp/aws: locked provider registry.terraform.io/hashicorp/aws 4.60.0
│ does not match configured version constraint >= 4.63.0, ~> 4.63.0; must use
│ terraform init -upgrade to allow selection of new versions

As it turned out, an unpinned module version was causing a newer module to be pulled upon terraform init. That new version of the module required the AWS provider 4.63.0. However, our AWS provider was pinned to 4.60.0.

A simple terraform init -upgrade updated our lockfile, and we were able to commit the change on our main branch. But the experience was unsettling - wasn't the lockfile supposed to prevent against this?

What is the Terraform lockfile?

The Terraform dependency lock file allows you the best of both worlds, as described in our previous blog:

1. Easy dependency version updates by running terraform init -upgrade to pull the latest allowable versions within a range

   1. Providers are specified as such:

      aws = {
        source  = "hashicorp/aws"
        version = "~> 4.0"
      }
      

   2. Our module was defined like this:

      module "xyz_lambda" {
        source  = "terraform-aws-modules/lambda/aws"
        version = "~> 4.0"
      
        # ...
      }
      

2. Idempotent (repeatable) builds using the lockfile generated by the above command

   1. The .terraform.lock.hcl file should be committed to source control.

However, it turns out that this file only locks provider versions. In fact, the docs are clear on this (but sometimes we don't all read the docs, do we?):

At present, the dependency lock file tracks only provider dependencies. Terraform does not remember version selections for remote modules, and so Terraform will always select the newest available module version that meets the specified version constraints. You can use an exact version constraint to ensure that Terraform will always select the same module version.

So there you have it - you should specify a specific version of the module if you want idempotent builds:

module "lambda_layer_gdcm" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "4.15.0"
  # ...
}

What about transitive dependencies?

As a commenter on the GitHub issue described, you can still run into issues:

The problem with this is that if you use a module that itself uses a module with a version [range] constraint instead of an exact version, there is no way to lock the version. So you end up needing to have exact versions in all modules as well. And that in turn means if you make a minor bug fix, you have to propagate the version increase up through multiple dependencies.

Because of this, you may want to consider checking the modules that you use to ensure they have fixed versions for their own submodules. Alternatively, follow the GitHub issue and ask HashiCorp to fix this aspect of Terraform.

Conclusion

For idempotent Terraform plans, when specifying Terraform module versions, you should choose a specific version rather than a range.

Shameless plug: If you need help with public cloud, declarative infrastructure, or Terraform specifically -- Platformers can probably help. Please reach out today!