Planning Terramate stacks in Atlantis without generated code

This is the second of two posts on running Terramate stacks through Atlantis with terramate-atlantis-config. If you’re just joining, the first post introduces the tool and the terms used here. It ended with Atlantis generating atlantis.yaml on every pull request, while Terramate’s own generated Terraform stayed committed. This post removes that too, follows one change to a global through both tools to show what breaks, and then works through the fix, the ordering of applies, and the corner cases that sit between the two tools.

Why commit generated code at all?

It’s worth asking the question honestly before answering it, because committing generated code is not a mistake. Terramate’s generate_hcl blocks turn globals into Terraform, and committing the result means a reviewer sees exactly the Terraform that will run. It also means the repository plans as-is, with no generation step anywhere in the pipeline.

The cost appears as the repository grows. Change one global in an account directory and the generated code changes in every stack below it: one meaningful line in review, and a page of mechanical ones. In practice it is easy to end up skimming the generated files, at which point the noise is still being paid for but the review is no longer happening.

There is a less obvious reason the generated files matter, though, and it is the one this post is really about. In a repository that Atlantis runs, those files are what tell Atlantis a stack was affected. Take them away and the signal goes with them.

The global change nobody planned

The demo has two AWS accounts and one stack outside them, shared/settings:

config/settings.json
modules/shared/main.tf
stacks/
├── aws/
│   ├── aws.tm.hcl
│   ├── acme-development/
│   │   ├── account.tm.hcl
│   │   └── eu-west-1/
│   │       ├── region.tm.hcl
│   │       ├── app/
│   │       └── network/
│   └── acme-production/
│       └── ... the same shape ...
├── imports/
│   ├── aws.tm.hcl
│   └── common.tm.hcl
└── shared/
    ├── shared.tm.hcl
    └── settings/

Each account directory sets an account_name global, which the stacks read through generated Terraform. The first pull request changes that global in the development account, and that is its only file:

$ git diff --stat main
 stacks/aws/acme-development/account.tm.hcl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Two stacks are affected, the development app and network, since both read account_name. With committed generated code, their _terramate_generated_aws.tf files would have changed too, and the tool’s default when_modified patterns (*.tf, *.tf.json and .terraform.lock.hcl, all relative to the stack’s own directory) would have matched them. With no generated code in Git, the pull request touches one .tm.hcl file two directories above the stacks, and no project’s patterns match it. Atlantis plans nothing.

The natural next question is whether Terramate’s own change detection could help. The tool has a --changed flag that keeps only the stacks Terramate considers changed, so we could generate projects only for those. On the same branch:

$ terramate list --changed
$ terramate-atlantis-config generate --changed --output - --no-header
version: 3
automerge: false
parallel_plan: true
parallel_apply: true
projects: []

terramate list --changed prints nothing. Its change detection maps a changed file up to the stack that contains it, and follows a stack’s modules and watch files, but it does not map a changed parent config file down to the stacks that inherit from it:

Atlantis and Terramate share the same blind spot here: both decide what a change affects by where the changed file sits, not by what reads it.

Telling Atlantis what each stack reads

If neither tool can infer the relationship at plan time, it has to be written down where Atlantis will look, and the only place Atlantis looks is each project’s when_modified. So the question becomes: what, besides its own directory, does a stack read? For Terramate there are four answers, and the demo has one of each:

when_modified_from_config adds each stack’s inputs of all four kinds to its project. In the demo it is switched on in the tool’s config file, alongside the ordering settings we’ll return to later:

output: atlantis.yaml

# Add the Terramate files each stack depends on to its
# when_modified, so a change outside the stack still plans it.
when_modified_from_config: true

# Turn each stack's `after` into Atlantis ordering.
depends_on: true
execution_order_groups: true

Here is what it produces for the development network stack, which sits five directories deep:

  - name: aws-acme-development-eu-west-1-network
    dir: stacks/aws/acme-development/eu-west-1/network
    autoplan:
      enabled: true
      when_modified:
        - '*.tf'
        - '*.tf.json'
        - .terraform.lock.hcl
        - '*.tm.hcl'
        - '*.tm'
        - ../*.tm.hcl
        - ../*.tm
        - ../../*.tm.hcl
        - ../../*.tm
        - ../../../*.tm.hcl
        - ../../../*.tm
        - ../../../../*.tm.hcl
        - ../../../../*.tm
        - ../../../../../*.tm.hcl
        - ../../../../../*.tm
        - ../../../../imports/aws.tm.hcl
        - ../../../../imports/common.tm.hcl

The first three entries are the defaults. Then come two globs per directory, from the stack up to the repository root, and finally the two files its config imports. Relative paths are hard to read in a list like this, so here is the directory each .tm.hcl glob resolves to, counting up from the stack, next to the file it would match today:

The third row is the one that matters for the first pull request: it matches the changed account.tm.hcl. The production stacks’ equivalent entry resolves to stacks/aws/acme-production/, which doesn’t, and neither does any entry of shared/settings. That is exactly the selection we wanted, and it is what Atlantis did:

Ran Plan for 2 projects:

1. project: `aws-acme-development-eu-west-1-network` dir: `stacks/aws/acme-development/eu-west-1/network` workspace: `default`
1. project: `aws-acme-development-eu-west-1-app` dir: `stacks/aws/acme-development/eu-west-1/app` workspace: `default`

Notice the design choice: the tool still generates every project, and leaves the selection to Atlantis’ ordinary file matching. Nothing about which stacks to plan is decided when the file is generated, which matters when a pull request keeps changing after it is opened.

The other three kinds of input behave the same way. Each of the demo’s next three pull requests changes a single file outside every stack:

The imported file is the subtle one. stacks/aws/aws.tm.hcl imports stacks/imports/aws.tm.hcl, so it shapes every stack under stacks/aws/, but shared/shared.tm.hcl imports only common.tm.hcl. The tool lists an imported file only in the projects whose config actually imports it, so shared/settings stays unplanned. The watch case is the mirror image: only shared/settings declares it, so only shared/settings gets the entry:

stack {
  name  = "settings"
  watch = ["/config/settings.json"]
  id    = "4d15ed0d-649f-44ad-ae3e-6c9b398ae82e"
}

Globs for config, exact paths for the rest

You might wonder why config files appear as globs while imports appear as exact paths. The answer comes from what Terramate itself reads. It reads every .tm and .tm.hcl file in a stack’s directory and its parents, so a pair of globs per directory matches exactly that set, including a file a pull request adds or deletes, and the generated file doesn’t change when someone adds a config file. That is why ../../../../*.tm.hcl is listed even though stacks/ holds no config today: the day someone adds one, it is already covered. An import, on the other hand, usually names one file, so it is listed as one file; a glob there would plan stacks for changes to files they never read.

The module entries are the exception in the other direction. The two app stacks call the shared module, and their lists end with:

        - ../../../../../modules/shared/**

That matches any file in the module directory, not only .tf files. It mirrors Terramate’s change detection, and it covers templates or policy files that a module reads with file(). Modules called from a module are followed too, so a change two modules deep still plans the stacks at the top. Remote module sources, from a registry or Git, are ignored; a version bump to one of those is an edit to a .tf file in the stack or to the config that generates it, both of which are already covered.

Two hooks, in order

Without generated code in Git, Atlantis has to generate it before it can plan anything, so the server-side config now runs two hooks:

repos:
  - id: /.*/
    pre_workflow_hooks:
      - run: terramate generate
        description: terramate generate
      - run: terramate-atlantis-config generate --output atlantis.yaml
        description: terramate-atlantis-config

Each pull request event now passes through both before Atlantis reads its config:

The order is not arbitrary. The tool finds module calls by reading the .tf files that exist when it runs, and in this demo the module call is in a hand-written main.tf, but in many Terramate repositories it is in generated code.

Running the first hook also means the Atlantis image now needs the terramate CLI, which it didn’t before, at the same version the repository was written for.

The repository’s .gitignore then excludes both kinds of generated file, _terramate_generated_*.tf and atlantis.yaml, so the sixth pull request can add a whole new stack with just two files, its stack.tm.hcl and main.tf, and still be planned.

Applying stacks in order

The config file above also switched on two ordering settings, and they solve a separate problem. In the demo, each account’s app stack declares that it runs after network:

stack {
  name  = "app"
  after = ["../network"]
  id    = "bd6801db-c87e-48f5-beb9-79580387fc14"
}

terramate run honors that ordering, but Atlantis knows nothing about it. depends_on turns it into Atlantis’ own depends_on, which lists project names, and execution_order_groups assigns each project the length of its longest chain of dependencies:

  - name: aws-acme-development-eu-west-1-app
    dir: stacks/aws/acme-development/eu-west-1/app
    execution_order_group: 1
    depends_on:
      - aws-acme-development-eu-west-1-network

A project with no dependencies is in group 0, which the tool expresses by leaving the key out. Across the whole demo, that gives two groups:

The two settings do different jobs, and it helps to keep them apart. Execution order groups decide the order in which Atlantis runs the projects of one command. depends_on decides whether a project may be applied at all before the projects it names. The fifth pull request changes both stacks and then tries to apply app on its own first, which only depends_on can refuse:

**Apply Failed**: Can't apply your project unless you apply its dependencies: [aws-acme-development-eu-west-1-network]

A plain atlantis apply afterwards applies network, then app.

Because depends_on matches project names, it needs them: with --create-project-name=false the tool fails with an error rather than produce entries Atlantis can’t match. And after and before resolve the same way they do for terramate run, so an entry such as tag:network or a directory that isn’t itself a stack expands to every stack it covers.

Running Terraform through terramate run

Neither demo needs it, but there is one more thing the tool can generate that belongs in a post about advanced use: an Atlantis workflow that runs Terraform through terramate run instead of calling it directly. The argument for it is that Terramate’s own checks, its safeguards, then apply inside Atlantis. Adding --generate-workflows to the demo gives every project workflow: terramate and appends this block:

workflows:
  terramate:
    apply:
      steps:
        - run: terramate run --no-recursive --disable-safeguards=git -- terraform apply "$PLANFILE"
    plan:
      steps:
        - run: terramate run --no-recursive --disable-safeguards=git -- terraform init -input=false > /dev/null
        - run: terramate run --no-recursive --disable-safeguards=git -- terraform plan -input=false -out="$PLANFILE"

Each part of those commands is there for a reason, and the workflow documentation explains them:

Without the second flag, the plan fails with current HEAD is out-of-date with the remote base branch, which reads like an Atlantis problem but is Terramate’s. The outdated-code safeguard stays on, and it is the one most worth having, though not in this post’s setup. Here the first hook has just generated the code, so the check always passes. In the first post’s setup, with generated code committed, it is what refuses to plan a commit that forgot to regenerate.

Three costs come with it. A top-level workflows block needs allow_custom_workflows: true on the server, alongside allowed_overrides: [workflow]. A project’s terraform_version no longer picks the binary, because a run step uses whatever terraform is on the server’s PATH, and Atlantis’ version switching only applies to its built-in steps. And the plan output is now one layer further from Terraform, which matters to the person reading a failed plan. I’d treat it as worth it when Terramate’s safeguards are something you rely on elsewhere, and unnecessary when they aren’t.

Corner cases

The mechanism above is simple, but it sits between two tools with their own rules, and a few cases behave in ways that are worth knowing before they surprise you.

Project names move with the tree

The first post’s demo named its projects acme-development-eu-west-1-app; this one names the same stack aws-acme-development-eu-west-1-app. Nothing was configured differently. The default auto-strip strategy removes the path prefix that all stacks share, and the shared/settings stack shrinks that prefix:

Adding one stack outside the shared prefix therefore renames every existing project, and any atlantis plan -p command, script or runbook that names a project by its old name stops matching. With depends_on switched on, the rename is at least consistent, because the entries are regenerated from the same names, but anything outside the file is not. Filters don’t have this effect, since names are always computed against every stack, but changes to the tree do.

Pinning the prefix makes the names independent of the tree:

$ terramate-atlantis-config generate --output - --no-header \
    --project-name-prefix stacks/aws | grep 'name:'
  - name: acme-development-eu-west-1-app
  - name: acme-development-eu-west-1-network
  - name: acme-production-eu-west-1-app
  - name: acme-production-eu-west-1-network
  - name: stacks-shared-settings

The stack outside the prefix keeps its full path, which is a reasonable price for names that only change when you change them.

Don’t combine when_modified_from_config with --changed

It’s tempting to use both flags: --changed to keep the file small, and when_modified_from_config to catch the indirect changes. But --changed removes projects before Atlantis ever sees them, and as the transcript earlier showed, it removes the very stacks affected by a parent config change. Their carefully derived when_modified never gets a chance to match.

If you do use --changed, for instance with a committed file and a repository that never changes globals, two details of its comparison are worth knowing. The base ref has to exist in the checkout, which a shallow or single-branch clone (the default of actions/checkout, and of Atlantis’ branch checkout strategy) doesn’t provide. And the comparison is direct, not from the merge base, so if main has moved on since the branch was cut, stacks changed only on main count as changed too. Passing the merge base avoids the second:

terramate-atlantis-config generate --changed \
  --changed-base-ref "$(git merge-base origin/main HEAD)"

Globals that override the derived list

Stack globals from the first post interact with the flag, and not all in the same way:

atlantis_extra_deps is the right tool for a file inside a stack directory that the defaults miss. Terramate counts any file in a stack’s directory, such as files/policy.json, as a change to it, while the tool’s default patterns only match Terraform files. atlantis_when_modified is the blunter instrument: a stack that sets it opts out of everything above, including the parent config globs, and is back to the problem this post started with.

Nested stacks

Terramate allows a stack inside another stack’s directory. The nested one inherits the parent’s globals, so its project gets the parent stack’s config files as well as its own. The reverse isn’t true: a change in the child’s directory does not plan the parent, because Terramate doesn’t apply the child’s config to it. This matches what each stack actually reads, which is the right test for every entry in the list.

Ordering through stacks that aren’t generated

A stack that is filtered out, skipped or unchanged isn’t a project, so it can’t appear in depends_on. Ordering is followed through it, as terramate run does. Take three stacks where c runs after b, and b runs after a:

A dependency cycle, which Terramate would also refuse to order, becomes a warning, and the stacks on it share an execution order group. With depends_on on, Atlantis can’t apply those projects until the cycle is broken, since each waits for the other.

What the tool can’t see

Some inputs are invisible to a tool that reads files at hook time. Module calls in .tf.json files aren’t read, matching Terramate’s own change detection. An import with a glob source, such as /imports/*.tm.hcl, lists the files that exist when the tool runs, so a pull request that only deletes one of them matches nothing. And a module outside the repository root is skipped with a warning rather than an error, since Atlantis couldn’t match a pull request’s files against it anyway:

Warning: when_modified from config: skipping module "../../outside" called from one/main.tf: it is outside the root directory

That warning comes from the tool’s autoplan documentation rather than from the demo, which has no such module. As with the warnings in the first post, when the hook runs it, it appears only on the hook’s job page in Atlantis, not in a pull request comment.

The size of the file

Each project gets two entries per directory between the stack and the root, so the five-deep stack above carries twelve config entries before its imports and modules. With the file generated on the server nobody reads it, so this costs little. If you commit atlantis.yaml instead, expect it to grow, and expect it to change whenever an import, watch list or module call does.

Limitations

The approach has one deliberate bias: it plans too much rather than too little. A glob over an account directory’s config files can’t tell a change to a global from a change to a comment, or a global every stack reads from one none of them does, so any edit there plans every stack below it. For a tool that decides whether infrastructure changes get reviewed, that seems like the right direction to err in, but it does mean noisy plans for cosmetic edits.

It also moves work onto the Atlantis server. The image needs terramate and the tool, both pinned, and both generation steps now run in Atlantis’ checkout of every pull request. The change-detection behavior described above was observed with the Terramate version the demo pins, and the tool’s documentation reports the same for the release after it; a future Terramate release could map parent changes down to stacks, which would make --changed a viable alternative.

Where to from here?

We’ve followed one global change from a quiet non-event to a precise plan, and the underlying rule is the one from the first post, applied one level deeper: anything Atlantis needs to know should be derived from what Terramate already reads, rather than maintained beside it. The ordering settings and the generated workflow are the same rule again, applied to after and to Terramate’s safeguards.

The demo repository is set up to be forked, and its pull requests are a good place to test a change to your own layout before it meets a real one. If your repository has a kind of input none of the four rows above describes, that is the case I’d most like to hear about.