Terraform modules have to come from somewhere, and in a monorepo the most
natural somewhere is the monorepo itself. The modules live in a modules/
directory alongside the stacks that use them, each stack being a directory
that Terraform plans and applies on its own. Each stack pins every module it
calls to a Git tag of that same repository:
module "orders_db" { source = "github.com/acme/infra//modules/db?ref=db-v1.1.0" name = "orders"}That arrangement has a lot going for it: a module and its callers can change
together in a single pull request, each stack can move to a new module version
on its own schedule, and there is no second repository to look after. What is
less obvious is what Terraform has to do to honor that source argument, and
how the cost of doing so grows with the repository. This post follows a
module source address down to the git clone it turns into, measures that
cost for a monorepo of more than 500 stacks, and projects where it is
heading. The monorepo is planned by Atlantis,
which runs terraform plan on every pull request. Then the post looks at two
ways to reduce that cost: shallow clones, which fetch only the tagged commit
and need one argument added to each source, and an internal module registry,
which is more work but changes what gets downloaded in the first place.
- About 12 hours a month
- Spent cloning the whole repository for modules, at laptop speed (about 1 MB/s): about 40 GB a month, averaged over six months of real plans, and growing with the repository.
- About 3 hours with shallow clones
- Shallow clones fetch only the tagged commit, not the whole history. The cost rises toward 6 hours as stacks move to newer module tags.
- Minutes with a registry
- A registry can serve each module as a small archive, an estimated second or so to fetch, or 9 to 15 minutes a month. Projected, not yet measured.
Terraform downloads packages, not modules
When Terraform reads a module source address like the one above, it splits it
into two parts at the //. The repository address before it, together with
the ?ref= arguments after the path, names a module package: something
Terraform knows how to download as a single unit, such as a Git repository at
a particular revision or an archive file at a URL. The path after the // is
a location inside that package, telling Terraform where to find the module
once the package is on disk.
Terraform never downloads “a module”. It downloads a package and then looks
inside it, so for a Git source, asking for modules/db at db-v1.1.0 really
means asking for the entire infra repository at that tag.
The download itself is handled by go-getter, the library Terraform uses to
fetch module packages, and for Git sources go-getter makes no attempt to be
economical. Unless the source address includes a depth argument, it runs a
plain git clone of the remote, with every branch, every tag and every
commit, and only then checks out the requested tag. The default is right
there in
get_git.go:
depth := 0 // 0 means "don't use shallow clone".
Terraform does avoid downloading the same package twice. Within a single
terraform init, two module calls that refer to the same package cause only
one download, and the second call
gets a copy
of the first one’s directory. But that reuse is keyed on the package address,
so two calls at different tags of the same repository are two separate full
clones, and nothing at all is shared between one working directory and the
next.
Seeing it on a small scale
It’s easier to get a feel for this with something small, so here is a
throwaway repository built to show the effect: two tiny modules, 125 commits,
and an 18 MB file that was committed once and later replaced with a different
one. One stack calls both modules, each at its own tag, through file://
Git sources, because Git ignores --depth for plain local paths and that
will matter in a moment. <dir> stands for the demo’s directory:
module "orders_db" { source = "git::file://<dir>/infra//modules/db?ref=db-v1.1.0" name = "orders"}
module "orders_secret" { source = "git::file://<dir>/infra//modules/secret?ref=secret-v1.0.0" path = "orders/db"}Running terraform get, which performs just the module-download part of
terraform init, gives:
$ terraform getDownloading git::file://<dir>/infra?ref=secret-v1.0.0 for orders_secret...- orders_secret in .terraform/modules/orders_secret/modules/secretDownloading git::file://<dir>/infra?ref=db-v1.1.0 for orders_db...- orders_db in .terraform/modules/orders_db/modules/db$ du -sh .terraform/modules 86M .terraform/modulesNotice where each module ended up: .terraform/modules/orders_db/modules/db.
The directory orders_db is a complete clone of the repository, history and
all, and the module Terraform actually wanted is one small directory inside
it. The two modules amount to a few hundred bytes of HCL, and they arrived
inside 86 MB of clones, about 70 MB of it Git history, because both versions
of that large file are still in the history even though only one of them is
in the tree.
How often Atlantis pays for it
A single clone of a single repository isn’t much to worry about. What makes it add up is how often it happens, and in a repository planned by Atlantis that is quite often.
Atlantis plans each pull request in its own working directory and keeps that
directory from one plan to the next, including .terraform/, where
Terraform keeps the modules it has installed. Terraform skips any module
that is already installed there, so in practice each stack downloads its
modules once per pull request, on its first plan. This post calls that first
plan a cold init. (If the stacks’ dependency lock files,
.terraform.lock.hcl, aren’t committed, Atlantis runs terraform init -upgrade, which reinstalls every module, and then every plan is a cold
init. The numbers below assume Atlantis doesn’t do that.)
The repository in question has about 550 stacks, and about 300 of them call its own modules through Git sources. A full clone is about 80 MB. Over six months, about 70 people opened pull requests against it, and with bot updates that came to around 160 pull requests in a typical month. Atlantis posts every plan as a comment on its pull request, which makes those comments a convenient record. Leaving out one unusually large pull request, they add up to about 600 plan runs and 1,450 stack inits a month. About 560 of those inits were cold. Some needed no clone at all, because their stack calls no Git modules, and others needed one for each tag their stack pins; together they cloned the repository about 580 times.
That works out to about 40 GB of Git data a month, spent entirely on fetching
modules. To turn that into time, 18 of the git clone commands that go-getter
runs, from small shallow clones up to the full repository, were timed on the
laptop. A straight line fits them reasonably well, at about 3.5 seconds per
clone plus about a second per megabyte, which puts a full clone of today’s
repository at around 80 seconds. Costing each month at the size the
repository had then, a month of cold inits averages about 12 hours of
cloning. Over half a year that is about 240 GB and 73 hours.
Every hour figure in this post is at the laptop’s speed. The Atlantis server very likely has a faster connection, so its hours will be fewer. The bytes, though, are the same wherever they’re downloaded, and so is the shape of the problem: each person waiting on a cold init is waiting for the whole repository, once per tag.
Shallow clones
The cheapest thing to try is the depth argument that go-getter already
checks for. Adding &depth=1 to every source turns each clone into
git clone --depth 1 --branch <tag>, which fetches only the tagged commit and
none of the history behind it. Here is the same demo stack with that one
change:
$ terraform getDownloading git::file://<dir>/infra?ref=db-v1.1.0&depth=1 for orders_db...- orders_db in .terraform/modules/orders_db/modules/dbDownloading git::file://<dir>/infra?ref=secret-v1.0.0&depth=1 for orders_secret...- orders_secret in .terraform/modules/orders_secret/modules/secret$ du -sh .terraform/modules 35M .terraform/modulesThe history is gone, but 35 MB remains, and almost all of it is the current
version of that large file, counted once in the shallow clone’s .git and
once in the checkout. A shallow clone still downloads the whole tree at the
tag: not just the module you asked for, but every other module and every
stack in the repository as they were at that commit.
In the real repository that tree is about 31 MB at a recent tag, most of it
images. Replaying the same six months of plans with depth=1
on every source gives about 9 GB and 3 hours a month, or 55 GB and 19 hours
over half a year. The modules Terraform installs are identical either way;
only the history is missing.
The ref has to be a tag or a branch rather than a commit, because go-getter
passes it to git clone --branch, which doesn’t accept commit IDs. And
because every source string changes, every stack reinstalls its modules and
gets planned once, so it’s kinder to Atlantis, and to reviewers, to roll it
out a group of stacks at a time.
Shallow clones are worth doing on their own. But they reduce the cost without changing its shape: every module fetch still downloads the repository as it was at that tag, and that tree keeps growing.
Letting a registry decide what to download
Terraform has a second kind of module source that works quite differently. A registry source names a module and leaves the version to a separate argument:
module "orders_db" { source = "registry.example.com/acme/db/aws" version = "1.1.0" name = "orders"}
module "orders_secret" { source = "registry.example.com/acme/secret/aws" version = "1.0.0" path = "orders/db"}For a source like this, Terraform doesn’t download anything directly. It asks
the registry which versions of the module exist, picks the one that matches
version, and then asks the registry where to get it. The registry answers
with an X-Terraform-Get header containing an ordinary source address, which
Terraform then hands to go-getter just as it would any other.
The interesting part is that the registry gets to choose that address. The public Terraform Registry, for example, points back at Git:
$ m=terraform-aws-modules/vpc/aws/6.0.1$ curl -s -o /dev/null -D - \ "https://registry.terraform.io/v1/modules/$m/download" \ | grep -i -e '^HTTP' -e '^x-terraform-get'HTTP/2 204x-terraform-get: git::https://github.com/terraform-aws-modules/terraform-aws-vpc?ref=a0307d4d1807de60b3868b96ef1b369808289157That works well enough for public modules, because each one lives in a
repository of its own. A registry that did the same for a monorepo would
change the syntax of every source and nothing else. What a monorepo needs is
a registry that stores each module version as its own archive, so that the
package Terraform downloads is the module and nothing more.
That is exactly what an internal registry such as
boring-registry does.
Publishing a module with boring-registry upload module packs the module’s
directory into a tar.gz in object storage, and when Terraform asks where to
download a version, the registry answers with a
pre-signed URL
for that one file. The demo has a small stand-in registry that serves
archives the same way, and pointing the stack above at it gives:
$ terraform getDownloading registry.example.com/acme/secret/aws 1.0.0 for orders_secret...- orders_secret in .terraform/modules/orders_secretDownloading registry.example.com/acme/db/aws 1.1.0 for orders_db...- orders_db in .terraform/modules/orders_db$ du -sh .terraform/modules 12K .terraform/modulesEach module now lands directly in its own directory, with no modules/db
path inside a clone and no .git directory anywhere.
Those archives are small. Built for every tag the real stacks used over the
six months, they come out at a median of about 6 KB. There is no registry in
front of this repository to time, so the fetch itself was estimated from the
same mechanism: downloading a small public module as an HTTPS archive took
about a second end to end with terraform get, and the two registry lookups
Terraform makes for each module took about 0.2 to 0.5 seconds between them
against the public registry.
Put next to the clones, the difference is hard to miss:
A cold init that needs one module becomes a handful of small HTTPS requests instead of a minute or so of cloning, and a month of them takes nine to fifteen minutes instead of hours.
Looking a year ahead
All of the numbers so far describe the last six months. The more useful question is where those numbers are going, and the repository’s own history gives a reasonable way to estimate that.
Measuring the size of a full clone every six months since the repository began shows a pattern that’s probably familiar to anyone who has looked after a long-lived monorepo. There is steady growth from ordinary commits, and then there are steps, each one a large binary file being committed. Git never forgets a file once it is committed, so every step is permanent for full clones, even after the file is shrunk or deleted. Shallow clones of the latest commit see the steps too, but only for as long as the file stays in the tree:
Since late 2024 a full clone has grown by about 1.7 MB a month on average, steps included. If that continues, it reaches about 100 MB a year from now, or anywhere from about 90 to 105 MB if the average is taken over a longer or shorter stretch of history. Feeding that growth into the same timing line, with plan activity held at the six-month average, gives a forecast for the time spent cloning:
At today’s repository size, a month of cold inits costs about 14 hours of full clones, a little more than the six-month average because the repository is bigger now than it was in the spring. A year from now it would be about 17 hours a month, and the year in between would add up to roughly 185 hours.
Shallow clones get a band rather than a line, because their cost depends on which tags the stacks pin, and a tag never changes. Many stacks still pin older tags, whose trees are small, so the average shallow clone today is about 16 MB and a month costs about 3 hours. A stack that stays where it is costs the same next year; that’s the bottom of the band. A stack that moves to a newer tag, which is the whole point of releasing modules, inherits the larger tree, and if every stack were on the newest tag a month would cost about 6 hours; that’s the top. Over the coming year, shallow clones land somewhere between about 40 and 75 hours, depending on how quickly stacks move.
The registry line stays flat by construction, because a module archive contains one module and doesn’t grow when the rest of the repository does. At a second or so per module, the whole year comes to about two and a half hours.
These are projections, and simple ones. They assume the repository keeps growing as it has, and that plan activity stays at the six-month average. Neither assumption will hold exactly, but together they show which way each option is heading.
What a registry asks of you
None of this makes a registry free. Every module version has to be published, so
the release process that cuts module tags today needs an upload step as well,
and that step has to be reliable, because a stack can’t use a version that was
tagged but never uploaded. Modules also have to be able to stand on their own. A
module that calls a sibling module with source = "../secret" works from a Git
clone, because the sibling happens to be on disk next to it, but an archive
contains only the one module, and Terraform stops with “Local module path
escapes module package”. Those calls have to become registry sources too.
There is also a new piece of infrastructure in the path of every plan. If the registry requires authentication, Terraform and tools such as Renovate need credentials for it, and if it is down, cold inits fail, much as they do today when GitHub can’t be reached. Those are real costs, but they are the ordinary costs of running any internal service, and they don’t grow with the repository.
Finally, the registry numbers in this post are projections. None of this repository’s modules has been served from a registry. The minutes above come from fetch times estimated with a public archive and public registry lookups, multiplied by the number of fetches replayed from six months of plans, not from a migration that has actually happened.
Where to from here?
The root of all this is a small detail in how Terraform treats module sources: it downloads packages, not modules, and for a Git source the package is the whole repository. In a small repository that detail never matters. In a busy monorepo of a few hundred stacks it adds up to tens of gigabytes a month, which at laptop speed is hours of cloning. And because Git keeps everything it has ever been given, that number only goes one way.
Shallow clones need one argument added to each source, take away between half and three quarters of the cost, and are cheap enough to do first. But they still download the repository’s tree, so what they leave behind grows with that tree and rises as stacks move to newer tags. An internal registry that serves each module as its own archive changes what a package is, and with it the cost of a cold init, from a clone of everything to a download of just the module that was asked for. For a monorepo that expects to keep growing, I think that’s the better place to end up.
Comments