We added CI to a Terraform setup that had none — but kept every apply behind a human gate, on purpose. The first CI run immediately surfaced three real bugs a clean local plan never would have. Here's the architecture, the OIDC gotchas, and why 'human-gated' is a design point, not a transitional embarrassment.
The Problem
Our infrastructure-as-code had grown up inside an application monorepo: Terraform for staging and production living in an infra/ directory, with one engineer reviewing every plan and running every terraform apply from their own machine. No CI for infra at all — while the application deployed through fully automated pipelines.
An audit when I got access made the risks concrete:
- ▸Staging was healthy — the live environment matched its Terraform almost exactly. Good.
- ▸Production was 89 lines of documented intent: a main.tf that had never been applied. The real account had nothing but defaults in it.
- ▸The account-baseline configuration (audit trails, threat detection, alerting, budgets) kept its Terraform state on the last applier's laptop — no remote backend. If that machine died, the state for the security baseline died with it.
- ▸Generic modules (networking, ECS platform, data stores) were tangled into one product's repo, with a second product on the roadmap that would need all of them.
The naive fix — "add CI, auto-apply on merge, like the app code" — was wrong for us, and knowing why it was wrong shaped everything: infra changes here have blast radii that include production databases and client-facing billing systems, the team is small, and a reviewed plan output genuinely is the best review artifact in infrastructure. The existing human-gated convention wasn't laziness; it was a load-bearing control. The problem was everything around it: no automated validation, no consistent plan environment, state fragility, and a bus factor of one.
The Solution
Repo split: generic vs. product. A new standalone infra repo with two top-level ideas: modules/ for the eight generic building blocks (network, ECS platform, ECS service, data, artifacts, OIDC, security baseline, and so on) and products/<name>/ for everything product-specific — bootstrap, baseline, per-environment roots. The next product reuses the modules; nothing product-flavored leaks into them. The security baseline got promoted from inline copy-paste into a proper callable module, and its state moved off a laptop into a real S3 backend via a 12-resource-per-account import runbook.
CI that plans always and applies never — on its own. Two workflows:
- ▸Plan on every PR: format check, validate, and a real
terraform planper root, posted to the PR as a comment. Runs under a read-only role. This is the review artifact. - ▸Apply via manual dispatch only: a human triggers it, types a confirmation string, and it runs inside a protected environment — the only place apply ever executes in CI. No auto-apply on merge, deliberately.
Each account gets two OIDC-assumed roles: a read-only plan role and an admin apply role, so the thing that runs on every PR physically cannot mutate anything.
What CI Immediately Taught Us
Here's the part worth the price of admission: everything planned clean locally, and the first real CI run failed on three bugs that no local run could ever have surfaced.
1. The OIDC trust policy that every tutorial gives you was wrong for our org. GitHub's "immutable subject claims" setting — enabled org-wide — changes the token's subject format from the classic repo:org/repo:ref to a form embedding org and repo IDs. Every trust policy written from the docs rejected real tokens. We confirmed the actual claim by reading CloudTrail's record of the failed AssumeRoleWithWebIdentity calls — the event logs the subject GitHub actually sent. Local plans never touch this code path; only a real CI run does.
2. Provider config that assumes a laptop. Our provider blocks used named AWS CLI profiles — fine locally with SSO, meaningless in CI where credentials exist only as ambient env vars from the assumed role. The workflows now synthesize a matching credentials profile before Terraform runs.
3. Read-only roles can't take state locks. The plan workflow died acquiring the S3 state lock — lock operations need write permissions the read-only role rightly lacks. Fix: -lock=false on the plan path, rather than widening a role whose entire purpose is that it can't write.
A pattern worth internalizing: a clean local plan validates your Terraform; only a real run in the real execution environment validates your pipeline.
Operating it taught us more. A hosted runner hung twice holding the state lock on a change that takes seconds — recovery meant reading the lock ID off the lock object and force-unlocking, and the durable fix was a hard timeout-minutes on the apply job (roughly double our largest-ever real apply), because neither GitHub's 6-hour default nor Terraform's lock-acquisition timeout bounds how long a job can hold a lock. And ECS's deployment circuit breaker produced the sneakiest failure mode we've seen: an apply registered a task definition pointing at an image tag that no longer existed, tasks failed to pull, and the breaker silently rolled back to the last good revision — every dashboard green, the change simply absent. The verification rule that came out of it: after any apply that can replace a task definition, check the running task's definition ARN directly. "It looks stable" and "it quietly reverted" are indistinguishable from the service's pointer alone.
What It Provided
- ▸Bus factor fixed without giving up the gate. Anyone can propose infra changes and get a trustworthy plan on the PR; applies still require a deliberate human act, now through one audited path instead of someone's terminal.
- ▸Production went from fiction to real. The full environment — network, ECS services, Multi-AZ database, DNS, TLS — was built out and applied through the new workflow: 100+ resources, zero errors, follow-up plan clean.
- ▸Review quality went up. Reviewers read an exact plan diff produced in a consistent environment, not a description of what the author believes will happen.
- ▸The gate caught a real near-miss. A branch cut before an urgent DNS fix merged would have destroyed that fix on apply. Re-planning immediately before apply — the habit the human gate enforces — caught it; the branch was rebased and re-planned before anything ran.
Future Implications
The industry default says the endpoint of maturity is full auto-apply. I think that's backwards for high-blast-radius infrastructure on small teams. The right frame is earned autonomy: automate the validation completely, keep the mutation gated, and only widen the gate where the blast radius has been engineered down — and that's more true, not less, as AI agents write a growing share of the Terraform. Our agent tooling drafts modules and reads plan output daily; the one thing it cannot do, by construction, is apply. The plan-always/apply-by-hand pattern turns out to be exactly the shape you want when your most prolific infra author is a machine: full-speed proposals, full-stop execution, human judgment exactly and only at the moment of irreversibility.