Deploy workflows are production code that runs with production credentials — and most teams' testing strategy for them is 'run it for real and see.' We ended up in a situation where real runs were impossible, which forced a better answer: lint, execute locally, and fail closed. Here's the preflight stack and what it caught.
The Problem
A deploy workflow is the most dangerous code most teams never test. It runs with credentials that can mutate production, it's full of subtle execution semantics (expression interpolation, job-level conditions, environment gates), and the standard way to find out whether it works is to merge it and run it against the real thing.
We hit this in its sharpest form. Three production deploy workflows had just been drafted — build, push, guarded ECS roll, a database-migration step, a credential smoke test — mirroring their staging counterparts but dispatch-only, no auto-trigger. And then two separate walls appeared at once:
- ▸The protected GitHub Environment the workflows depended on couldn't be created yet — that needs repo admin access, which sat with someone else. No environment, no real run.
- ▸Separately, an org-wide CI billing failure took down all GitHub Actions for a stretch. Every check showed a bare
failurewith zero recorded steps; the actual cause was only visible in the check run's annotations via the API.
So: three workflows that gate production deploys, zero ability to execute them for real, and a reviewer who had just found five genuine issues in them — which rather undermined the comforting idea that workflow YAML is too simple to need testing. The five, generalized:
- ▸No branch restriction on manual dispatch.
workflow_dispatchlets any user with write access run the workflow from any ref. The compensating control — an environment branch policy — didn't exist yet. Any branch could have deployed to prod. - ▸Missing concurrency groups on two of the three. With dispatch-only deploys, "dispatch, notice the wrong ref, dispatch again" is the normal recovery path, not an edge case — overlapping rolls must queue, not race.
- ▸Fixed-duration cloud waiters.
aws ecs wait services-stableis a hardcoded ~10-minute waiter (we confirmed in the SDK source — not tunable) that errors mid-step rather than failing cleanly on slow deploys. - ▸A smoke-test step with no guard. If the roll step was skipped, the smoke test fell through and tried to run a task with a literal string
"None"as its task definition. - ▸Pushing a mutable
:latesttag alongside the SHA tag — meaning a future infra change that reshapes a task definition would silently pick up whatever image happened to be pushed most recently.
Five real bugs, found by one careful human review. The question preflighting answers: what catches the sixth, when the reviewer is tired?
The Solution: A Preflight Stack
What we converged on is a small test pyramid for pipelines — cheap and static at the bottom, realistic and gated at the top.
Layer 1: Lint the workflows like code. actionlint catches a surprising amount: it's how we confirmed that matrix isn't even a valid context in a job-level if: expression — the kind of fact you otherwise learn by pushing a broken workflow and reading the failure.
Layer 2: Put guards in the file, not in settings. The branch-restriction fix was a job-level if: github.ref == 'refs/heads/main' guard in the workflow itself, rather than relying solely on the environment's branch policy. Two reasons. First, the environment didn't exist yet — an in-file guard works on day zero. Second, and more durable: a guard in the file shows up in the PR diff. Reviewers can see it, lint can check it, and it can't be silently loosened in a settings page nobody watches. Out-of-band protections are fine as a second layer; they're a poor only layer.
Layer 3: Execute the workflows locally with act. act runs your workflow's real steps in a container — it simulates the GitHub runner, not your cloud. That boundary is exactly what makes it safe, if you use it deliberately: we never supplied real AWS credentials, so the OIDC auth step fails closed before any build-push-deploy step can produce a side effect — the same failure boundary as a genuinely missing deploy role. Everything before that point (checkout, guard evaluation, scripting, env wiring) executes for real.
That let us verify the important thing while the environment setup was still blocked: on a feature branch, the ref guard evaluated false and the job skipped entirely — zero steps run. From a main checkout, it evaluated true and proceeded through the real early steps. The guard demonstrably works in both directions, proven before it ever gated a real deploy.
Layer 4: Explicit deadlines everywhere. The fixed cloud waiters got replaced with manual polls against an explicit 30-minute deadline — a loop you can read, tune, and that fails with a clear message instead of a mid-step SDK error. Same philosophy one level up: the apply/deploy jobs themselves got timeout-minutes set to roughly double the largest legitimate run, after watching a hosted runner hang holding a lock. Neither GitHub's 6-hour default nor any built-in covers "this job is wedged"; you have to say what too long means.
What Preflight Caught — and What It Didn't
It caught its own tool's lie. act turned out to cache github.ref in a way that survives removing and re-adding a git worktree at the same path — even after clearing its cache directory. A stale ref meant the guard test could pass or fail based on filesystem history rather than the actual branch. The fix: stop relying on git-based ref detection entirely and pass an explicit event payload (act -e event.json with the ref spelled out). A good preflight is deterministic about its inputs; if the tool infers context, pin the context.
It did not catch an interpolation bug — and the miss is instructive. One workflow passed a JSON-array repository variable into a command line using GitHub's expression syntax directly inside a run: block. GitHub substitutes that expression as raw text before the shell ever parses the line — so the JSON's embedded quotes shattered the surrounding argument. It only surfaced on the first fully real run, because preflight runs never exercised that step with the real variable populated. The fix (and the rule we keep now): never interpolate GitHub expressions directly into shell script text; pass them through env: and let the shell expand the variable, because then the string boundary is resolved before expansion. Preflight shrinks the set of first-run surprises; it doesn't eliminate it. Design the first real run expecting survivors.
And one adjacent lesson about why the pipeline must be the only build path: during the blocked period, images were built and pushed manually from a laptop — an Apple Silicon machine, whose Docker builds default to arm64. ECS Fargate wants x86_64. The result is an image that pushes fine and fails only at task start, with a manifest error three steps removed from the cause. CI runners are x86_64; laptops are whatever they are. Every manual escape hatch you keep open is a class of bug your preflighted pipeline can't protect you from.
What It Provided
- ▸Confidence before the gate existed. The workflows were validated — guards proven, scripts executed, failure modes rehearsed — while the real execution path was still administratively blocked. When the environment finally landed, the first real runs held.
- ▸Review fixes verified, not assumed. The five review findings didn't just get committed; the most safety-critical one was demonstrated working under local execution.
- ▸A reusable boundary pattern. "Withhold the credential and let auth fail closed" turns out to be a general-purpose way to dry-run anything whose side effects live behind an auth step.
Future Implications
The asymmetry here is stark: application code gets unit tests, type checkers, review, staging, and canaries — while the workflow that ships all of that gets eyeballed YAML and a prayer. That made a kind of sense when pipelines were three lines of make deploy. It stopped making sense when they became hundreds of lines of conditions, guards, polls, and credential exchanges.
I'd bet on pipeline testing becoming ordinary discipline within a few years, for the same reason infrastructure testing did: the blast radius keeps growing while the authorship keeps getting cheaper. AI tooling now drafts workflow YAML fluently, which means the volume of plausible-looking pipeline code is about to go up by an order of magnitude. Plausible-looking is exactly the failure mode preflighting exists for. Lint it, run it locally, fail it closed — then let it near production.