We're replacing a legacy data-pipeline invoicing system with a rebuilt engine — and invoices are money, so "looks right" isn't a bar you can ship against. Here's the local harness we built to run the same real payroll through both systems side by side, and the bugs it caught that no code review would have.
The Problem
The platform I work on processes payroll and invoicing for a portfolio of clients. The invoicing logic lives in a legacy data-pipeline monorepo (Dagster-based) that has grown organically for years. A new engine is being built to replace it — cleaner config model, proper APIs, a real portal in front of it.
The hard constraint: an invoice is a bill someone pays. If the new engine produces a number that's $400 off, that's not a rendering bug — it's a real dollar discrepancy with a real client. Before any cutover, we needed to answer one question with evidence, per client: does the new engine produce the same dollars as the old one, for the same real payroll?
That question is surprisingly hard to answer when both systems are cloud-coupled. The new stack assumed AWS RDS for its database, S3 for document storage, and an identity provider for SSO. The legacy stack had its own web of services. Running "both systems on the same input" meant either testing in shared cloud environments (slow, contended, risky) or finding a way to bring the whole thing down to a laptop.
The Solution: A Docker Compose Harness with Cloud Stand-Ins
We built a local harness — a Docker Compose stack that swaps every cloud dependency for a local stand-in:
- ▸Postgres container in place of RDS
- ▸MinIO in place of S3 (same API surface, so application code doesn't change)
- ▸A local identity stand-in in place of the real SSO provider
- ▸Adminer for quick database inspection, and even a browser-based terminal (ttyd) for poking around
Port assignments were chosen deliberately so the new stack never collides with the legacy one: the new web app on :3010 because the legacy UI already owns :3000, APIs on :8000/:8010, the harness Postgres on :5433 next to the native one on :5432.
That last detail matters because the whole point is running both stacks simultaneously. Day to day this looks like two tmux sessions: one running the legacy pipeline (orchestrator, workflow API, UI), one running the new stack via Compose. Same machine, same real client payroll data, two complete invoicing systems.
The validation loop:
- ▸Pick a real client and a real pay period.
- ▸Run the payroll through the legacy pipeline.
- ▸Run the same payroll through the new engine in the harness.
- ▸Compare the money at the database level — direct SQL against the warehouse schema where both systems' line items land. Never the UIs.
That last rule is non-negotiable. UIs round, aggregate, and omit. When you're hunting a $980 discrepancy, you query the source tables with aggregate queries and you reconcile line item groups one by one. (And on tables carrying employee data, aggregates only — never SELECT *. More on data discipline in a future post.)
What It Caught
The harness earned its keep almost immediately, because "run everything locally against real data" turns vague doubts into traceable diffs. Three findings stand out:
A config default that silently dropped an entire premium category. The new engine folded a whole class of insurance premium dollars into the invoice only when a specific config flag was set — and the config-suggestion tool defaulted that flag to off. Nothing at review time checked whether "off" was right for the client's actual payroll data, and the reconciliation step still reported success. On one client, this was a five-figure delta per invoice. The fix was one config value; the lesson was bigger: we proposed a review-time warning that fires whenever the underlying data carries dollars that the config is about to ignore. A reconciliation check that can pass while an entire category of money goes missing isn't a reconciliation check.
Hardcoded one-off logic buried in the legacy code. One client's fee carve-out wasn't in configuration at all — it was a literal if account_id in {...} deep in the legacy pipeline, invisible unless you knew to grep for the account code. The new engine's config model had no way to express it because nobody knew it existed. You don't find this by reading the config repo. You find it by running a real invoice through both systems and refusing to stop until every dollar of the delta has a name.
Fees that only exist when the physical world intervenes. The legacy system charged a small flat fee per physical paycheck actually printed — a generic, config-driven mechanism the new engine's model didn't cover. Sounds trivial, except a sweep showed a double-digit number of clients had the fee configured. A $150 delta on one invoice turned into a known, quantified model gap across the portfolio, with a concrete proposal (two config fields and one condition) instead of a shrug.
The common thread: none of these were "bugs" in the sense a unit test would catch. They were model gaps and silent defaults, and the only instrument that detects those is running real inputs through both implementations and explaining the residual to zero.
What the Harness Provided
- ▸A cutover standard. "Delta is zero, or every dollar of the delta is explained and accepted" became the bar for migrating a client. That's a sentence you can put in front of leadership.
- ▸Speed. A full compare-and-root-cause cycle on a client went from "coordinate access to shared environments" to an afternoon on a laptop.
- ▸Safety. Mistakes in a local harness cost nothing. Mistakes in a shared environment during validation become somebody's incident.
- ▸Reusable onboarding rehearsal. The same harness that validates the rebuild now doubles as the rehearsal space for onboarding each new client to the new engine — run them locally first, reconcile, then promote the config.
Future Implications
The part I didn't expect: the harness turned out to be the single most important enabler for using AI agents on this codebase. An agent that can spin up the full stack locally, run a real flow end to end, and query the results is an agent that can verify its own work instead of pattern-matching plausible code. Everything interesting we've since done with AI tooling sits on top of this harness — which is the next post.
If you're sitting on a rebuild-vs-legacy migration and your validation story is "we'll compare in staging," I'd push hard on making the whole thing run locally first. The stand-ins (Postgres for RDS, MinIO for S3, a fake IdP) are a weekend of work. The confidence is compounding.