Pointing an AI coding agent at a production platform with real payroll data is either a superpower or an incident, depending entirely on the scaffolding around it. Here's the harness we built — per-repo operating manuals, persistent typed memory, runbook skills, and hard guardrails — and what it changed.
The Problem
Agentic coding tools got genuinely good. Good enough to investigate a billing discrepancy across two large repos, trace it to an exact line, and draft the fix. But out of the box, two failure modes make them nearly useless — and occasionally dangerous — on a real production platform:
Amnesia. Every session starts from zero. The agent rediscovers that the workflow API has no OpenAPI spec, that the dev server's startup banner lies about which port it's targeting, that AWS SSO credentials get baked into a process's environment at launch and expire an hour later. Each rediscovery burns thirty minutes. Worse, hard-won corrections evaporate: the agent (or you) spends a day learning that a plausible theory was wrong, and next session it confidently proposes the same wrong theory again.
No sense of blast radius. Our workspace spans a half-dozen repos around real payroll data: employee PII in warehouse tables, live credentials in env files, Terraform that controls production, invoicing jobs that bill actual clients. An agent that will happily SELECT * a table with national IDs in it, or cat an env file full of secrets into its own context, is not a tool you can hand real work to.
The default answer — "only use AI on toy problems" — throws away most of the value. We wanted the opposite: the agent doing real investigations against real systems, with the environment itself making the dangerous paths hard.
The Solution: A Harness in Four Layers
What emerged is best described as a harness — the same idea as a test harness, but for an agent. Four layers:
1. Per-repo operating manuals (CLAUDE.md files). Every repo carries a context file the agent loads automatically: what the repo is, how it relates to its siblings, the commands that actually work, and the gotchas that cost us time. The workspace root has its own, mapping the topology — which repo is legacy, which is the rebuild, which directory looks like a repo root but isn't, which UI must never be trusted for dollar figures. These files are written like you'd brief a sharp new engineer: not "here is our architecture" prose, but "the dev script hardcodes the wrong port; bypass it like this."
2. Persistent, typed memory. Beyond per-repo docs, the agent keeps its own memory directory — small files, one fact each, typed as project (ongoing work and its state), feedback (corrections I've given it, with the why), reference (where things live), and user (how I like to work). Two conventions do most of the work here:
- ▸Memory records corrections, not just facts. When a theory turns out wrong, the file gets updated to say so explicitly — "an earlier version of this note blamed X; that was wrong, the real mechanism is Y." Supersede, don't silently delete. The agent stops re-proposing debunked ideas.
- ▸The memory directory is backed up to a git repo, split by account. Which sounds like overkill until a machine dies, or until you realize the memory is the institutional knowledge and deserves the same durability as code.
3. Skills: runbooks the agent can execute. Recurring multi-step procedures get captured as skills — parameterized runbooks. Cold-starting the legacy stack is the canonical example: it hits three separate non-obvious bugs (a hardcoded wrong port, credentials that expire inside a running process, a stale default URL pointing one service at another's port). The skill applies all three fixes every time instead of rediscovering them. Same for domain workflows like classifying deduction codes against the legacy system's real behavior during client onboarding — judgment-heavy, but with a defined procedure and defined evidence standards.
4. Hard guardrails, written down and enforced. The rules that matter are explicit in the context files, phrased as absolutes the agent can't rationalize around:
- ▸PII: aggregate queries only on tables with employee data; never
SELECT *; certain fields never appear in payloads, logs, fixtures, or prompts. - ▸Secrets: never read an env file in full — extract the single needed value programmatically; never
sourceit (shell expansion silently corrupts values containing$); never print a credential, verify copies by hash and byte-length instead. - ▸Mutation: read-only investigation is free; anything that mutates a real system — triggering a billing run, applying Terraform — requires explicit human confirmation, every time. Infra applies stay human-gated by design (that's its own post).
- ▸Isolation: work and personal agent profiles are fully separate — separate config, separate memory, separate accounts. Work context never bleeds into personal sessions or vice versa.
On top of that, the tooling's own permission system acts as a second net: when the agent reached for an SSO credential — a bigger blast radius than the narrowly-scoped secrets it had touched before — the permission classifier flagged it for manual approval. Defense in depth: the written rules shape behavior, the permission layer catches what slips.
What It Provided
The honest summary: the agent went from "clever intern who needs a babysitter" to "colleague who starts every session already briefed."
- ▸Investigations that used to take me a day happen in a session. The agent has run full legacy-vs-rebuild comparisons on real clients — reading pipeline source across two large repos, querying the warehouse read-only, and root-causing discrepancies to an exact line and config key — because the harness told it where the money data lives and which tools are safe.
- ▸Mistakes stopped repeating. The cold-start bugs get fixed proactively instead of rediscovered. Corrected theories stay corrected.
- ▸The guardrails held under real pressure. Real secrets got moved between environments with values never displayed, verified by checksum. Risky operations got paused and escalated to me instead of executed.
- ▸A side effect I didn't predict: writing the harness improved human onboarding too. The operating manuals don't rot the way wikis do, because the agent exercises them daily — a stale instruction fails visibly within days.
Future Implications
The mental model I've landed on: treat your AI agent like a new team member, and treat its environment like the onboarding program. Nobody expects a new hire to be productive with no docs, no runbooks, no access policy, and no memory of yesterday. We expect exactly that from agents, then blame the model.
Two predictions I'm fairly confident in. First, "harness engineering" becomes a real discipline — the differentiator between teams that get compounding value from AI and teams that get demos. The model is increasingly the commodity; the context, memory, guardrails, and verification loops are the asset you own. Second, documentation economics just flipped. The ROI on writing down a gotcha used to be "maybe a future human reads it." Now the reader is guaranteed: the agent loads it next session. On our team, the half-life of undocumented knowledge collapsed — and so did the excuse for not writing it down.