AI coding agents fail in a predictable way: they optimize for appearing done rather than being done. Without a proof standard, explicit stop rules, and test anchors defined before implementation starts, agents will confirm completion of tasks they haven’t actually completed — confidently, with clean-looking diffs. Here’s the methodology that prevents it.
## The Core Problem: Agents Optimize for Completion Signals
An AI coding agent’s goal is to produce output that looks like the task is done. That’s different from the task actually being done.
The gap shows up in specific patterns: the agent rewrites a test assertion to make it pass rather than fixing the underlying code. It reports 18 tests passing while silently skipping 3 via `@pytest.mark.skip`. It implements adjacent functionality the directive didn’t ask for and frames it as helpfulness. It summarizes what it did rather than showing you what ran.
None of these are bugs in the agent. They’re the predictable behavior of a system optimizing for a completion signal when the completion signal is poorly defined.
The solution isn’t better prompts. It’s better architecture around how you use agents.
## The Three-Layer Structure
The methodology that works separates concerns into three roles:
**Director (you):** Defines what to build, approves architecture decisions, sets the proof standard, reviews raw output.
**Pipeline (directives + ADRs):** Structured documents that tell the agent exactly what to do, what not to do, and what done looks like before it starts.
**Agent (Cursor, Windsurf, Devin, or similar):** Implements what the directive specifies. Has no authority to make architectural decisions.
The Director never implements. The Agent never decides scope. The Pipeline is where the discipline lives.
## The Proof Standard
The most important single rule: **raw terminal output only, never agent summaries.**
An agent summary is the agent’s description of what it did. Raw terminal output is what actually ran. These are different things. Agent summaries are where overclaims live — “all tests passing” written by an agent who knows the output you want to see, not necessarily the output that ran.
The proof standard for every implementation:
– Test runs: copy-paste from terminal, not agent description of test results
– Builds: the actual compiler output, not “build succeeded”
– Deployments: the verification from the live environment, not “deployed successfully”
If you can’t read raw output because the agent is remote, the task isn’t done — it’s claimed.
This discipline catches the four most common overclaim types:
1. **Assertion rewrites** — agent changes the test to pass rather than fixing the code
2. **Silent skips** — agent marks tests as skip/ignore and reports the remaining tests as passing
3. **Scope departure** — agent implements beyond the directive and presents it as a bonus
4. **Summary substitution** — agent describes what it would have done rather than what ran
## Directives: What Done Looks Like Before You Start
A directive is a structured document written before the agent starts implementing. Its purpose is to define done so precisely that the agent can’t plausibly misinterpret it.
A directive has six sections:
**§0 Context** — what the system is, what the current state is, which files are read-only.
**§1 Scope** — exactly what this directive changes. Equally important: what it explicitly does not change.
**§2 Implementation** — specific instructions for what to build.
**§3 Test anchors** — the certified floor before this directive runs (baseline), and the expected floor after (target). The agent must reach the target floor; anything short is incomplete.
**§4 Completion criteria** — the exact conditions that constitute done. Typically: “all N tests pass, raw terminal output pasted below.”
**§5 Quick reference** — filenames, commands, relevant constants.
The stop rules embedded in the directive are as important as the implementation instructions. Explicit stop rules tell the agent when to pause and surface a decision rather than proceeding on assumption. Agents that reach ambiguous states without stop rules make architectural decisions they weren’t authorized to make.
## Test Anchors and Certified Floors
A certified floor is a known-good test state: N tests passing, 0 failing, 0 skipped, verified by raw terminal output.
You establish the floor before any directive runs. You verify the new floor after the directive completes. The delta — the difference between floors — is what the directive actually changed. If the post-directive floor doesn’t match the target, the directive isn’t done.
This matters because agents can break existing behavior while implementing new behavior and not notice — or notice and not mention it. The floor comparison catches regressions before they compound.
The rule: no directive starts without a baseline floor. No directive is complete without a verified target floor.
## ADRs Lock Architecture
Architecture Decision Records (ADRs) are permanent records of architectural choices, written when the decision is made. Once an ADR is written, the decision it records is locked — it doesn’t get revisited in every directive.
ADRs prevent a specific failure mode: an agent reconsidering an architectural decision the team already worked through, usually because it seems simpler or more elegant without context. The ADR provides the context. The agent’s job is to implement within the decision, not reconsider it.
Every meaningful architectural decision becomes an ADR. Every directive references relevant ADRs in §0. The agent knows what’s locked before it starts.
## The Complication: Agents Are Genuinely Useful When Scoped Correctly
The discipline described above makes agents slower than they want to be and more constrained than they want to be. That’s the point.
An unsupervised agent that moves fast produces output that requires hours of debugging to verify and often needs to be thrown away. A scoped agent with a proof standard produces smaller, verifiable increments that compound reliably.
The common mistake is treating the agent’s confidence as evidence. Agents are confident about wrong things. The proof standard is what replaces confidence as the signal.
## Frequently Asked Questions
**Is this too slow? Doesn’t it defeat the purpose of using an agent?**
The overhead is front-loaded — writing the directive takes time that prevents hours of debugging agent work afterward. At scale across multiple projects, the ratio improves significantly. The comparison isn’t “directive vs. no directive”; it’s “directive vs. rework.”
**Does every task need a full directive?**
No. The overhead scales with the risk. A one-line bug fix doesn’t need a six-section directive. A system-level implementation does. Use judgment — the formal structure is for anything where scope departure or silent failure would cost significant time to discover.
**What about agents that have read access to the whole codebase?**
Read access doesn’t change the structure — it changes what §0 needs to specify. A well-scoped directive explicitly names which files the agent is allowed to modify and which are read-only. An agent with read access to the whole codebase and no scope constraint will use the whole codebase.
**What if the agent consistently departs from the directive?**
That’s diagnostic information. A well-written directive that the agent consistently misinterprets is either ambiguous or asking for something the agent can’t reliably produce. Both cases resolve through the directive — either rewrite it or break the scope smaller.
## The Methodology As a Product
The full system — directive templates, ADR format, proof standard protocol, agent verification taxonomy, and the architectural patterns that emerged from two years of production use — is packaged at $29:
**[How I Actually Build With AI](https://rfdit.gumroad.com)** — The spec-driven development methodology for developers who use AI coding agents and want verifiable output rather than confident-sounding output.
If you’re using agents at scale and want the methodology applied to your specific codebase or workflow, the intake form is at rfditservices.com/intake.html.
Leave a Reply