The harness matters more than the model
Agentic and testing harnesses are the control plane behind AI coding — and where token spend actually diverges between tools.
“Harness” in AI coding is not the same thing as a Jest or Playwright suite — though it often includes one.
Two layers show up everywhere serious teams ship agent-written code:
- Agentic harness — the control plane around the model: file access, terminal, memory, sub-agents, and guardrails. SWE-bench is the canonical example: an isolated environment where an agent’s fix is scored objectively, not by how convincing the chat sounded.
- Testing harness — the feedback loop that runs generated code, captures failures (logs, stack traces, test output), and feeds structured errors back to the agent until the loop passes or a human steps in.
Optimizing the harness beats tweaking the system prompt. That is the practical takeaway from agentic harness engineering: the wrapper shapes outcomes more than the base model name on the invoice.
Why feedback loops beat bigger context windows
The expensive failure mode is not one bad generation — it is death by accumulation: fix A breaks B, the chat history swells, every retry re-sends the whole thread, and token burn compounds while quality drifts.
A well-designed harness short-circuits that:
- Backend: seed data, run integration tests, parse SQL or ORM errors down to the failing line — not the full server log.
- Frontend: intent-based checks (did the user reach checkout?) instead of brittle DOM selectors the agent hallucinated.
- Mobile / desktop: visual or simulator feedback when text logs would force guesswork.
The shift is from stuffing src/ into context toward multi-agent orchestration: one agent plans and budgets tokens, another writes, a third owns test specs, and a parser agent turns raw stderr into a small structured payload for the next pass.
Same model, different bill
Cursor, Claude Code, and agent-first desktops like Antigravity can all sit on similar foundation models. Token divergence usually comes from workflow design, not model tier:
| Pattern | What the harness optimizes for | Token risk |
|---|---|---|
| IDE chat (human-in-the-loop) | Fast iteration on small edits; context streams with every click | High — chat history accumulates |
| CLI / delegated agent | Plan once, execute in terminal, read stdout/stderr directly | Lower for multi-file sweeps |
| Unified orchestration | Sub-agents on cheaper tiers, compaction between loops | Flexible — if you enforce budgets |
The non-obvious insight: you are not paying for “Claude” or “GPT” in isolation. You are paying for how aggressively the tool retains, summarizes, or discards context between turns.
A token budget I would actually enforce
These are team-level guardrails, not magic savings guarantees:
- Tiered models — reserve the largest model for planning and architecture calls. Parsing logs, generating boilerplate tests, and syntax linting belong on smaller/cheaper models.
- Sub-agent ceilings — cap tokens and iterations per loop. When the agent is spinning, stop and reset context instead of letting the thread grow.
- Structured context — pull the function or module you need (search, graph,
@file) instead of dumping whole trees. Context engineering replaced prompt engineering for a reason. - Route by task shape — multi-file or architectural change → delegate to an autonomous agent with compaction. Single-file UI tweak → stay in the IDE chat.
On this site I work mostly through Cursor Agent with a human on direction — see How this site shipped with AI for the receipts story. The harness lesson still applies: my loop is brief → build → review → correct, with pointed fixes instead of endless chat. The tool provides the agentic shell; I provide the stop rules.
What I am not claiming
- Harness design is not a substitute for code review or production observability.
- Vendor marketing about “5× cheaper” depends on task mix — measure your own loops.
- A testing harness that auto-merges on green is how incidents start. Pass/fail is input to a human or a gated pipeline, not the finish line.
If you take one thing away: when AI coding feels expensive or flaky, inspect the control plane first — context retention, error shaping, agent boundaries — before swapping models.