The Second Harness Tax

In June I argued that the system you build around a model is the part of your stack nobody can copy. I called it a moat and I meant it.

This month a team at UC Berkeley put a price on the general case, and the number is not flattering to anyone who has been accumulating scaffolding for two years.

I rebuilt their experiment with my own configuration loaded. I can now measure exactly what it costs. I cannot show you a single resolved task it bought.

What HarnessTax actually ran

HarnessTax is the fixed-model harness comparison nobody had published.¹1Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia, "HarnessTax: How Much Does the Harness Matter for Coding Agents?", UC Berkeley Sky Computing Lab and Arena Intelligence, September 2026. Zaharia is also a co-author on the Databricks benchmark cited later, which is how the two connect. Twenty-one model-and-harness pairs. Seven models, each run through Claude Code, Codex CLI and Pi. Thirty SWE-bench Lite instances and thirty Terminal-Bench 2.0 tasks, three repetitions per pair, capped at 100 agent turns, each harness in its native configuration at its high effort setting. Success comes from each benchmark's official evaluator, never from a model judge. Costs are computed against one fixed price list dated 1 September 2026, applied identically across harnesses, so the money is comparable rather than merely reported.

Their headline is one sentence: "Harness choice has little effect on task success rate, but can significantly affect the cost."

The Claude Fable 5 row is the one to hold onto. 96.7% in Pi at $0.666 per attempt. 97.8% in Claude Code at $1.329. Same model, same tasks, 1.1 points, twice the bill. Averaged over every model they could run in all three harnesses, Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and 1.5x Pi on Terminal-Bench 2.0, by geometric mean of cost ratios.

Turn counts do not explain it. Pi and Claude Code averaged 15.4 and 15.3 turns per attempt on the same tasks.

Berkeley put the average harness effect within about two points on SWE-bench Lite and about five on Terminal-Bench 2.0, while the cost gaps run to 5x. Their confidence intervals show the same shape: Pi's 96.7% carries 91.1 to 100.0 and Claude Code's 97.8% carries 93.3 to 100.0, while the cost intervals, $0.494 to $0.875 against $1.101 to $1.601, do not touch.²2Overlapping intervals are not a test of equivalence, and I am not claiming one. The effect-size statement is Berkeley's; the intervals are here because they show the asymmetry between the two columns at a glance. Intervals are bootstrap, 10,000 resamples of 30 task means.

Pick a model and watch it hold:

One more finding, and it is the one a vendor would least enjoy. Across the six Anthropic and OpenAI models and both benchmarks, an alternative harness achieved the highest observed success rate in nine of twelve comparisons. Your Claude models may not need Claude Code.

Where the money goes

The most useful figure in the study is the one that explains the bill rather than reporting it.

Pi declares four tools. Read, write, edit, bash. That is the entire surface and it reaches the Pareto frontier on both benchmarks.

Claude Code declares 23, and describing them costs 76,995 characters before the model sees one line of the problem.

The third panel is the one worth sitting with. Claude Code's standing instructions are shorter than Codex's, 13,465 characters against 23,502.³3The dispersion is not symmetric either: Claude Code's standing-instruction length has a standard deviation of 9,444 characters against Codex's 1,042. Claude Code's instruction payload varies a great deal across attempts and Codex's barely moves. Both figures are means over 630 attempts. The vendor's weight is not its prose. It is the tool cupboard.

Net result on the first call: 27,011 tokens for Claude Code against 1,972 for Pi. Thirteen times, paid on every turn, before any work happens.

The toy-benchmark objection does not survive

The obvious defence is that SWE-bench Lite and Terminal-Bench are small, self-contained, public, and probably in the training data. Berkeley say the last part themselves.

Databricks had already closed that door. In July their engineering team published an internal benchmark built from real pull requests their own engineers merged into a multi-million-line codebase spanning Scala, Go, Rust, TypeScript, Python, Bazel and protobuf. They held back the tests, reviewed every task by hand, and refused an LLM judge because it "rewards sounding right over being right."

Same answer. Running one model at one effort through two harnesses changed cost by more than 2x while quality stayed flat, and Pi sent about three times less context per turn.

What I ran

Six SWE-bench Lite instances, one repetition, nine arms.

Seven of the nine ran on glm-5.3-flash through a single LiteLLM gateway deployment, so the model, the route and the sampling parameters were fixed and the only thing varying was the rig around them. The remaining two arms ran Claude Code on Sonnet 5 on native auth, as a separate check that the pattern was not an artifact of a cheap model; they carry no token accounting and contribute to nothing below except the resolved column.

Grading ran the official swebench 5.0.2 evaluator, which executes each repository's real pytest suite inside the instance container. Nothing was graded by a model. Before believing any of it I ran the control in both directions: the gold patch resolves, an empty patch does not.⁴4This is the cheapest control in benchmarking and the one most often skipped. An evaluator that cannot fail certifies whatever you point it at. Running the empty patch takes minutes and tells you whether your harness plumbing is actually reaching the grader.

arm	first-call context	input tokens, 6 tasks	resolved
OpenCode, vanilla	6,855	1,194,440	6/6
Prime Agent, vanilla	2,861	1,242,211	5/6
Pi, upstream	3,246	1,824,788	4/6
OpenCode, my config	22,071	2,481,127	5/6
Claude Code, vanilla	17,519	3,997,303	5/6
Prime Agent, my config	23,234	4,849,913	4/6
Claude Code, my config	54,263	5,929,000	5/6
Claude Code vanilla, Sonnet 5	n/a	n/a	5/6
Claude Code my config, Sonnet 5	n/a	n/a	4/6

The replication first. Claude Code out of the box spent 2.19x the input tokens that upstream Pi did. Berkeley report 2.0x. Those agree in direction and magnitude, and they are not the same statistic: theirs is a geometric mean of dollar-cost ratios across seven models on thirty tasks, mine is input-token count for one model on six. Worth reporting, not worth over-reading.

The context ratio did not replicate as cleanly. I measured 5.4x on the first call where Berkeley report over 10x. Their figure averages seven models and includes the task prompt; mine is one cheap model on one easy instance. Theirs is the one to cite.

The second tax

Then I ran the three harnesses I actually use with two years of my own configuration loaded in.

On Prime Agent, my configuration is 8.1x heavier than the harness it sits inside. Across the full six-task run the multiplier settles between 1.48x and 3.90x, because first-call weight amortises over a long trajectory.

The row that bothers me is the top one. Prime Agent with my config spent 4,849,913 input tokens against Claude Code vanilla's 3,997,303. I picked the lean harness, loaded it, and made it heavier than the thing I was avoiding.

In all three harnesses the configured arm resolved the same number or fewer. I am not going to turn that into a finding. With a 4/6-to-6/6 spread at one repetition it is a difference of one or two instances and it is not separable from luck. The honest statement is narrower: nowhere did the extra spend show up as a better result.

What is actually heavy in there

I assumed the 109 skills were the weight. They cost 249 tokens, all of them together.

The 22 standing-instruction files cost 31,191, which is two thirds of everything my configuration costs.

component	tokens
rules files (22)	31,191
project and machine instruction files	5,959
auto-memory index	5,098
sub-agent definitions (27)	2,049
output style	1,620
MCP servers (6)	759
skills (109)	249
slash commands (17)	82
plugin tree, 2.8 GB on disk	~0

Every component I would enjoy pruning comes to under 1,100 tokens combined.⁵5Each figure is a subtraction: a one-turn trivial prompt measured in an isolated HOME copy against a 65,856-token baseline, with one component restored at a time, read from the CLI's own usage JSON and verified with stream-json output to be a single API call rather than a sum across turns. The 249-token figure for 109 skills is a property of Claude Code's progressive disclosure, which loads a one-line description until a skill is invoked. It is a fact about this harness, not about packaged procedures in general. Twenty-seven sub-agent definitions cost eight times what 109 skills cost, because a skill loads a description and an agent definition loads whole.

Which is the exact mirror of the vendor's problem. Claude Code's weight is its tool schemas and its prose is lean. Mine is prose, and my tool surface is a rounding error.

The instrument that lied to me for a day

Six weeks ago I published the rule that a per-component cost is a property of your configuration rather than of the component. I then walked straight into it.

Every measurement in my first pass ran through my own LiteLLM gateway, to keep the experiment free. Claude Code defers MCP tool definitions by default, and that deferral switches off under a custom ANTHROPIC_BASE_URL.⁶6Documented at code.claude.com/docs/en/mcp. Only tool names and server instructions enter context until a tool is actually called, which is why six live MCP servers cost 759 tokens on native auth. Route the same session through a proxy and every schema loads eagerly.

MCP server	native	via gateway	overstated
m365	1,658	131,828	79x
chrome-devtools	413	6,217	15x
playwright	295	4,140	14x
all six live	759	12,622	17x

I had a draft of this post telling you to go delete your MCP servers.

side quest sits underneath that one, and it is the part I find harder to shrug off.

What this could and could not tell me

The cost columns are solid. They are large, consistent, and counted by an instrument outside my own code.

The success column is not. Four out of six to six out of six across all nine arms, at one repetition, is a spread of one or two instances. I could fix that with thirty tasks at three repetitions, the way Berkeley did. It is a weekend of gateway spend and I may run it.

But it would buy a cleaner version of the same limitation, which is the one that matters. These are self-contained repair tasks that arrive fully specified, and that is not what my week looks like. My real requests are half-stated, assume things I never said, and depend on decisions made months ago. That gap is precisely what my configuration exists to fill. No benchmark I ran tests it, and the two public ones in this post are constructed in a way that rules it out: Databricks describe rewriting each PR description to state the problem and strip the explanation, because leaving in why a fix was right "makes the task too easy."

So the position is narrow. I have a real number for what my configuration costs and no number at all for what it returns on my own work. More tasks would not give me one.

What I changed

Three things, and none of them is a cull.

Vanilla by default. Sessions now open with nothing loaded and gain the one thing the job needs. Pi reaching the frontier on four tools is the argument: the burden of proof belongs to what you add.

Two rigs instead of one. A minimal profile for work that arrives fully specified, which is most of a week, and a maximal profile for work that depends on what I decided in March. I had been paying maximal rates for minimal work all day.

Measurement with a version stamp. Every component number now carries the harness version it was taken against, because the 192,000-token note proved that a number without one decays silently into a wrong decision.

Both sides of the argument agree on one thing

Berkeley close by noting that "as models become more capable, coding agents may need less of today's scaffolding."

The people betting hardest against that arrive at the same place from the opposite direction. Prime Intellect shipped Prime Agent in August on the argument that a harness should CRUD its own prompts, skills, memory and sub-agents from its own trajectory, because "static, hand-engineered sub-agents, prompts, skills, and memory are set once at design time and never adapt to what the agent learns while running."

One camp says strip the scaffolding. The other says stop hand-writing it. Neither thinks the future is a person maintaining 22 files of standing instructions, which is what I have.

The part of June I still hold

The harness is still the moat. I had the wrong idea of what the harness is.

I thought it was the pile: two years of rules and skills and memory, compounding, a thing you build. That version now has a price tag from my own measurements and nothing on the other side of the ledger.

What survives was never in the pile. Knowing this job needs the full rig and that one needs four tools and nothing else. Knowing which of my 22 files is doing real work on a Tuesday and which three I could close the laptop without. That is shaped to me, nobody can license it, and it is the one part that sharpens as the pile shrinks.

I spent two years thinking the work was building it. The work is knowing when to leave it at home.

Go get one number for yours. /context, five seconds. Then run a week on the light profile and find out what you actually miss.

---

Related reading on this site: 62.7 and 99.9 Are the Same Model for the ARC Prize run where one model under two harnesses produced a 45-point swing and the weaker harness won, Four Upgrades to the Agent Loop for four teams who each improved one component and measured no movement, Codex vs Claude Code vs OpenCodeshippedAI Development & Agents·May 12, 2026Codex, Claude Code, OpenCode: A Comparison from Inside the HarnessA working comparison of Codex, Claude Code, and OpenCode written from inside one of them. The shared floor they all ship, where they actually differ in the last 90 days of releases, why model character still matters even after the "harness is the IP" finding, and a situational-fit matrix for picking your starting harness. for what the three harnesses feel like from inside rather than on a benchmark, Inventory Your Harness for the six-component audit that tells you what you are actually carrying, and Pruning Your AI Agent Skills LibraryshippedAI Development & Agents·Jan 10, 2026Pruning Your AI Agent Skills Library: A Practical Guide to Skill ConsolidationLearn how to audit, consolidate, and optimize your AI agent's skills library using practical patterns that reduce overlap and improve reliability. for the cut I made before I knew which parts were expensive.