Four Upgrades to the Agent Loop. None of Them Moved the Number.

Dan Luu gave a coding agent twenty-six different instructions about how to verify its own work. Test-driven development. Property-based testing. Fuzzing, mutation testing, metamorphic testing, differential testing. ACL2, Alloy, Kani, Lean 4, Spin, TLA+, Verus, Creusot, and three SMT solvers with Z3, cvc5 and Yices all installed and available. "Audit first." "Make no mistakes." Eighty runs per condition at each of two effort levels, hidden tests, and his predictions written down before anything ran.

Default scored above average. That's the condition with no added instructions.

That went up on 7 September. It's the fourth result in a month that lands the same way, and the other three sit in different parts of the loop: a cache eviction policy, a critic model, a code-search backend. Four teams upgraded one component of an agent system and got nothing for it.

Two things before the evidence, because both of them discount it. These four surfaced together in one research pass built to find agent-loop stories from the same few weeks, so they were selected partly because they agree, and I have no idea what the population of component upgrades that never got written up looks like. And they are not the same grade: one peer-reviewed paper, one preprint with no confidence intervals anywhere in it, one four-day-old single-author repo, and one self-described preliminary pilot. Solid describes the direction here, not the rigour.

The null is still the strong half. I went looking for the matching positive result and exactly one of the four studies ran the arm that would show it.

Twenty-six ways to say verify

The task was a Zstd implementation in Rust, reused from his earlier evaluation so the baseline was already established.¹1Dan Luu, "Testing agentic coding with a variety of testing techniques", 7 September 2026. The agent is codex on GPT-5.6 Sol at medium and xhigh effort. 26 prompt conditions plus 4 skill conditions, roughly 80 runs per condition per effort level, so the "out of 160" counts below are both efforts pooled. Twenty-six prompt conditions, plus four packaged skills: the official Hegel skill, the Trail of Bits property-testing skill, a Rust testing skill from a collection with 250k GitHub stars, and one he wrote himself in about two minutes.

Nothing wildly outperformed. His summary of it: "if you ask agents to use a particular test technique or test library, this approach doesn't change as much as you'd hope."

The failure mode is more useful than the ranking. Agents mostly wrote their usual unit tests inside the new framework's skin. Under the differential-testing instruction, 135 of 160 runs did something you could call differential testing, and it was almost all trivial, because the agent implemented the same buggy logic twice and compared it against itself. The formal-methods conditions produced proofs of properties that were irrelevant or vacuous. Under TDD, agents had a failing test before substantial implementation in 67 of 160 runs, against 0 of 160 for Default, so the instruction did change behaviour. It just didn't buy correctness.

The skills cost real money for it. Hegel's skill was invoked in 157 of 160 runs, ran 26% more expensive at medium effort and 41% at xhigh, and came out no better on correctness.

Two limits ride with this, and the first one is his. He prints no confidence intervals anywhere in the post. The chart carries bootstrap uncertainty on hover and the prose never restates it as a number, so there is no error bar to quote for any condition. Second, this is one task family in one language with a spot-check on an IMAP RFC parser, and the four skills tested were Hegel's official one, Trail of Bits', a Rust testing skill from a 250k-star collection, plus one he wrote himself in about two minutes. A small and non-random set.

One thing did work, in the ten runs out of 160 where it happened at all. When agents generated random structured inputs under the fuzzing condition, they found real bugs half the time. Several were non-trivial. His read is that the capability is present and the default behaviour doesn't reach for it. That's a training-distribution problem, not an instruction-wording problem, and no addendum in the set fixed it.

The critic can be 0.6B. The refiner can't.

Seventeen days earlier, a group at UC Irvine and Drexel published the first stage-wise model-size sweep of the generate, critique, revise pipeline.²2Zhuoyi Yang and others, Asymmetric Capacity Allocation in Self-Refinement Pipelines, arXiv 2608.21345, submitted 21 August 2026. Qwen3 at six sizes from 0.6B to 32B and Gemma 3 at four sizes from 1B to 27B, across five benchmarks. The paper reports ranges and standard deviations across each sweep, not confidence intervals or significance tests. Vary one stage's model size, pin the other two, repeat for all three seats.

The critic seat barely registers. Its standard deviation across the whole sweep stays below 3.1 percentage points on every benchmark and below 0.21 on one of them. The refiner seat reaches 20.60 points on the code-optimisation benchmark, against 1.63 for the critic on that same task. Across all ten benchmark-and-family combinations, the critic is the lowest-variance seat in every single row.

So a 0.6B critic and a 32B critic produce roughly the same pipeline. The obvious conclusion is that critique is theatre, and the paper's second finding kills it: even the 0.6B critic beats running no critique at all, by +3.71 to +11.06 points depending on the benchmark. Critique earns its seat. Paying for a better one doesn't.

The mechanism is where this gets interesting for anyone wiring a loop. The authors hand-scored 50 critiques from each size on a five-point rubric and found the big critic isn't producing better feedback, it's producing more detections. The refiner then fails to convert the extra detections into better revisions. So the bottleneck sits one stage downstream of the thing everyone upgrades.

Then the part that should change what you build. Twelve of the thirty refiner configurations scored worse than the unrefined generation they started from. The authors hand-analysed 50 of those degradation events from the worst cell, a 32B generator and 32B critic feeding a 0.6B refiner. In 41 of the 50, the critique was not misleading. The small refiner had correct input, correct feedback, and edited correct content anyway. In the 9 cases where the critique was misleading, the 32B refiner ignored the bad advice and the 0.6B refiner obeyed it.

Robustness to imperfect feedback looks like a refiner property. In the one cell they hand-checked it in, you could not buy it in the critic seat at any size.

The scope is honest and it is narrow: a single refinement iteration, open-weight models at 32B and below, no retrieval, no tool use, no multi-round loops. The authors say so themselves in the limitations. Treat it as an allocation rule for the workhorse tier, not a law.

The most confident idea was the worst one

On 10 September someone published a reproducibility study of the agentic KV-cache eviction literature and failed to beat the baseline three times.³3github.com/gauravapiscean/agentic-kv-cache, both commits dated 10 September 2026. Single author, unreviewed, 22 stars at the time I read it. Reproduces from a cold checkout with make setup data repro and ships its committed results/ so the tables check without downloading the traces. The baseline is radix-leaf LRU, which is what vLLM and SGLang actually run.

The three attempts, in increasing order of mechanism complexity: a Bayesian hazard estimator replacing recency with P(session returns); that plus a physically modelled recompute-cost term, since recomputing tokens late in a chain costs more per byte than tokens near the start; and that plus coherent session-granularity eviction instead of dropping the N globally-oldest leaves.

Every one lost at every cache size. On effective recompute cost against LRU-leaf: -3.2%, -38.9%, -81.0% at 8,000 blocks, and -4.5%, -90.4%, -207.8% at 20,000. The ordering is the finding. Each mechanism he added made it worse than the one before.

A five-minute TTL policy produced byte-identical results to plain LRU in every single run. Not similar. Identical, because LRU always evicted first and the timer never once fired.

That's the whole diagnosis. Under capacity pressure the waste isn't sessions idling past a timeout, it's tight tool loops: 33.1% of recompute tokens arrive less than ten seconds after the previous request, and only 17.5% come from gaps longer than five minutes. At a median inter-request gap of 2.1 seconds, a liveness predictor has nothing to discriminate on. The policies weren't badly designed. They were designed for the provider-side regime, where per-customer capacity is effectively unlimited and entries die on a timer, and then deployed in the capacity-bound regime, where the two want opposite optimisations.

His own summary: "I went in assuming the liveness framing and it cost me three failed policies. Establishing which regime you're in first would have saved all of it."

There's a free gift buried in the methodology section and it's awesome. In an early run, Belady's algorithm lost to LRU. Belady is the offline optimal, so that result is impossible and it means the simulator is broken. The cause was missing refcount pins, which let a non-recency policy evict blocks in the chain it was currently building. LRU happens to be immune to that by construction, so every policy comparison comes out silently wrong in LRU's favour. His advice: make "does Belady beat LRU?" the first test you run.

This is pure simulation, and he says so in his own limitations: it answers what should stay in cache, not what happens to throughput, latency or SLOs under real load. Single author, four days old when I read it, nobody has replicated it. He also reports a +4 to +6 percentage point offset against Mooncake's published curve that he could not explain after testing five metric definitions, and left it in the README rather than tuning it away. I'd rather read that than a clean number.

The backend was right and it didn't matter

The fourth one is the only study in the set that ran the comparison I actually wanted.⁴4Pengcheng Xu, "Grep beats LSP? Why coding agents ignore your fancier tools", 12 August 2026, with the underlying study at agentconnect-md/lsp-vs-grep-token-study. Opus 4.8, Sonnet 4.6 and Haiku 4.5 across several Python and TypeScript repos. The author calls it a preliminary pilot with two to three rollouts per cell and I am taking him at his word on that.

Start with what the better backend bought. On find-every-caller tasks, LSP-backed navigation hit 1.00 precision against grep's 0.76. That is a real capability difference and it isn't an artifact: semantic references exclude the string matches grep can't tell from real ones.

Recall stayed near 0.66 in both arms.

Precision went up, recall didn't move, and the binding constraint on "find all the callers" turned out to be how thoroughly the agent worked rather than how precise its tool was. On the stronger models the precision gain arrived with higher token use. Given a free choice on simple localisation tasks, the agents picked the semantic tool between 0% and 6% of the time, and forcing them down the semantic path first dropped task success from 100 down to 89 percent.

Then the arm that moves. Same pyright backend. Same underlying set of references. The only change was the response shape: instead of returning bare file, line and column objects, the tool returned the same locations with two lines of inline source around each, matching what grep already prints.

Multi-file rename, Opus 4.8	pass@1	Follow-up file reads	Tokens
grep	1.00	4.3	2,451
LSP, locations only	0.67	15.2	4,131
LSP, same results plus inline context	0.83	3.2	3,336

Pass@1 went from 0.67 to 0.83, on the order of two to three runs per cell, so a single flipped run moves that by ten points or more. Follow-up file reads fell from 15.2 per episode to 3.2, which is below grep's own 4.3. Tokens went down, not up. A semantically correct tool returning bare coordinates forces an extra read action for every result it returns, and the agent pays for that in a currency the retrieval benchmark never measured.

The author refuses the headline his post got summarised into, and he's right to: "These results are conditional rather than categorical." Grep doesn't beat LSP. The two-to-three rollouts per cell can't carry a categorical claim about anything, and on noisy repositories the LSP arm won outright. What the pilot does support is narrower and more useful, which is that holding a backend fixed and changing only what it hands back moved more than swapping the backend did.

Four studies, two arms each

Each of these upgraded a part. One of them also held the part fixed and changed what it hands back. Step through and the tally builds itself.

What the four have in common

Every one of these teams treated a component as the unit of improvement. Swap in a better critic. A smarter eviction policy. A semantically correct retriever. A better-worded instruction. Four subsystems, four teams who don't read each other's work, and the component swap came back flat every time.

Exactly one of them ran the other arm.

That's the retrieval study, and there the other arm is literally the wire format. Same pyright backend, same set of references, and the choice is whether each one arrives as a coordinate or as a coordinate with its line attached. The agent that gets coordinates has to go read the file. The agent that gets the line does the work. It's the only controlled interface comparison in the set and it's a pilot with two to three rollouts per cell.

The refinement paper points the same direction and did not test it, and I want to be exact about which half is theirs. Theirs is the mechanism analysis: the 32B critic isn't producing better feedback, it's producing more detections, and the small refiner converts the extra detections into damage. Mine is the reading that this makes it an interface result. Every condition they ran varied model size at a fixed seat. Nobody held the sizes fixed and changed what the critique contains, so there is no arm in that paper separating the signal from its consumer, and the version of their finding that survives without my gloss is narrower: one seat in the pipeline matters and another one doesn't.

The cache study stops in the same place. Its diagnosis is a mismatch between a policy and the regime it's deployed in, which is a question about the cache's environment rather than the cache's cleverness, and he found the mismatch and stopped there. No matched-regime arm.

Dan Luu has no such arm at all. What his data supports is narrower than the story I'd like to tell with it: correctness tracked which instructions the agent actually followed, and the one condition that changed outcomes when it landed was fuzzing, followed in 10 of 160 runs.

So the ledger reads four of four on the null and one of four on the alternative. I'd rather publish that than the version where it's four for four both ways.

Two more results from the same window point the same direction without being part of the four. The ICML paper on multi-agent teams isolates the exchange step directly: when agents read each other's complete candidate solutions, mean pairwise distance between their solutions falls from 0.315 to 0.229 after one round, and the only configuration whose gains survive model diversity is the one where proposers never see each other's outputs.⁵5Summer Eunhyung Ann, Haokun Liu and Chenhao Tan, The Interaction Tax: When Communication Erases Diversity in Multi-Agent Teams, ICML 2026. In their 2x2 factorial the diversity coefficient is +0.188 with a 95% CI of [0.073, 0.299], while the synthesis-method coefficient sits near zero. The authors flag that removing one task drops the diversity coefficient to +0.014 with the interval crossing zero, so the headline is task-dependent and they say so. More agents didn't help. What they were allowed to see did.

And Armature ran 16,893 sandboxed sessions watching Claude Code, Codex and Cursor pick third-party services for the same ask, and got agreement in 42% of categories.⁶6Armature, "Which tools do Claude Code, Codex and Cursor choose?", 3 September 2026. 5,292 of the 16,893 sessions passed validity checks and were published. Note what "tool" means here: third-party vendors and services, not CLIs or extensions. Armature discloses at the top of the post that it sells growth services to dev tools, which is a direct commercial interest in the outcome it's measuring, so treat the per-vendor shares as directional. Codex searched the web in 94% of sessions; Claude Code in about 30%, browsing three times more pages when it did. Same ask, same repo class, different harness, different answer. That isn't a model-quality finding either.

I have been shipping the null result for a year

My own setup is built almost entirely out of the intervention Dan Luu just measured. Standing instructions in config files telling the assistant to verify the artifact instead of the exit code, to run the control in both directions, to check the file before believing the summary. Written in prose, loaded every session, addressed to a model.

One of them asked it to log every correction I gave about writing, so the corrections would accumulate into something I could tune against. Over forty-five days I gave forty-one. Ten reached the ledger.

That's a 24% capture rate on an instruction sitting in always-on context, which is roughly Dan Luu's shape at n=1 on my own stack: the instruction was read, it was even followed sometimes, and it didn't move the number I cared about.

The fix wasn't a better instruction. On 11 September that one moved into a hook on UserPromptSubmit, which fires on every prompt I send and writes the row itself. Before it, the ledger held 9 rows spread across six months. In the three days after, it took 26 more, of which 18 were a manual backfill of one session and roughly 8 are the hook's own capture.

The general form is the same one the four studies keep finding sideways. An instruction is a component: it sits in the loop, it gets read, and whether it fires is the model's call. A hook is an interface: it sits between the turns, and whether it fires is not a judgement call at all. Anthropic's own guidance on Opus 4.8 has the sharp edge of this, which is that a filtering instruction like "only report high-severity issues" can leave the reasoning intact while the findings get judged below the stated bar and dropped, so recall falls and the output looks cleaner and more confident. The instruction was obeyed. That was the problem side quest.

The experiment nobody ran

None of the four set out to test components against interfaces. One found it sideways on a rename benchmark, one found a regime mismatch and stopped, and two never went near it.

So this is four measurements pointing at a claim that one of them tested, selected by a pass that was looking for agreement, and that's when I should be slowest to take it. The clean version is one study, one loop, one task set, three arms: upgrade the component, change what it hands downstream, change neither. Nobody has run it. The nearest thing is a pilot at two to three rollouts per cell.

What I can act on before that exists is narrower, and it doesn't need the general claim to be true. Every tool in my loop returns something, and I have never once measured how many follow-up reads each tool's response provokes. In the one place anyone looked, that number was 15.2 per episode, and getting it to 3.2 cost a change to the response format and nothing else.

---

Related reading on this site: 62.7 and 99.9 Are the Same Model for the positive version of this, where the same model under two harnesses swings 45 points and the reasoning knob stops mattering, I checked the 90% token cut against 30 days of my own Claude Code for the longer version of the hook-binds, note-suggests distinction against my own session data, and The Composition ThresholdshippedAI Systems & Architecture·Jun 21, 2026The Composition Threshold: When a Bigger Skill Library Starts to Mean a Better AgentA repo doing autonomous research on 900 markdown skills looks like more-is-better hype. Open it and the count turns out to be a symptom of strict composition, not a substitute for it. This is where a skill library stops being a liability and starts being a capability, and what that says about agents that grow their own skills. for a repo where adding more components looked like the whole story and the composition rules underneath turned out to be it.