Four Slots Left

At 06:38 on the last morning of the contest I had four publish slots, a hard deadline five hours out, and a publisher loop that had been reporting "no unpublished supply" every five minutes for six hours.

The loop was telling the truth. Everything staged had already gone out. What it could not see was the twelve finished experiments sitting in directories it had never been told to look at, and two published logbooks that were scoring zero for reasons that had nothing to do with their science. A queue that only counts what it already knows about reads exactly like an empty queue.

That last morning is a decent summary of the whole ten days.

Ten days, entered late

Hugging Face and alphaXiv ran a reproduction contest through July. Point a coding agent at an ICML 2026 submission, check its claims, publish a public logbook, and an automated judge scores every claim as verified, falsified, toy or inconclusive. I wrote up what it is and why it is unusual two weeks ago, mid-contest, and ended it by saying the interesting question was empirical and there would be enough data in a week to start answering it. There is. This is that piece.

I entered on July 23 with ten days left. The leader was already past a thousand points and finished near 3,900.¹1Final board: ProCreations 3,929, ai-sherpa 3,863, SabaPivot 3,395. Points are per claim, so a farm running 400 papers accumulates faster than any hand-checked lane can. The top of that board is automated farms running hundreds of papers, and there was never a version of this where I caught them, so I aimed at the three $500 quality awards instead and let the leaderboard be a byproduct.

The final number was 732 points across 154 judged logbooks, 851 individual claim verdicts, 17th of 370 entrants. All three award entries went in on time.

732

Points

17/370

Final rank

154

Logbooks judged

262

Verdicts worth 0

Verified

12514.7%

2 pts each

Falsified

182.1%

2 pts each

Toy

44652.4%

1 pt each

Inconclusive

26230.8%

0 pts each

Table view

Verdict	Claims	Share	Points each	Points
Verified	125	14.7%	2	250
Falsified	18	2.1%	2	36
Toy	446	52.4%	1	446
Inconclusive	262	30.8%	0	0
Total	851	100%	—	732

The useful part is not the total. It is which of those 851 verdicts were worth anything, and why so many were not.

The machine

Two lanes, run deliberately differently.

The volume lane was Vulcan, an always-on agent on a DGX box in the house, working from a fixed runbook on free local compute. It stages, it never publishes. That rule existed because the output of this pipeline is a public accusation about somebody else's work, and no unattended process of mine gets to make one.

The other lane was me and Claude Code, on the parts the first lane could not do: which papers to attempt, what the claim actually says, and every publish decision.

The only hard constraint in the entire contest is that Hugging Face lets an account create 20 new pages a day and the organizers said flatly they could not lift it. Auditing is free and uncapped. Publishing is not.³3154 published logbooks in ten days against a ceiling of 20 new pages a day. The cap counts creating a page, not editing one already up, which is why the August 1 correction cost no slot and every new logbook cost one. So the shape of the operation is to bank work continuously and spend twenty slots in a single pass, which is why the final morning came down to four.

Vulcan also spent its first forty minutes producing nothing while its status posts looked like progress, because they were box-health heartbeats rather than work. It was not idle. It was structurally blocked, missing three tools its runbook assumed it had inside its own sandbox.²2No orx, no read access to the anchored-claims file, no trackio. All three were present in my environment and absent in its sandbox, which is why the runbook read as complete when I wrote it. It published within seven minutes of being unblocked. Briefing an agent is not equipping it, and the check is to run its first three commands as that agent before handing it the runbook.

The night the agents started accusing people

On July 25 four audit agents came back with a falsification on four different papers. Three were wrong.

I told that story mid-contest and the short version holds: each one arrived with a number, a line reference and reasoning that survived a first read, and each had accused a paper of something the paper never said. One had proved a mangled OCR of a formula false. One had found a difference smaller than its own rounding error.

The third is the one worth adding, because it is specific to how this contest is built and it will recur in anything similar. The claims you are scored against are anchored to the camera-ready paper, which is not on arXiv and sits behind an OpenReview wall. One paper's claim list cites a Theorem 3.4. My copy numbers its results 3.1, 3.3 and 3.5. The agent could not find the theorem, so it reported the claim as unsupported.

A missing theorem number is evidence of a version gap. It is never evidence that a claim is false. But an agent holding one document and a list of assertions about a different document has no way to tell those apart, and it will resolve the ambiguity in whichever direction its prompt leans.

The rule that came out of it fits on a line and went straight into the agent spec: does the paper disagree with itself, or does the claim disagree with your copy of the paper? Only the first one is a falsification.

The one that was real

For contrast, the entry that became the falsification submission.

The last-iterate-prox paper types its Theorem 4.4 with a constant step size. Its own proof, thousands of lines further down the same source file, opens by assuming that step size depends on the length of the run. Those are two different theorems, and the experiment separates them cleanly: under the typeset step the error plateaus at roughly 0.15 across horizons from 200 to 100,000, and under the horizon-dependent one it decays from 0.0129 to 0.00095.

A second claim in the same paper attributes a bounded-variance assumption to a cited work. The paper says the opposite at four separate places in its own text, including a comparison table with a check mark in the column headed "unbounded variance," and again in its conclusion.

Twelve other participants reproduced that paper. Seven marked the claim verified, five called it inconclusive, including the contest's own reference logbook.⁴4Counted off the contest verdicts dataset: 13 judged logbooks on that paper, 7 verified, 5 inconclusive, 1 falsified. Mine is the falsification. Nobody else caught it, and it is sitting in print.

The one I gave back

The entry I am most confident in is a retraction.

On July 30 the judge accepted my falsification of Claim 2 on a paper about adversarial attacks in high-dimensional linear classifiers. Two points, banked, scored, done. Then I ran an independent clean-room agent in a container with no access to any of my earlier work, and it reproduced my mathematics exactly and told me the label was in the wrong place.

Claim 2 is about a particular dependence between two quantities, and that dependence survives untouched. Its own negative control settles it: the corrected constant lands within 0.4σ of simulation, the classical form is rejected at about 30σ. The defect was real but sat one claim over, where the paper asserts an exact characterisation and the printed indicator is off by exactly 2.00000 on one of its two metrics.

So I moved the falsification and gave up an accepted one to be right about where the error was.side quest That was the second time the clean room overturned my own result.

Back to the last morning

The two logbooks scoring zero on August 3 were both scoring zero for parsing reasons.

One used PASS and FAIL vocabulary. The logbook builder accepts exactly four words, so it rendered blank verdicts on all six claims. The other had a section header, ## C2 formulation: resolved, that matched the parser's claim regex, so four separate anchored claims arrived at the judge as one blank claim.

The work inside both was strong. Exact identities to machine precision, a negative control on every claim, and completely invisible. I found them by running the builder's own parser and asking it what the judge would see, instead of grepping the file and trusting my eyes. Ninety seconds, nine points, on the last morning available.

That failure mode has nothing to do with reproducing papers. Any pipeline whose output is consumed by another program has a version of it, and the fix is always the same: read the artifact back through the consumer's parser, not through yours.

Why 18 of my logbooks scored zero

The real autopsy is bigger than two parsing bugs.

Of 154 judged logbooks, 18 scored zero, and every one of them came back uniformly inconclusive across every claim.⁵5Not approximately uniform. All 18, every claim, no exceptions, which is what took it out of the range of coincidence. The field graded 6,855 logbooks in total across 370 entrants. Uniformity like that is a structural signature, not eighteen independent scientific failures. The judge's stored evidence on one of them says so directly:

The logbook evaluates the formula d*ln(n/d)/n at different parameter values and checks scaling ratios, but this is circular ... computing a formula's properties from the formula itself. No distribution was constructed, no halfspace learning experiment was run, and no empirical population error was measured.

Evaluating a paper's formula is not reproducing the paper's claim, and the judge caught it every time.side quest My fast recipe was largely formula evaluation, which is why 262 of my 851 claim verdicts came back inconclusive and worth nothing. The recipe was built for throughput and it delivered throughput. No quantity of it was ever going to score.

Strip out the papers and this is a general fact about automated checking. An agent that checks a formula against itself produces output that looks exactly like verification. It is fluent, it cites the source, it reports a ratio, and it is worth zero. The only thing separating a sound audit from a circular one is whether the experiment could have come out otherwise.

Does it work

Two weeks ago I said the contest paired a known-weak instrument with an unusually strong correction loop, and that whether the loop outran the instrument was an empirical question.

From inside, the loop won on the thing I could measure. The judge was not fooled by circular checking, not once across 18 attempts at it, and its written reasons were specific enough to act on. It accepted a falsification that was wrong in a way no automated scorer could have caught, which a human then withdrew. Both halves of that are the design working as intended.

What the run does not support is any claim about the papers. My 154 logbooks include zero conclusions about the state of the ICML literature, because the instrument that produced them was tuned for throughput and I can now name exactly why 262 of its verdicts were empty.

If I run it again the order changes. Screen for claims that can actually be checked before picking papers by how interesting they are. Ask the consumer's parser what it sees before publishing anything. Make every audit run a mechanism that could fail rather than a formula that cannot. And keep the human pointed at the falsifications specifically, because a wrong reproduction costs two points and a wrong accusation costs a stranger something they cannot get back.

Logbooks are at huggingface.co/RyeCatcher.

Related reading on this site: Maximizing Toward the Local Minimum for the same failure in a different engine, where a green dashboard hid hundreds of runs of one experiment, Reproducing Warp Decode on BlackwellshippedAI Systems & Architecture·Apr 9, 2026reproducing-warp-decode-blackwellFirst independent reproduction of Cursor's Warp Decode on Blackwell. 38% faster Gemma 4, crossover at batch 24 vs vLLM, fusion failure analysis. for what an honest single-paper reproduction costs when you do it slowly, Three Sessions, One Company for why the coordination cost of parallel agent lanes relocates rather than disappears, and Acquired Before Benchmarked for claims that got accepted well ahead of anyone checking them.