62.7 and 99.9 Are the Same Model

ARC Prize's two headline numbers for GPT-6 Astra on ARC-AGI-3 Semi-Private are 62.7% at $26,098 and 99.9% at $18,817. Same model, same task set, same evaluator, same week. ARC calls both of them SOTA.

Those two runs used different harnesses. They also used different reasoning effort, max against high, so the pair on its own cannot tell you which of the two did the work.¹1ARC-AGI-3 scores in Relative Human Action Efficiency (RHAE), which combines task completion with per-level action efficiency against first-time human baselines. A percentage here is not a fraction of tasks solved. ARC's methodology docs carry the definition.

ARC also ran every other effort level under both harnesses, which settles it, and the settled version is more interesting than the headline. That table is the first thing below.

The result landed on 3 September. It's one of four ARC results published since late July that all get quoted for the same claim, which is that the scaffolding around a model sets its benchmark score. I think the claim is right. I also think three of those four numbers can't carry it, and the work is sorting out which one can.

The harness effect, matched effort for effort

ARC Prize published the first third-party evaluation of Astra on 3 September. They ran six reasoning-effort levels under each of two harnesses: their Standard one, and what they call a Provider Adapter harness. Twelve configurations, one model, one task set.

Reasoning effort	Standard	Provider Adapter	Delta
max	62.7% · $26,098	98.6% · $17,332	+35.9
xhigh	59.3% · $37,317	98.4% · $18,147	+39.1
high	54.8% · $40,705	99.9% · $18,817	+45.1
medium	38.6% · $48,090	98.4% · $19,285	+59.8
low	17.5% · $38,166	98.0% · $21,298	+80.5
none	35.2% · $49,791	96.7% · $23,457	+61.5

Read the rows and the harness effect survives the control: +35.9 points at max, +45.1 at high, and it never falls below +35.9 anywhere in the table.

The columns carry something better. Reasoning effort moves the Standard harness 45.2 points, from 17.5 up to 62.7. It moves the Provider Adapter 3.2 points, from 96.7 to 99.9.²2The Standard column is not even monotonic in effort: low scores 17.5% while none scores 35.2%. Whatever the harness is doing to the model at low effort, it is worse than giving it no reasoning budget at all. ARC does not explain it and I would not want to guess. The adapter does not only raise the score. It makes the reasoning knob nearly stop mattering, which is a different and more useful claim than anything in the headline.

It is also cheaper in all six rows, against a Standard column that runs up to $49,791. Cheaper, at every effort level, while scoring higher.

The adapter, in ARC's own words, "preserves opaque reasoning state between requests and uses compaction for longer conversations, allowing the model to reuse prior work."

Speed and tokens moved with it. Across the 167 game-and-effort pairs both harnesses solved, the adapter runs were roughly 3.66x faster by aggregate elapsed time and used 49% fewer total tokens. Astra at max effort under the adapter used fewer actions than the human baseline on 96.0% of levels, averaging 51.7% fewer actions per level. ARC had expected the opposite, that models would need far more exploration than humans even where they succeeded.

The Provider Adapter preserves OpenAI's opaque reasoning state, and opaque means what it says. Nobody outside OpenAI can inspect what is being carried forward. So the spread measures one vendor's proprietary feature working. A reader running a model with no such feature is holding a number about somebody else's stack.

The usual framing has scaffolding as a tax you pay for reliability. This run charges you less for the better version.

The vendor grading its own API

Five weeks earlier, OpenAI published its own investigation into why GPT-5.6 Sol was doing badly on the same benchmark. Under the official harness on the public task set, Sol scored 13.3%. With two settings changed, it scored 38.3%, with 6x fewer output tokens.³3The post is internally inconsistent about its own baseline. It opens saying Sol "scored just 7.8%" and later states "With the official harness, GPT-5.6 Sol scored 13.3%." Two official-harness numbers for one model in one post, unexplained. The same post records GPT-5.5 at 0.4% on that harness, which is the number that shows how far the reported floor can sit from the model. It's a small thing and it points the same direction as the argument.

Retained reasoning. The ARC harness discarded the model's private reasoning after every action. Sol could see its past moves and short notes, and could not see the thinking that produced them. Every turn, it re-derived the game from scratch.

Compaction instead of rolling truncation. The harness dropped the oldest messages once context passed 175,000 characters. On a benchmark where the whole task is learning an environment by exploring it, the oldest messages are the part that explains the environment.

OpenAI estimates the average human tester scored 48% on that set. So 13.3% reads as a model that can't play 2D puzzle games, and 38.3% reads as a model approaching human performance. Same weights either way.

The caveat rides along: both settings are OpenAI's own Responses API features, and the post closes by recommending readers adopt them. It's a vendor benchmarking its own product surface, published by the vendor. Discount it accordingly, and then notice that it replicates.

The 30 that was never a baseline

NVIDIA's AVO result is the one I see quoted most, and it's the one that needs the most care.

On 21 August, NVIDIA reported 100.00 RHAE on the ARC-AGI-3 public set: all 25 environments, all 183 levels, in 6,624 environment actions, running Claude Opus 5. The architecture story underneath is the good part. AVO was built for autonomous GPU-kernel optimisation, and the same agent core transferred to interactive reasoning with a different task interface bolted on.⁴4AVO fed the model exact 64x64 text grids with no images at all, following VISTA's direct-interaction principles rather than Tycho's explicit programmatic world models. VISTA's own primary configuration renders a 512x512 PNG. The text-only result is a useful data point any time you assume an interactive environment needs vision.

The subtitle does the travelling: "elevates Claude Opus 5 from a 30% model baseline to 100%."

NVIDIA refuses that reading in its own post. Twice. On the system comparison: "This should not be interpreted as a controlled ablation: the two systems differ in agent backend, observation representation, memory, context management, and other implementation details." And on the 30% figure specifically: "ARC Prize separately reports approximately 30% for Claude Opus 5 at High reasoning effort. Our run used the same model family under a different reasoning setting and a substantially different agent system and evaluation setup. These numbers therefore should not be interpreted as a direct measurement of the performance contribution of AVO."

Different reasoning setting. Different evaluation setup. The 30 and the 100 were never two conditions of one experiment.

The comparison in that post that holds up best is the one nobody quotes. AVO solved the same 183 levels with the same model in 6,624 actions against VISTA's 7,542, about 12% fewer. Same model, same levels. That beats 30-to-100 on control, though NVIDIA attaches the same non-ablation caveat to it and the caveat is fair: two whole agent systems still differ in more than one place. It is a smaller number carrying better evidence, and it went nowhere because 12% doesn't travel on social.

Nothing was hidden here. NVIDIA wrote the disclaimer, published it under the chart, and then watched the subtitle do the rounds anyway.

The $0.67 run that had no harness at all

The fourth result is the odd one, and it's awesome on its own terms.

On 1 September a single developer published an 8-layer transformer trained from scratch at test time on one RTX 5090, about 1.5 hours, $0.67 of total lifetime compute covering both training from initialisation and inference across every task. It scored 44% on ARC-AGI-1 public eval.⁵5And 7% on ARC-AGI-2, which is the number that puts the 44% in proportion. The harder set is much harder. No LLM, no recursion, no harness. That matches results which had been credited to recursive architectures.

The ablations are the content. Switching 3D RoPE to 1D drops it to about 24%. Removing the per-task embeddings drops it to about 24%. Training on outputs only beat training on inputs and outputs, 40% to 44%, while the test loss got worse over the same change, which is a clean little instance of a validation proxy pointing the wrong way.

So on this benchmark the answer was representation and training data. The scaffolding had nothing to do with it.

The scope limits ship with the number, and they're large. ARC-AGI-1 is grid transduction, a different benchmark from ARC-AGI-3 and a different class of task from anything agentic. It's the public eval set, where answers have circulated online for years, and the author says so more loudly than his critics do. Nothing here transfers to tool use or long-horizon work.

"The scaffold did it" is one of several competing attributions, and this result keeps the others live.

The author asks why nobody found this sooner: "Maybe the cost of experimentation was high enough that they couldn't run ablations properly? Blindsided by LLMs or using harnesses?" Coming from the person holding the 67-cent receipt, that question has some standing. It also sits next to a bookkeeping question that nobody ever answered side quest.

Four slots, and only one result fills them

I read these leaderboards to pick models I'm going to run. All four of those numbers came past me as evidence for something I already believe, which is exactly when I should be slowest to take them.

Before crediting a benchmark delta to anything, there are four slots to account for, and a headline only means what it says when the other three held still.

1. The model. Was the same model run in both conditions, at the same reasoning effort? If not, nothing in the comparison is about the harness. It is also the slot the Astra headline pair fails and its matched rows pass, which is why the table above exists. The 30-to-100 reading fails it outright, by NVIDIA's own statement.

2. The harness. Retained reasoning, context policy, tool availability, observation format, action interface. Name the ones that changed. "A better harness" is a placeholder where the finding goes.

3. Training and test data. Public eval set or held out? On a benchmark the author points out has been open six years with a million-dollar prize attached, the answers circulate. A public-set score and a semi-private score are different measurements wearing the same units.

4. Bookkeeping. What does the parameter count exclude. What does the cost figure include. Which set was it. Lifetime compute versus per-task compute is a multiplier, not a rounding difference.

Run the four against the set. Astra clears all of them, because ARC ran both harnesses at every effort level and published the matched rows. OpenAI's result clears slot two, and slot one only as far as the post says, since it never states the baseline's effort level and reports two different baselines for it. It fails three outright: public set, self-reported. AVO fails slot one. mdlARC fails slot three, and it was never a harness result to begin with.

One of four. That's the yield on seven weeks of results that were all, in the moment, quoted as the same finding.

The two settings that showed up in both write-ups

The convergence is where this stops being a reading exercise and starts being work you can do.

ARC's Provider Adapter preserves reasoning state between requests and compacts long conversations. OpenAI's harness retains reasoning across tool calls and compacts instead of truncating. Two organisations wrote those reports without coordinating, on different frontier models, and both landed on the same two mechanisms.

A vendor claim on its own is marketing. An independent evaluation landing on the same two levers, without coordination, is a finding.

Three things none of this licenses.

These are ARC results, and ARC-AGI-3 is a deterministic, closed-ended game environment. ARC itself declines the AGI claim on that basis. A 37-point harness spread on exploratory grid games says nothing about the spread on a support queue or a code review.

Retained reasoning is a property of specific provider APIs. A harness that can't get opaque state back from its provider can't buy that delta at any price. That makes it a procurement question as much as an engineering one, and procurement questions have a habit of arriving after the architecture is set.

And nobody has replicated the harness delta on a non-OpenAI model.⁷7AVO is the obvious objection, since it ran Claude Opus 5 and reached 100.00. It is not the same claim: public set rather than semi-private, no matched control against a weaker harness, and a whole agent system rather than one provider-state mechanism switched on. The Provider Adapter result is one model at one vendor.

So the open question is whether the premium survives contact with a model whose reasoning state isn't proprietary. Until someone runs that, what we know is that 37 points exist between two harnesses on GPT-6 Astra, and we don't know whether they're a property of harnesses or a property of one company's API.

---

Related reading on this site: Inventory Your Harness: The Six Components for what "change the harness" refers to concretely, Tabular foundation models got acquired before they got benchmarked for the same gap between what a result claims and what independent evidence supports, and I checked the 90% token cut against 30 days of my own Claude Code for the version of this where I ran a published claim against my own data instead of taking it.