What Researchers Found When They Looked Inside Biology AI Models

In June 2025 I wrote that the tools for biology AI were mature. That was ages ago in AI time. I was excited about models that read DNA and proteins, and I never asked whether they read them at all.

This fall people started checking. Researchers handed frontier models the job of editing other models' weights. Two groups argued in print over how to score cell models, and a lot of the fight turned out to be about the ruler. And one team shuffled the DNA going into a disease-prediction model and got the same score back, to three decimal places.

I've written about evals a lot, and this is the best batch I've seen for showing why they're worth the trouble. Several headline numbers had a plainer explanation sitting right next to them. The nice part is that most of this was published in enough detail for a stranger to rerun it.

---

Three ways to look closely

You can check these models without a wet lab.

Open the model and change something inside it, a handful of weights, and see which behaviour shifts.

Compare it with a dumb baseline. Predict the average response for every experiment, or guess the effect of two gene changes by adding up each one alone, and see whether the expensive model beats that.

Or break the input and watch the answer. Scramble the DNA letters and run the model again. If the answer doesn't move, the model wasn't leaning on the DNA.

---

A model edits a model

Researchers at MIT CSAIL had an AI agent, OpenAI's GPT-6 Astra, edit a small open model called Qwen3.5-0.8B. The agent found a direction inside the model tied to getting stuck in loops and subtracted it from 0.016% of the weights. It only ever saw practice problems from training sets, never the test. Their post reports the model's score on MATH-500, a set of competition math problems, going from 7.9% to 48.1%. That's the run where the agent got 50 practice passes. With 10 or 25 it got 12% and 14%.

The reason is in their logs. The model ran with its thinking mode on, and before the edit 93% of its answers hit the length limit while it was still talking to itself. After the edit, about a third did. It's a loop fix, and the authors call it one.

Then someone checked the obvious thing. Shapor Naghibzadeh ran the unedited model with thinking switched off, which stops it spending its whole length limit on thinking, and got about 45%. With thinking off, the edit added nothing: 44% edited against 45% unedited. That's one run, about two points either way. A model with 0.8 billion weights mostly needed to stop talking to itself.

In September I wrote about the same model scoring like two different models on one test, depending on the setup around it. Here too, a setting moved the score nearly as much as the edit did. The authors draw the limit themselves, in a sentence I'd frame: "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."

Matthew Honnibal's critique is about packaging. The pitch is an agent that understands networks by reading their raw weights. He points out that only the tiny-network experiments worked that way, with the weights pasted in as numbers and no code. The language-model work used Python and ran the model on practice problems. He's right that these are several techniques under one headline. He also grants that working out the edit is no small thing, and I'd agree with that too.

The tiny-network test is easy to replay. It hands a model four small networks trained to play Breakout, Pong, Pac-Man and Freeway, written out as pages of plain numbers, and asks which game each one plays. I replayed it on Claude Opus 5.5 at the same high effort setting, five tries per network. It named the game 20 times out of 20. From a page of nothing but numbers, that's awesome.

Now the fine print. At its default setting it got 20 of 28. Claude scored the answers by reading them, one reader and no second scorer, on small samples, so this doesn't show the two models are equal. Shapor got 4 of 4 from Opus too, and when he shuffled the Breakout board the models still said Breakout. So the models may be reading something other than the layout of the board.

The biology version is rougher. Diego del Alamo posted that he asked Opus 5 to do interpretability work on ProteinMPNN, a model that writes an amino-acid sequence to fit a protein shape. It edited a small slice of the weights and pushed membrane-protein designs toward SolubleMPNN, a sibling trained on proteins that stay dissolved in water. On the membrane-facing surface, the share of oily amino acids dropped to about where SolubleMPNN puts it. One of his four starting models was already most of the way there.

He's open about the gaps when people ask. There's no lab test and no comparison with ordinary fine-tuning. He gave the model 100 SolubleMPNN redesigns per structure to aim at. His employer won't let him release the weights, and he doesn't think it scales well.

I like who did it: a protein engineer asked a chat model to do interpretability on a tool from his field, and it did some. Still, every edit to an existing language model that I've read about removes an obstacle.

---

The fight over the ruler

In August 2025, Ahlmann-Eltze, Huber and Anders published a short paper in Nature Methods with a blunt title: deep-learning-based gene perturbation effect prediction "does not yet outperform simple linear baselines". They asked five foundation models and two other deep learning models to predict how a cell's gene activity shifts when one or two of its genes are changed, and lined them up against deliberately simple guesses. None of them beat the simple guesses. The hedge in that title is one word long, and I think it was the right word.

This October, Miller and colleagues argued in Nature Biotechnology that the models were better than that, and the scoring had hidden it. Their point is about averaging. Changing one gene usually moves a handful of genes out of thousands. Average the error over all of them and the real change barely registers, so a model that always predicts the average response looks accurate.

So they tested each scoring method before trusting it. They fed it a prediction they knew was good, half of an experiment's cells predicting the other half, and one they knew was empty, the plain average. A good scoring method has to tell those two apart, and two widely used ones often couldn't. On the methods that could, most of the models beat the baselines. On the old methods, they got the old result.

They checked the rulers on 14 datasets and 18 metrics. The model comparison ran on four datasets, mostly with newer models than the five foundation models in the first paper. So the rulers got a wide test and the models a narrow one.¹1The weighted scores came from earlier work by the same group.

In May I wrote that evals were the new bottleneck, and that the thing measuring a model needs calibrating like any other lab instrument. Miller's group did that: a positive and a negative control on the ruler itself, before trusting it. They're clear it isn't settled. A score that passes its controls may still miss the biology, and on pairs of gene changes the add-two-effects guess still holds up well.

---

Shuffle the DNA, get the same answer

The shuffle study looks at models that pair a biology model with a chat-style language model. The biology model, Evo 2 for DNA or ESM3 for protein, turns a sequence into a long list of numbers. The language model gets that list, some text about the gene or protein, and a question, and writes its answer in plain English.

The paper comes from Fang, Zitnik and colleagues, out of Marinka Zitnik's lab. It checks six of these systems across DNA, protein and single-cell tasks, with four tests:

• Shuffle: scramble an input, rebuild the numbers, and see if the score moves.
• Swap: give the model one sample's DNA numbers next to another sample's text, and see which one the answer follows.
• Probe: fit a simple classifier on the numbers alone, to check the information is in there at all.
• Trace: read the model's written reasoning against what it was given.

Four of the six use their inputs. ChatNT, a DNA model, drops to a coin flip on a splice-site task when its DNA is scrambled. So the shuffle can move a model. The authors add that a scrambled sequence has no right answer, so the drop only shows that the answer changes.

Two don't, on the tasks tested. In BioReason and BioReason-Pro, the Evo 2 and ESM3 numbers contribute little. BioReason predicts diseases from a stretch of DNA plus text about the gene and its pathways. It scored 0.847 with the DNA intact and 0.847 with the DNA scrambled.²2That's one checkpoint on 1,449 questions, where it got 1,227 right with the DNA and 1,228 with it scrambled. My first guess? A broken shuffle. ChatNT says otherwise, and so does the swap test.

The shuffle did change 101 answers, about as many for the better as for the worse. When the DNA and the text disagreed, it went with the text almost every time. BioReason-Pro follows its text too, and that text is the output of another prediction model, so the thing it trusts most is another model's guess about the protein.

The DNA information is in there. A simple probe on the Evo 2 numbers reads a little more than the text alone gives you, though most of that lead comes from one gene. So the numbers carry some signal, and the model went with the text anyway.

The text carries most of the answer. A lookup table that maps each gene name to its most common disease scores 0.917 on held-out queries. A smaller version of the model that the authors trained themselves scores 0.901 on the same queries. A gene name alone nearly solves the task. If your training prompts always hand you the gene name and the pathway, I'm not sure why you'd ever learn to read DNA. The two systems that lean on text are the two whose training prompts always include text that points at the answer. The authors say their comparison can't prove that's the cause, and they didn't test it. I'd want to see it run once more with the gene name hidden.³3Across dozens of BioReason checkpoints, higher scores didn't reliably come with more use of the DNA.

A benchmark score can't tell you whether a model used the biology. This is one of the cleanest checks I've seen for it, and the shuffle costs one extra run. The swap and the probe take more work, since they rebuild the numbers and fit a classifier. A lab wouldn't trust an assay that reads the same on a scrambled sample as on the real one. Biology models deserve the same control. When someone shows me one number for a DNA model, I want two.

---

Ask for the second number

A year ago these models were cool. This fall a chat model edited one of them, and the most useful result in the whole batch came from scrambling the DNA and pressing run again.

Most of this checking worked because the work came with code or logs. The shuffle study shipped its code. The weight-edit team released the edited model and its logs, which is how a stranger could switch thinking off and get 45%. Del Alamo didn't release his weights, so nobody can rerun his.

So if you pitch me a DNA model, expect two questions. What's the score, and what's the score with the DNA scrambled?

---

Related reading on this site: 62.7 and 99.9 Are the Same Model for how much the setup around a model can move its score, Tabular foundation models got acquired before they got benchmarked for another field where the independent checks trail the headline, and Four Upgrades to the Agent Loop. None of Them Moved the Number. for what a simple baseline does to an expensive upgrade.