Bonsai 27B: Frontier Reasoning at 1.7 Bits a Weight

The 1-bit build answered a multi-step reasoning question on my MacBook this morning, at about 20 tokens a second, with nothing in the cloud. Not on a maxed-out machine, either. People run local models on 128GB Macs all the time; mine has 36GB, and even the larger ternary build left most of it sitting free. The same model family has a build that fits on an iPhone. That sentence has not been true before, and the reason it is true now is not the one you would guess.

This is not a 4-bit quant of a big model squeezed onto a laptop. It is Qwen3.6-27B rebuilt so the weights themselves are binary or ternary, trained that way rather than compressed afterward. PrismML calls it Bonsai. I spent a morning running both families locally, scoring them on reasoning, and pointing one at my own knowledge base through a tool loop. Here is what I found.

What Bonsai is, and who built it

PrismML is a Caltech spin-out. The CEO is Babak Hassibi, whose name you know from information theory; the founding team is Sahin Lale, Omead Pooladzandi, and Reza Sadri.¹1PrismML raised $16.25M in seed and SAFE funding from Khosla Ventures, Cerberus Capital, and Caltech. It emerged from stealth in April 2026 with the 1-bit Bonsai 8B; the 27B landed July 14, 2026, under Apache 2.0. The whole thing is built on Caltech intellectual property, and the models ship under Apache 2.0.

Bonsai 27B is Qwen3.6-27B, architecture untouched, moved into two low-bit representations. The ternary build stores each weight as one of {-1, 0, +1}. The binary build stores each as {-1, +1}, a single sign bit. Both keep an FP16 scale per group of 128 weights. The model is a hybrid-attention design (about 75% linear attention), carries a 262K-token context, and does vision, reasoning, and native tool calls. It ships with a speculative-decoding layer called DSpark on top.

The numbers, measured bits-to-bits rather than by marketing label, run ahead of what the labels suggest. Ternary Bonsai 27B averages 94.6% of the FP16 score at roughly 1.71 bits per weight, which packs to 5.9GB. The 1-bit build keeps 89.5% at 1.125 bits per weight and 3.9GB.²2FP16 Qwen3.6-27B is 54GB and averages 85.07 on the whitepaper's 15-benchmark suite. Ternary Bonsai scores 80.49 at 5.9GB; the 1-bit scores 76.11 at 3.9GB. The 5.9GB is the native target; today's ternary GGUF is about 7GB because current kernels pack each ternary weight into a 2-bit slot. The 1-bit build has no such gap. Source: the Bonsai 27B whitepaper, PrismML, July 2026. For comparison, a normal FP16 27B is 54GB, and even a standard 4-bit build lands near 14 to 18GB.

The same tradeoff, as a control you can move:

The distinction to get right is that this is native low-bit via quantization-aware training, not quantization. Post-training quant takes a finished FP16 model and rounds its weights down. Bonsai starts from the pretrained Qwen and trains it into the binary or ternary representation while holding onto its behavior. That difference explains why the model does something the labels say should be impossible.

Why sub-4-bit has been a wall

Everyone knew binary and ternary weights were possible in principle. The problem was that capable models fell apart below about 4 bits, and they fell apart in a specific, ugly way.

Push a conventional model below 4 bits with standard quantization and it does not degrade smoothly. It collapses. Chain-of-thought reasoning drifts or contradicts itself, tool calls stop parsing, agentic loops lose coherence partway through. And because surface fluency survives, the collapse is easy to miss. On short-form benchmarks like MMLU-Redux the aggressive builds look fine; on reasoning benchmarks the floor drops out. The most aggressive conventional Qwen build, IQ2_XXS, scores 88.93 on MMLU-Redux but falls to 57.5 on AIME26 and 56.4 on LiveCodeBench.³3A brittle model that fails on the hard tasks while passing the easy ones is worse in production than one that is uniformly weaker, because it erodes trust in the whole system and makes behavior hard to predict. This is the argument the whitepaper builds its case around.

There has been one prior line that avoided the collapse: BitNet and its 1.58-bit successor, which pretrain a network from scratch directly in low-bit. That works, but it throws away every existing pretrained model and demands a full pretraining run for each new one, at a cost few can absorb. It also has not scaled; the largest native 1-bit models stayed around 2B parameters. Bonsai takes the opposite path. It starts from an off-the-shelf pretrained model and moves it into the low-bit regime, so the years of pretraining are kept, not repeated.

Running it locally: the hard part

I ran everything on a 36GB M4 Max, through llama.cpp. The 1-bit build was easy: its Q1_0 format is merged into mainline llama.cpp, so brew install llama.cpp and a hf download were enough. The ternary build was not easy, and the reason is worth flagging for anyone about to try this.

The ternary format and the Metal kernel that runs it are coupled, and they are not yet unified upstream. Stock Homebrew llama.cpp (build 9960) parses the mainline g64 ternary file fine, then aborts on the first matrix multiply because Metal has no kernel for that weight type:

ggml-metal-device.cpp:919: not implemented
Asserting on type 42

PrismML ships a fork binary that has the kernel. But the fork only reads its own native g128 file, and refuses the mainline g64 with a tensor-offset error. So mainline reads the file it cannot run, and the fork runs the file mainline cannot read.

The second thing that cost me time was the model itself. Bonsai 27B is a reasoning model, and thinking is on by default. On my first request it returned an empty answer. It had spent the entire token budget in its reasoning trace before reaching the reply. On a trivial prompt it generated between 1,000 and 5,000 characters of thinking; asking it to write a Fibonacci function, it burned through 1,200 tokens reasoning and never emitted code. The fix is to turn reasoning down for simple work:

{
  "messages": [{"role": "user", "content": "Write fib(n) iteratively in Python."}],
  "chat_template_kwargs": {"enable_thinking": false},
  "max_tokens": 600
}

With thinking off, the same prompt returned correct, documented code in about ten seconds. Leave reasoning high for hard problems, where it earns the wait, and drop it to off or low for everything else. You can also cap it server-side with --reasoning-budget.

Does it hold up

I ran both families through a battery of reasoning traps, the kind that separate careful models from confident ones. Both went five for five.

The bat-and-ball problem (answer five cents, not ten) came back at $0.05. The average-speed problem, where the lazy answer is to average 60 and 40, came back at 52.5 mph. Letter-counting across "strawberry raspberry" came back at 6, past the tokenization failure mode that trips many larger models. Transitive logic and a distractor question both landed correctly. For a 3.9GB model, that is a strong showing, and it matches the whitepaper's claim that the reasoning behaviors survive the compression.

Throughput split the way physics predicts. The 1-bit build ran at about 19 tokens a second, the ternary at about 8. Ternary carries roughly twice the bytes per weight, and token generation on a laptop is bound by how fast you can stream weights out of memory, so half the speed is the expected result.

One caveat, stated straight: those absolute numbers are a loaded lower bound. My machine was under sustained thermal load with about fifty other processes running, and the ternary build was on the older fork binary. PrismML's own clean figures for an M4 Pro are 26 tokens a second for binary and 18 for ternary, and my M4 Max should clock higher than that unthrottled. The 2x ratio between the two builds is the durable signal; the absolute rates on your machine will be higher than mine were.

I should also be honest about what my test did not show. The scoreboard says ternary keeps 94.6% of FP16 and 1-bit keeps 89.5%, a five-point gap. My seven prompts did not surface it. Both builds aced all of them, because these traps are not hard enough to separate a five-point difference. To see that gap you need the full 15-benchmark suite at real difficulty, not a smoke test.

For a check that is not mine and not PrismML's, an informal independent benchmark on a Jetson Orin put Ternary-Bonsai-8B at 85.0% across 98 questions, level with the dense Qwen3.5-4B at 85.2%.⁴4Informal benchmark by ArmanJR, published on GitHub (PrismML-Bonsai-vs-Qwen3.5-Benchmark), comparing Bonsai variants against six Qwen3.5 models on an NVIDIA Jetson Orin. The author frames it as a rough directional check, not a rigorous ranking.

The part I actually wanted to test: tools over my own data

A model that reasons on-device is interesting. A model that can reach my own knowledge base on-device, and act on it, is the thing worth building toward. So I wired one up.

llama.cpp's web UI has a built-in MCP client that runs in the browser, which means it talks to tool servers over HTTP. My Obsidian search server speaks stdio, not HTTP, so I bridged it: mcp-proxy wraps the stdio server and exposes it as a streamable-HTTP endpoint with CORS enabled.

uvx mcp-proxy --host 127.0.0.1 --port 9091 --allow-origin '*' --stateless -- ~/bin/obsidian-search-server

Then I added http://127.0.0.1:9091/mcp in the web UI's MCP panel, enabled the tool, and asked the ternary 27B to search my notes on my GPU server and local model serving and summarize what I have documented.

It reasoned about the request, called search_vault twice and search_all once, hit my vector backend, and wrote a summary that cited my actual note titles, with the correct hardware details pulled from them. Not a hallucinated answer dressed up as retrieval. A tool loop that ran, returned my data, and got grounded in it.

Every part of that loop ran locally. The model ran on my laptop. So did the tool server. The data lived on the same machine. The only thing that moved was a search query to my own vector store, and nothing touched an external API.

side quest

What it means, and the limits

For years "edge" has meant small models, because the memory budgets are small. A flagship phone gives a single app a few gigabytes; a laptop shares 16 to 48GB across everything. That ceiling was a capability limit, not just a size limit, because sustained reasoning and dependable tool use did not appear reliably until a much larger scale. So the 27B tier got treated as inherently cloud-bound. Bonsai removes the pairing. It puts the capable tier inside the same devices that until now admitted only small models, because the barrier was its bytes, not its intelligence.

The limits deserve naming, so here they are. It over-thinks by default, and you have to manage the reasoning budget. The low-bit toolchain is fragmenting, and you will hit format-versus-kernel mismatches until the kernels land upstream. My throughput numbers are a loaded lower bound. The quality gap between the two builds is unproven on anything I ran; I take the 94.6-versus-89.5 on the whitepaper's word and the independent Jetson result. And I could not sweep social reception today, my X tooling was returning auth errors, so the reactions I have are from the web and Reddit, not from practitioners on X.

None of that changes the basic fact. I have a frontier-tier reasoning model, doing tool-use over my private notes, running on a machine I can close and put in a bag.

Related reading on this site: the case for computing in memoryshippedCutting-Edge AI·Feb 26, 2026The Model IS the Computer: What Compute-in-Memory Means for AI DeploymentTaalas baked Llama 3.1 8B into transistors, hitting 17k tok/s at 200W. The architecture matters more than the benchmark, and it changes what's worth building. for why moving fewer bytes per token, not raw compute, sets the speed ceiling, agent architectures with MCPshippedAI Systems & Architecture·Mar 24, 2025Agent Architectures with Model Context Protocol: A Technical SurveyTechnical survey of architectural patterns for implementing AI agents with Model Context Protocol, including comparative analysis of frameworks. for how the tool-loop plumbing works, and building RAG infrastructure on a local GPU boxshippedPractical Applications·Oct 24, 2025DGX Lab: Building a Complete RAG Infrastructure - From Ollama to Qdrant to AnythingLLM - Day 3Building a complete self-hosted RAG infrastructure with 8 integrated services on a single DGX workstation, supporting everything from medical AI fine-tuning to document Q&A. for the vector backend the search tool reaches.