I open-sourced OpenBonsai: compress your own BF16 models and fine-tunes into ~2 GB ternary GGUFs
OpenBonsai, an open-source tool, has been released to compress BF16 models and fine-tunes into approximately 2 GB ternary GGUFs. Developed for WebBrain, a browser agent aiming for local intelligence via WebGPU, OpenBonsai shows promising benchmark results. For example, in GSM8K (thinking off), it scored 65/100 compared to the original BF16's 77/100, while Prism Bonsai 2 achieved 77/100. Users are encouraged to compress their own fine-tunes and share their experiences.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Oct 29, 2026, 08:13 UTC
IngestedOffset at this time: UTC+0Oct 29, 2026, 20:00 UTC
- Published
- Oct 29, 2026, 08:13
- Ingested
- Oct 29, 2026, 20:00
- Source type
- Dev community
- Tier
- Community
- Source status
- Sync delayed
Tier is a per-source editorial setting, not a per-item score.
I've been experimenting with tiny models for WebBrain , my open-source browser agent. The long-term goal is to run as much intelligence as possible locally, ideally through WebGPU.
One approach I've been particularly interested in is Prism ML's Ternary Bonsai: compressing a 9B model to roughly 2 GB using ternary weights (-1, 0, +1).
Prism released the weights and inference runtime, but not the complete BF16-to-ternary compression pipeline. You can fine-tune their existing ternary models, but there's no publicly available way to compress your own BF16 fine-tune into their format.
So I tried building one.
Introducing OpenBonsai, an Apache-2.0 framework for post-training ternary quantization, compatible with Prism's PTQ1_0 GGUF format.
How it works
A recent Intel Labs China paper, "Attend to Your Own Thoughts" (Wang et al.), inspired part of the approach: using the model's own reasoning traces for calibration rather than relying exclusively on generic text.
The pipeline combines:
- Signed Hadamard rotations to reduce weight outliers
- Sequential GPTQ to quantize weights to {-1, 0, +1}
- FP16 group scales (one per 128 weights)
- Self-generated reasoning traces for calibration
- Scale distillation against the original BF16 model's output distributions
- Export to Prism's PTQ1_0 GGUF format
Results
Qwen3.5-9B: ~18 GB → 1.997 GB
That's the complete GGUF file, not just the transformer weights. It includes the compressed embeddings, LM head, all other model tensors, group scales, tokenizer, chat template, and metadata. Runtime memory requirements are higher.
The compression runs in approximately 5 hours on a single RTX 5090 (32 GB).
Our current compressed model scores:
Benchmark Original BF16 OpenBonsai Prism Bonsai 2 GSM8K (thinking off) 77/100 65/100 77/100 HumanEval (thinking off) 44/50 34/50 45/50 GSM8K (thinking on) 36/50 32/50 46/50 HumanEval (thinking on) 22/25 14/25 24/25
Some interesting findings
1. Distillation helps substantially.
Scale distillation improved development-set NLL from 3.48 to 2.89 and ARC-Easy accuracy from 13/20 to 19/20.
2. Calibration data matters enormously.