I open-sourced OpenBonsai: compress your own BF16 models and fine-tunes into ~2 GB ternary GGUFs
OpenBonsai, an open-source tool, has been released to compress BF16 models and fine-tunes into approximately 2 GB ternary GGUFs. Developed for WebBrain, a browser agent aiming for local intelligence via WebGPU, OpenBonsai shows promising benchmark results. For example, in GSM8K (thinking off), it scored 65/100 compared to the original BF16's 77/100, while Prism Bonsai 2 achieved 77/100. Users are encouraged to compress their own fine-tunes and share their experiences.
时间与来源
时间显示为 UTC
显示时区：UTC
本地时区尚不可用，暂时显示 UTC。
发布当时偏移：UTC+02026年10月29日 08:13 UTC
收录当时偏移：UTC+02026年10月29日 20:00 UTC
- 发布
- 2026年10月29日 08:13
- 收录
- 2026年10月29日 20:00
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 同步延迟
档位是按信源手工设定的编辑判断，不是逐条打分。
I've been experimenting with tiny models for WebBrain , my open-source browser agent. The long-term goal is to run as much intelligence as possible locally, ideally through WebGPU.
One approach I've been particularly interested in is Prism ML's Ternary Bonsai: compressing a 9B model to roughly 2 GB using ternary weights (-1, 0, +1).
Prism released the weights and inference runtime, but not the complete BF16-to-ternary compression pipeline. You can fine-tune their existing ternary models, but there's no publicly available way to compress your own BF16 fine-tune into their format.
So I tried building one.
Introducing OpenBonsai, an Apache-2.0 framework for post-training ternary quantization, compatible with Prism's PTQ1_0 GGUF format.
How it works
A recent Intel Labs China paper, "Attend to Your Own Thoughts" (Wang et al.), inspired part of the approach: using the model's own reasoning traces for calibration rather than relying exclusively on generic text.
The pipeline combines:
- Signed Hadamard rotations to reduce weight outliers
- Sequential GPTQ to quantize weights to {-1, 0, +1}
- FP16 group scales (one per 128 weights)
- Self-generated reasoning traces for calibration
- Scale distillation against the original BF16 model's output distributions
- Export to Prism's PTQ1_0 GGUF format
Results
Qwen3.5-9B: ~18 GB → 1.997 GB
That's the complete GGUF file, not just the transformer weights. It includes the compressed embeddings, LM head, all other model tensors, group scales, tokenizer, chat template, and metadata. Runtime memory requirements are higher.
The compression runs in approximately 5 hours on a single RTX 5090 (32 GB).
Our current compressed model scores:
Benchmark Original BF16 OpenBonsai Prism Bonsai 2 GSM8K (thinking off) 77/100 65/100 77/100 HumanEval (thinking off) 44/50 34/50 45/50 GSM8K (thinking on) 36/50 32/50 46/50 HumanEval (thinking on) 22/25 14/25 24/25
Some interesting findings
1. Distillation helps substantially.
Scale distillation improved development-set NLL from 3.48 to 2.89 and ARC-Easy accuracy from 13/20 to 19/20.
2. Calibration data matters enormously.