4-bit GLM-5.2 (753B MoE) on 4× DGX Spark: 70.8% on Terminal-Bench 2.1 vs 81.0% for the full model

TL;DR: Full GLM-5.2 (753B MoE) quantized to Int4-Int8Mix + NVFP4 4-bit KV cache, TP=4 across 4× DGX Spark (GB10) at 100K context, run on Terminal-Bench 2.1 with the same agent scaffold (Terminus-2) as the official numbers. Result: 63/89 = 70.8% vs the official full-precision 81.0%. Caveat up front: I never ran the full model through my pipeline — the ~10-pt gap bundles quantization plus my 100K-vs-256K context cap, a smaller token budget, and unmatched sampling. So read it as: my whole 4-bit/100K desktop setup lands ~87% of the official number. The run took 72.5 hours, the engine crashed twice, and one recipe hard-wedged all four nodes — war stories below.

The rig

- 4× DGX Spark / GB10 (sm_121a, 128 GB unified each, ~273 GB/s), ConnectX-7 on a 100 G RoCE fabric (MikroTik CRS504), TP=4.

- Weights: GLM-5.2 Int4-Int8Mix (experts 4-bit → Marlin MoE, attention 8-bit), ~378 GB on disk.

- KV cache: NVFP4 4-bit for the sparse-MLA path — this is what unlocks 100K context (fp8 KV tops out ~64K here).

- Decode: MTP speculative decode (depth 3) + FULL CUDA graphs → ~27.5 tok/s (eager: ~17–21, safer on memory).

- Stack: vLLM rebuilt for sm_121a (Triton sparse-MLA + DeepGEMM bypass — upstream DeepGEMM rejects sm_121), max-num-seqs 2, gmu 0.90.

The benchmark

Terminal-Bench 2.1 (89 tasks) via Harbor + Terminus-2: pass@1, -n 1, ~3 h cap/task, driven from a cheap droplet over Tailscale. Fresh container per task, multi-step jobs, graded on final container state — no partial credit.

Results

Official (full) This run (4-bit) TB 2.1 81.0% Agent Terminus-2 Context 256K max_new_tokens ~48K Timeout 4 h Sampling temp 1.0 / top_p 1.0 Runs reported single/avg? Hardware datacenter Fine print, honestly:

- 70.8% (63/89) is the like-for-like number. I also get a "clean" 72.4% (63/87) by dropping 2 tasks the harness physically can't start (qemu-*: "Failed to start tmux session", reproducible — the model never gets to attempt them). That only adjusts my denominator, so raw stays the headline.

- Single pass@1 run → 95% CI on 70.8% is roughly ±9 pts (~62–80%), so the gap is real but not cleanly outside single-run noise; "~87% retention" (or ~89% clean) is a point estimate.

- 81.0% source: the Terminus-2 @256K figure from Z.ai's release blog (their 82.7% is Claude Code averaged over 5 runs — different scaffold, not compared).

- Where the gap comes from — I can't cleanly separate: quant (small per-token errors plausibly compounding over hundreds of agent steps — hypothesis, no QA ablation), the 100K context cap (forced history summarization on long tasks — a deliberate memory trade-off, not model degradation), and the smaller token budget. My shorter timeout cuts against me, so that axis is conservative.

War stories

1. Unified memory is a head-trip. The "VRAM" is the system RAM: raising gpu-memory-utilization leaves less free RAM, and KV, activations, and the OS fight over the same 128 GB. gmu 0.83 → "No available memory for cache blocks"; 0.90 works but leaves ~2–2.5 GB free.

2. The recipe that ate the cluster. One "everything at 200K / fp8-KV, verbatim" attempt hard-wedged all four nodes — SSH-dead, full power-cycle to recover. On memory-tight unified-memory boxes the reference recipe can take the whole cluster down, not just OOM a process.

3. Two engine crashes mid-run — request-triggered vLLM bugs (a scheduler KeyError → EngineDeadError; later RuntimeError: cancelled), not OOM, not NCCL. The nasty part: rank-0 died but 3 workers stayed up holding ~115 GB each — and docker rm -f (SIGKILL) wedges the relaunch. You must SIGTERM each worker, watch memory actually free, then relaunch (~9 min). A dead endpoint got hammered ~1 h before I caught one crash.

4. Honest numbers require auditing the errored bucket. TB2.1 separates "errored" from "failed". extract-elf errored on a connection error (my crash, not the model) — re-run clean, it genuinely failed → ambiguous error converted to a real verdict. The 2 qemu-* errors reproduce every time → harness, excluded. Harbor gotcha: it refuses to resume with a changed config, so you can't -x a task mid-run — instead delete the errored trial dirs and resume with the original config; it re-runs exactly those (raw 63/89 backed up first). If you take one thing away: audit errored-vs-failed before quoting a pass-rate.

Takeaways

- A 753B open-weight model at 4-bit on four ~$4K desktops lands ~87% (point estimate, one run) of the official full-precision score on one of the hardest agentic benchmarks.

- Infra failures silently masquerade as model failures — the errored bucket is where the truth hides.

- Long unattended vLLM runs need an auto-relaunch plan and a clean-shutdown-before-relaunch dance.