GLM-5.2 on 8xB200: the deployment math nobody spells out - NVFP4 + 2x TP=4 replicas should beat TP=8 by ~2x. Full config guidance inside.
BuzzRadr reports on optimal GLM-5.2 deployment on 8xB200 nodes. Analysis suggests NVFP4 with two TP=4 replicas per node significantly outperforms FP8 TP=8, potentially doubling thr…
We have 8xB200 nodes and users keep asking us how to serve GLM-5.2 on them. Our engineering team went through everything published so far, and the optimal config is not the obvious one. Sharing the analysis because most of it applies wherever you rent or rack your B200s.
The model
GLM-5.2: ~750B total / ~40B active MoE (256 experts, top-8 routing, ~5.9% sparsity), DSA + MLA attention, 1M context, MIT license. Weights: ~744 GB in FP8, ~459 GB in NVFP4 (KV cache stays FP8).
The hardware math
8x B200 SXM = 1,440 GB HBM3e aggregate, 8 TB/s per GPU, NVLink 5 (900 GB/s/GPU).
The non-obvious part: MoE decode at moderate concurrency streams ~40B active params + KV cache from HBM every step - it's bandwidth-bound, not compute-bound. That's why B200 over H200 at the same FP8 precision is only ~1.2x perf/$ (tracks the HBM bandwidth ratio, not the 2.3x FLOPs ratio). The lever that actually moves the number is NVFP4: half the weight bytes to read per step, and Hopper has no FP4 tensor cores at all.
The published numbers (InferenceX / SemiAnalysis - SGLang v0.5.12 + EAGLE MTP, ISL 8192 / OSL 1024)
These are GLM-5 runs - same architecture family. For 5.2 on Blackwell, what's public so far is provider-level speed, but we haven't found full concurrency-sweep tables (tok/s/GPU vs conc vs TPOT) on a documented 8xB200 config - happy to be corrected. At 8K context we expect 5.2 to land close to GLM-5, since its IndexShare change mainly pays off at long context.
FP8, TP=8 (whole node, one engine):
Conc tok/s/GPU tok/s/user TPOT (ms) 4 417 100.9 9.9 16 953 56.9 17.6 64 1,619 23.6 42.5 256 1,947 11.9 84.2 NVFP4, TP=4 (half the node):
Conc tok/s/GPU tok/s/user TPOT (ms) 4 1,039 121.2 8.3 16 2,228 66.3 15.1 64 3,740 26.8 37.3 128 4,116 17.6 56.7 Source: https://inferencex.semianalysis.com/blog/b200-glm5-nvfp4-vs-h200-fp8-3-6x-perf-per-dollar
What falls out of the math
- NVFP4 fits in 4 GPUs - 459 GB weights in 720 GB HBM leaves ~230 GB for KV cache. So one node supports two independent TP=4 replicas behind a load balancer: 2 x 4 x 4,116 ≈ 33k tok/s aggregate, vs ~15.6k for FP8 TP=8. Roughly 2x the node throughput and better per-user speed at matched load. Caveat: this is arithmetic on published single-replica data - we haven't seen a published 2-replica-per-node test, and we'd expect some loss to scheduler/NCCL contention.
- TP=8 NVFP4 buys latency, nothing else: 140 tok/s/user at conc 4 vs 121 on TP=4, at half the per-GPU throughput. Only justified by hard TPOT SLAs.
- Cost: at SemiAnalysis's $1.95/GPU/hr B200 TCO, NVFP4 lands around $0.13/M tokens at the throughput end of the curve. Their H200 FP8 reference: $1.06/M at 80 tok/s/user - a ~3.5x perf/$ gap on this model family.
- 1M context fits on paper (FP8 KV in 1,440 GB), but a single 1M-token prefill monopolizes an aggregated engine. If long-context is your workload, plan for disaggregated prefill from day one. IndexShare claims 2.9x FLOP reduction at 1M; we found no independent TTFT measurements yet.
- The version trap that produces silently wrong outputs: SGLang #21291 ) - bad generations, no crash. Use v0.5.10+, which defaults to FlashInfer TRT-LLM sparse MLA on sm100 ( #21783 ). MTP flags the benchmarks used: --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4. Note: v0.5.10+ covers the FP8 path; the NVFP4 5.2 checkpoints need SGLang v0.5.13.post1+ (vLLM v0.23.0+).
Brings us to our questions:
- Anyone benchmarked the community NVFP4 quants of 5.2 for quality vs FP8?
- Anyone running the 2x TP=4 layout in production - how much of the theoretical 2x survives contact with a real scheduler?
- MTP acceptance rates on 5.2 across workloads - GLM-5 data implies ~40-55% decode uplift, but it's workload-dependent.
We're standing up GLM-5.2 on our own 8xB200 nodes over the next couple of weeks and will post measured numbers - FP8 vs NVFP4, TP=8 vs 2x TP=4, plus long-context TTFT - as a follow-up, with full bench_serving JSONs.