LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)
LayerStoRm推出了开源专家流式传输技术，使得GLM-5.3-Flash UD-Q4_K_XL模型能够在2块RTX 5090和2块RTX 5080 GPU的配置上，以每秒24.5个token的速度运行，并支持1M的上下文。该配置总计96 GB显存，通过利用RAM存储固定的专家模型，支持了一个186 GiB的专家混合（MoE）模型。该项目的代码库已在GitHub上开源。
- 发布
- 2026年9月7日 01:14
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
时间以 UTC 显示
更多信息
LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.
LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GiB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).
Measurements:
- 24.5 tok/s decode @8k, 27.0 tok/s u/0k .
- 159 tok/s prefill @27k.
Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.
The current box has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine and the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.
A single-command autoconfig derives the whole recipe from the weights + your hardware and explains every field — other RTX 50-series mixes should derive a working recipe too (only this box has been measured). Currently NVIDIA SM120 only.
Repo: https://github.com/kkontosis/LayerStoRm