Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph
Llama 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一位用户在配备 128 GB M5 Max 芯片的 MacBook Pro 上，成功运行了 Qwen3.8-Flash-Next (79 GB, 2-bit) 模型，上下文长度达 350K，持续了 3.5 小时。该设置使用了 llama.cpp b10686 (Metal) 和通过 YaRN 实现的 358,400-token 上下文槽。模型在前 100K 上下文表现良好，但之后在长尾任务中开始出现角色混淆，将用户消息与自身输出混杂。性能下降的可能原因包括 2-bit 量化或预览模型的长上下文质量。
Setup: MacBook Pro M5 Max, 128 GB unified, macOS 26.5.2 · llama.cpp b10686 (Metal, 12 threads, batch 2048, flash-attn, kv-unified, ngram-mod spec decode) · Qwen3.8-Flash-Next UD-Q2_K_XL (Unsloth), 78.9 GB · 358,400-token context slot via YaRN from the native 262,144, fp16 KV. Weights + full 350K KV fit under the default 96 GB GPU wired limit — no sysctl hack.
The session: one slot, 100 turns, two conversations. Conv 1 grew 0 → 48K ctx on prefix reuse; after a ~20 min idle the slot kept only its 5.5K system prefix, so the next turn cold-prefilled the whole 105K prompt in 333 s — the run's longest prefill — and the conversation kept growing to 169,425 ctx, the session's deepest point (350K was slot capacity, never filled). Slot reset; conv 2 grew to ~125K where I stopped capture.
The graph: x = slot context size where each measurement happened; y = printed tokens/s, log scale (the two phases span ~2 decades). Green = prompt processing, red = token generation. Dots = in-flight checkpoints, squares = per-turn finals. No smoothing, no fitting.
- Prefill (green): the smooth top curve is cold prefills — 1,561 t/s at the first checkpoint (5.6K ctx), tapering to 318 t/s at 111K as the KV fills. The green band below is what a normal turn looks like: a few thousand new tokens at each depth (77–854 t/s, out to 169K ctx), because prefix reuse means only the delta gets prefilled.
- Decode (red): one clean taper — ~30–35 t/s at small ctx → ~21 at 45K → 13–15 at 100–125K → 11.5 t/s at 169K. The dip to 7.7 t/s around ~140K is macOS Low Power Mode; still usable.
One caveat on the decode numbers: they are effective throughput with ngram-mod spec decode enabled (draft acceptance ranged 0–81% depending on content), not base-model speed.
Practical read: with prefix reuse a turn's prefill is seconds; the 5.5-minute prefill happened exactly once, after an idle gap. Decode stayed interactive out to 169K ctx.
Experience: strong for the first ~100K ctx. Past that, on long-tail tasks, it started mixing up user messages with its own prior output (role confusion), worsening with use. Ruled out: KV quant (ran fp16) and rope extrapolation (worst turns well under native 262K). Remaining suspects: the 2-bit quant and/or preview-model long-context quality.