Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Llama model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A user successfully ran Qwen3.8-Flash-Next (79 GB, 2-bit) with a 350K context for 3.5 hours on a 128 GB M5 Max MacBook Pro. The setup utilized llama.cpp b10686 with Metal and a 358,400-token context slot via YaRN. Performance was strong for the first 100K context, but beyond that, the model exhibited role confusion, mixing user messages with its own output. Suspects for this degradation include the 2-bit quantization or the preview model's long-context quality.
Setup: MacBook Pro M5 Max, 128 GB unified, macOS 26.5.2 · llama.cpp b10686 (Metal, 12 threads, batch 2048, flash-attn, kv-unified, ngram-mod spec decode) · Qwen3.8-Flash-Next UD-Q2_K_XL (Unsloth), 78.9 GB · 358,400-token context slot via YaRN from the native 262,144, fp16 KV. Weights + full 350K KV fit under the default 96 GB GPU wired limit — no sysctl hack.
The session: one slot, 100 turns, two conversations. Conv 1 grew 0 → 48K ctx on prefix reuse; after a ~20 min idle the slot kept only its 5.5K system prefix, so the next turn cold-prefilled the whole 105K prompt in 333 s — the run's longest prefill — and the conversation kept growing to 169,425 ctx, the session's deepest point (350K was slot capacity, never filled). Slot reset; conv 2 grew to ~125K where I stopped capture.
The graph: x = slot context size where each measurement happened; y = printed tokens/s, log scale (the two phases span ~2 decades). Green = prompt processing, red = token generation. Dots = in-flight checkpoints, squares = per-turn finals. No smoothing, no fitting.
- Prefill (green): the smooth top curve is cold prefills — 1,561 t/s at the first checkpoint (5.6K ctx), tapering to 318 t/s at 111K as the KV fills. The green band below is what a normal turn looks like: a few thousand new tokens at each depth (77–854 t/s, out to 169K ctx), because prefix reuse means only the delta gets prefilled.
- Decode (red): one clean taper — ~30–35 t/s at small ctx → ~21 at 45K → 13–15 at 100–125K → 11.5 t/s at 169K. The dip to 7.7 t/s around ~140K is macOS Low Power Mode; still usable.
One caveat on the decode numbers: they are effective throughput with ngram-mod spec decode enabled (draft acceptance ranged 0–81% depending on content), not base-model speed.
Practical read: with prefix reuse a turn's prefill is seconds; the 5.5-minute prefill happened exactly once, after an idle gap. Decode stayed interactive out to 169K ctx.
Experience: strong for the first ~100K ctx. Past that, on long-tail tasks, it started mixing up user messages with its own prior output (role confusion), worsening with use. Ruled out: KV quant (ran fp16) and rope extrapolation (worst turns well under native 262K). Remaining suspects: the 2-bit quant and/or preview-model long-context quality.