50% tg increase with offloading "hot" experts to VRAM
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
Llama 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一位开发者通过将“热门”专家模型卸载到VRAM，在llama.cpp中为MoE模型（如Qwen 3.8 Flash Next）实现了50%的性能提升（从20 t/s提高到30 t/s）。此优化在模型无法完全载入VRAM时尤为有效。这个由Opus开发的llama.cpp分支已在编码工作负载上进行测试，并且仅在模型无法完全适应VRAM时才有用，但其被上游接受的可能性尚不确定。
I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next.
The idea is simple: instead of offloading entire layers to the GPU, I offload only the “hot” experts. I found that certain groups of experts remain relatively stable across coding, refactoring, and code-review workloads.
A couple of important caveats: this llama.cpp fork has been tested only on coding workloads, and it’s useful only when the full model cannot fit in VRAM. Will upstream ever accept it? Probably not. Opus did the low-level implementation, and I don’t feel like showing up to explain every line