Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
Qwen 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一篇最新论文介绍了一种针对稀疏MoE推理模型（特别是Qwen 35B A4B+）的优化方法。该方法无需重新训练或微调，只需在运行时调整路由器，以扩大后期Transformer层中的专家选择预算（N≥K），而早期层保持不变。通过对额外专家应用线性衰减因子，可以将推理token减少8.5%。这项技术将Qwen 3.6 35B A3B模型转换为Qwen 3.6 35B A4B+。
I want to share a short paper just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models.
The idea: Instead of retraining anything, we just tweak the router at runtime . Specifically, we expand the expert selection budget (N≥K N ≥ K ) only in the late transformer layers, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes Qwen 3.6 35B A4B+ !
What we found — "Succinct Convergence": When you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the same correct answer via significantly shorter reasoning trajectories.
Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B):
- 📉 8.5% reduction in mean reasoning tokens
- ⚡ 10.9% drop in latency (p=6.5×10−6)
- 🎯 Accuracy unchanged ( 84.5% vs 84.0% native, p=0.77 — statistically indistinguishable )
- 🆓 Zero training cost — pure inference-time routing modification
Links:
- 📄 Paper: https://zenodo.org/records/22255483
there you can also take a look to my github repo (with beta version code) and the detailed json results of MMLU-Pro benchmark.
In the future i hope i can make same experimentation with a larger model like DeepSeek V4 Flash Q2.0
I'm a Non-native english speaker, part of this post was generated , for translation reason with the help of AI.