Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Qwen model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A recent paper explores an optimization for sparse Mixture-of-Experts (MoE) reasoning models, specifically Qwen 35B A4B+. The method involves tweaking the router at runtime to expand the expert selection budget (N≥K) in the late transformer layers, while early layers remain untouched. This linear decay factor applied to extra experts can reduce reasoning tokens by 8.5% without requiring retraining or fine-tuning. This technique transforms Qwen 3.6 35B A3B into Qwen 3.6 35B A4B+.
I want to share a short paper just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models.
The idea: Instead of retraining anything, we just tweak the router at runtime . Specifically, we expand the expert selection budget (N≥K N ≥ K ) only in the late transformer layers, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes Qwen 3.6 35B A4B+ !
What we found — "Succinct Convergence": When you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the same correct answer via significantly shorter reasoning trajectories.
Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B):
- 📉 8.5% reduction in mean reasoning tokens
- ⚡ 10.9% drop in latency (p=6.5×10−6)
- 🎯 Accuracy unchanged ( 84.5% vs 84.0% native, p=0.77 — statistically indistinguishable )
- 🆓 Zero training cost — pure inference-time routing modification
Links:
- 📄 Paper: https://zenodo.org/records/22255483
there you can also take a look to my github repo (with beta version code) and the detailed json results of MMLU-Pro benchmark.
In the future i hope i can make same experimentation with a larger model like DeepSeek V4 Flash Q2.0
I'm a Non-native english speaker, part of this post was generated , for translation reason with the help of AI.