N-gram vs Experts explained

Since Qwen's dropped the Qwen4Exp architecture bomb that focus on offloading parameters to n-gram instead of pure mixture of experts, I dug into this and learned quite a lot. Here's the summary. Expect mistakes from human's writing lol.

TLDR: MoEs do reasoning, N-grams do recalling. At the current tech frontier, N-gram can offload upto ~25% weight before losing advantage and we get most benefit from using SSD to store those instead of RAM. So 176B models became 125 (RAM) +51B (SSD) instead.

Long version's below.

Experts (moe) are arithmetic work. A router inspects model's hidden state, chooses a few feed-forward blocks, and those blocks multiply. The choice arrives late, after the layer has already begun, and the payload is large so sending experts to a disk is usually slow. The machine discovers what it needs too late and then hauls gigabytes across a slow bus only to compute them at once.

An n-gram table is memory of another kind. It stores vectors for short local phrases, addressed by a hash of the last few tokens. Those addresses exist as soon as the tokens exist. The network does not multiply the whole table. It gathers a handful of rows, often only a few kB, and folds them into the stream. Qwen 3.8 Flash Next (Qwen4Exp) keeps about 125B parameters in the moe network, another 51B in n-gram table, and activates only about ~6B for each token. The extra 51B are capacity, not extra arithmetic work. So it runs fast like a 125B-A6B model with a caveat but taking advantage of the 176B trained parameters.

Experts do reasoning work, n-gram do recalling work. But you cannot be lazy and only recalling without reasoning to get the job done. Or the quality will drop, reasoning llm became memory fetching machine instead. There is a limit to how far you can offload experts into the table. Under a fixed budget, giving the table roughly 20-25% of the total parameters tends to help. Early layers stop wasting depth on rebuilding common local patterns and the deeper stack can reason.