Higher acceptance length, slower prose: Ling’s n=1/2/3 MTP test on one Spark

The missing control is visible in sudoingX’s Ling-3.0-flash benchmark graphics. The earlier table leaves Ling’s no-speculation baseline as “not measured.” The later code/prose graphic fills it in: about 23 tok/s without the drafter, against 40.9 on code and 38.7 on prose with MTP n=1, at short prompts. That makes the tuning claim easier to inspect.

The later graphic compares short code and prose workloads with and without drafting. A separate correction in the repository isolates CUDA graphs: the earlier “two flags” result changed graphs and multi-token prediction (MTP) together, so it could not tell you which change produced the gain.

The corrected August 22 measurements on one 128GB DGX Spark, using the official INT4 checkpoint and the vendor’s vLLM fork, were:

Configuration Reported tok/s, short coding task Eager execution, MTP off 20.8 CUDA graphs, MTP off 22.9 CUDA graphs, MTP n=1 40.9 That is roughly 10% more throughput from graphs relative to the eager baseline, followed by roughly 79% more from MTP relative to the graph baseline. Those percentages have different denominators.

Then comes the setting that is tempting to turn up. Here n is num_speculative_tokens: how many draft tokens are proposed per step. vLLM’s documented acceptance-length metric includes a bonus token per verification step, so an average above 1 with n=1 is possible. The creator reports the metric from a vendor fork; its exact historical counting implementation is not supplied.

MTP setting n=1 n=2 n=3 Mean acceptance length 1.87 2.39 2.77 Prose, 512-token output, tok/s 38.7 34.8 33.6 Prose, 2,048-token output, tok/s 37.3 33.6 31.6 Code throughput stayed approximately flat within the reported run-to-run variation. Prose got slower as the mean accepted draft length increased. Acceptance length is not an acceptance percentage, and it is not the optimization objective.

sudoingX describes the configuration in the deployment thread; the pinned benchmark notes contain both tables. These are the author’s measurements, without an independent rerun here. The sweep describes streaming and server-side token counts but does not fully specify its timing denominator, so the numbers should stay labeled as reported throughput.

For this checkpoint and these workloads, n=1 was the useful setting. The transferable experiment is to isolate the no-MTP baseline, then compare draft settings on the output types you actually generate.