PSA for M5Ultra owners running LLMs: set your prefill step to 8192

For M5Ultra owners running LLMs, setting the prefill step size to 8192 is recommended. This adjustment significantly impacts both PP performance and the drafter's ability to fetch logits for MTP, also affecting dflash. Tests show substantial gains up to 8k, with some engines like omlx supporting adaptive step sizes that can match or even exceed 8k performance from raw mlx-vlm, especially at 64k context and above, and offering a 10% improvement on smaller contexts.

This report provides specific performance benchmarks for M5Ultra users, unlike general advice, showing gains up to 8k and adaptive step size benefits.