Qwen 3.8 27B UD-IQ4_XS even faster on 16GB CUDA

A new development, building on Raymond's KV cache streaming fork, allows Qwen 3.8 27B UD-IQ4_XS to run even faster on 16GB CUDA. The innovation involves speculative decoding (MTP or DFlash2) by ejecting the speculative model from VRAM when not fully utilized and reloading it when context is low. Optimal performance is achieved by ejecting the model after a certain number of pages (256 bytes each) once KV streaming begins.

This report details a method for speculative decoding on 16GB CUDA, unlike earlier approaches that did not dynamically manage VRAM by ejecting and reloading the speculative model.