Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp)
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Qwen model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A user successfully configured a Qwen 3.8 27B model to run at 50 tokens/second with a 100,000-token context window on a 16GB consumer GPU, specifically an RTX 4070 Ti SUPER. The setup utilized beellama.cpp and involved specific parameters like --ctx-size 100000, --fit-ctx 100000, and --flash-attn on to optimize performance and VRAM usage. Key cache precision was set to kvarn5 and value cache to kvarn4, with kv-tail-tokens 1024 to maintain recent token precision.
I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed.
🧠 Key Components
- Model: Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller from jrell on Hugging Face . It's a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget.
- Chat Template: I used the Jinja template from peculiar-ragdoll's Qwen-Sharp-Chat-Templates . It helps use fewer thinking tokens without noticeably affecting quality, which is great for speed.
- Inference Engine: This is crucial. I used beellama.cpp ( GitHub link ) because it supports the kvarn KV cache types, which are key to this optimization.
🖥️ Optimized llama-server Command (Windows)
Here's the polished command I'm running. The magic is in the kvarn cache settings and the tail precision.
%LLAMA_DIR%/llama-server.exe ^ -m %MODEL_PATH% ^ -a %MODEL_NAME% ^ --port 11434 ^ --temp 1.0 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.0 ^ --presence-penalty 0.0 ^ --repeat-penalty 1.0 ^ --parallel 1 ^ --n-gpu-layers 99 ^ --batch-size 1024 ^ --ubatch-size 256 ^ --flash-attn on ^ --spec-type draft-mtp ^ --spec-draft-n-max 2 ^ --cache-type-k kvarn5 ^ <-- Key: Higher precision for K cache --cache-type-v kvarn4 ^ <-- Key: Balanced precision for V cache --kv-tail-tokens 1024 ^ <-- Keeps recent tokens at full precision --ctx-size 100000 ^ --fit-ctx 100000 ^ --jinja ^ --chat-template-kwargs "{\"preserve_thinking\": true, \"reasoning_effort\":\"medium\"}" ^ --chat-template-file %MODEL_JINJA% ^ --no-mmproj-offload ^ --threads 7 ^ --threads-batch 8 ^ --metrics ^ --verbosity 3 ^ --perf
📊 Results & Optimization Notes
Metric Result Note Generation Speed 47-50 tokens/second Excellent for a 27B dense model. Context Window 100,000 tokens Successfully pushed from 88k by optimizing the cache. VRAM Usage ~15.93 GB (70 MB free) Perfectly tuned to the limit for maximum context. KV Cache Type kvarn5 (K) / kvarn4 (V) Uses the kvarn types from beellama. The asymmetric mix balances memory and quality. Precision Tail --kv-tail-tokens 1024 This is key. It keeps recent tokens at higher precision, preserving output quality. What I Optimized:
- KV Cache Quantization: Moving from kvarn5/kvarn5 to kvarn5/kvarn4 saved ~6% VRAM, allowing the context size to increase from 88k to 100k with minimal quality loss.
- Batch Sizes: Set --batch-size 1024 and --ubatch-size 256 to balance prompt processing speed and VRAM usage.
- Threading: Adjusted --threads 7 and --threads-batch 8 for my Ryzen 7 CPU.
- Speculative Decoding: Using --spec-type draft-mtp with 2 draft tokens (the model supports this) gave a huge speed boost.
The near-lossless kvarn quantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible.
Hopefully, this helps anyone trying to squeeze maximum performance out of a 16GB card! Cheers.