Qwen3.6-27B: NVFP4/FP8 agent loops vs flawless BF16. Config or quant issue?

Hi everyone,

I'm trying to determine if I'm dealing with a misconfiguration in my stack or if this is an inherent limitation of current quantization methods for agentic workflows. I recently set up a dedicated rig with an RTX PRO 6000 Blackwell and have been benchmarking Qwen3.6-27B, but I'm hitting severe reliability issues with quantized models that don't exist in BF16.

Hardware & Software Stack

- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (Power limited to 450W out of 600W TDP)

- Driver: 610.43.02

- CPU/RAM: Ryzen 9 7950X, 2x64GB DDR5

- OS: Ubuntu Server 24.04 (isolated bare-metal environment)

- CUDA: 13.0

- Inference Engine: vLLM 0.24.0 (Running natively as a systemd service, no Docker)

- Model: Qwen3.6-27B (Official chat template applied consistently)

Environment Variables

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True OMP_NUM_THREADS=12 VLLM_USE_DEEP_GEMM=0 FLASHINFER_CUDA_ARCH_LIST=12.0f

vLLM Launch Command

vllm serve Qwen/Qwen3.6-27B \ --served-model-name "local/qwen3.6-27b" \ --max-model-len 262144 \ --max-num-seqs 8 \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --disable-custom-all-reduce \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --chat-template /var/lib/vllm/chat-templates/qwen3.6/chat_template-unsloth.jinja \ --default-chat-template-kwargs '{"preserve_thinking":true,"enable_thinking":true}' \ --generation-config vllm \ --override-generation-config '{"bos_token_id":248044,"do_sample":true,"eos_token_id":[248046,248044],"pad_token_id":248044,"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \ --host 0.0.0.0 \ --port 8000

(Note: When testing quantized versions, I simply swap the model path to the respective NVFP4/FP8 checkpoints while keeping all flags identical).

The Baseline: BF16 Works Flawlessly

The BF16 version works perfectly out of the box. I've integrated it with OpenCode, VS Code Copilot (via OAI-compatible provider extension), and oh-my-pi. No complaints whatsoever; agentic loops complete successfully and reasoning is stable.

The Problem: NVFP4 & FP8 Degradation

After switching to NVIDIA's fresh NVFP4 quantization , the model becomes unreliable in "Thinking" mode (enable_thinking=true, preserve_thinking=true). I am using the official sampling parameters from the Qwen3.6-27B repo (as seen in the override config above).

Observed symptoms: 1. Mid-task halting: The agent simply stops generating mid-workflow. Prompting "continue" resumes it, but this never happens in BF16. 2. Failure loops: When the model encounters a problem it cannot solve, it gets stuck in a loop repeating the same failure message over and over until context exhaustion. Repetition penalty doesn't fix it: Increasing repetition_penalty from 1.0 to 1.05–1.1 doesn't break the loop - it just makes the model alternate between TWO different failure phrases (e.g., "I tried to do X and failed" → "Unable to do X" → "I tried to do X and failed" → "Unable to do X" ) instead of repeating one phrase.

FP8 Shows Similar (But Rarer) Issues

I also tested the official FP8 checkpoint from the Qwen repo . The exact same halting and looping behaviors occur, though significantly less frequently than with NVFP4. Again, BF16 remains completely unaffected.

My Questions

For those running Blackwell/professional GPUs with vLLM 0.24.0+: