DeepSeek V4 Flash with DSpark via SGLang

Hello guys. Sharing my experience with deploying DS-4-Falsh with DSpark on HGX-H200

For context of my setups etc (including how the hell i have access to H200), you can read my previous posts

So mostly, its about my comparison of marlin moe_backend, that i used on my previous main deploy config with EAGLE (1-1-2), and flashinfer i saw in official blog ( https://www.lmsys.org/blog/2026-07-06-dspark-sglang ).

In general i can say, that DSpark absolutely faster then EAGLE. Didnt save exact tables of comparison, but here agents conclusion i found in older chats

1. DSpark 3.2x faster at bs=1 2. DSpark +46% throughput at bs=24 3. EAGLE maintains ~97-100% acceptance at all batch sizes, but only drafts 2 tokens/step. DSpark's acceptance drops with batch size (100% → 88% at bs=24), but it drafts 6 tokens/step — so even at 88%, it accepts 5.27 tokens/step vs EAGLE's 1.95. That's 2.7× more accepted tokens per step, which is where the 46% throughput gain comes from.

Here is command and my benches to test inference speed etc. Won`t say that im experienced one, so im free to take your advices and all

docker run -d \ --name deepseek-v4-flash-X \ --restart unless-stopped \ --gpus '"device=X,X,X,X"' \ --shm-size 32g \ --ipc=host \ -e SGLANG_RAGGED_VERIFY_MODE=static \ -e SGLANG_PREP_IN_CUDA_GRAPH=0 \ -v /data/models/deepseek-v4-flash-dspark:/model \ -p 500X:30000 \ lmsysorg/sglang:dev-dspark \ sglang serve \ --trust-remote-code \ --model-path /model \ --served-model-name deepseek-v4-flash \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --moe-runner-backend marlin \ --mem-fraction-static 0.88 \ --cuda-graph-max-bs-decode 24 \ --max-running-requests 24 \ --kv-cache-dtype fp8_e4m3 \ --enable-metrics \ --enable-cache-report \ --reasoning-parser deepseek-v4 \ --tool-call-parser deepseekv4 \ --speculative-algorithm DSPARK \ --enable-hierarchical-cache \ --hicache-ratio 4 \ --hicache-size 0 \ --hicache-write-policy write_through

In short, thats final results i got, when asked my AI to benchmark

TTFT Mean (ms — lower = better)

Input Conc Flashinfer Marlin Delta 5K 1 257 260 tie 5K 5 327 330 tie 5K 10 380 307 Marlin -19% 5K 20 661 597 Marlin -10% 10K 1 255 266 tie 10K 5 410 410 tie 10K 10 369 398 FI -7% 10K 20 645 548 Marlin -15% 20K 1 274 286 tie 20K 5 433 427 tie 20K 10 468 442 tie 20K 20 757 797 tie

TTFT P99 (ms — lower = better)

Input Conc Flashinfer Marlin Under 5s? 5K 1 270 275 ✅ ✅ 5K 5 485 483 ✅ ✅ 5K 10 490 411 ✅ ✅ 5K 20 5552 4438 ❌ ❌ 10K 1 298 285 ✅ ✅ 10K 5 642 507 ✅ ✅ 10K 10 498 510 ✅ ✅ 10K 20 901 804 ✅ ✅ 20K 1 299 323 ✅ ✅ 20K 5 601 538 ✅ ✅ 20K 10 741 766 ✅ ✅ 20K 20 1803 1886 ✅ ✅

Accept Length (max 6.0)

Input Conc Flashinfer Marlin 5K 1 5.22 5.52 5K 20 5.22 5.43 10K 1 5.23 5.44 10K 20 5.30 5.47 20K 1 5.30 5.47 20K 20 5.36 5.50 Here`s scripts i run to test

===================================== INSTANCE: flashinfer ===================================== -- Prefill warmup (per input shape) -- warmup: input=5000 conc=1 prompts=2 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. warmup: input=10000 conc=1 prompts=2 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} warmup: input=20000 conc=1 prompts=2 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. -- Batch decode warmup (c=20) -- warmup: input=5000 conc=20 prompts=24 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. warmup: input=20000 conc=20 prompts=24 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. -- Benchmark -- --- flashinfer input=5000 conc=1 prompts=10 --- Peak concurrent requests: 2 Accept length: 5.22 Mean TTFT (ms): 257.42 P90 TTFT (ms): 263.26 P99 TTFT (ms): 270.24 --- flashinfer input=5000 conc=5 prompts=15 --- Peak concurrent requests: 8 Accept length: 5.22 Mean TTFT (ms): 326.97 P90 TTFT (ms): 482.28 P99 TTFT (ms): 484.74 --- flashinfer input=5000 conc=10 prompts=30 --- Peak concurrent requests: 15 Accept length: 5.22 Mean TTFT (ms): 379.88 P90 TTFT (ms): 487.03 P99 TTFT (ms): 490.11 --- flashinfer input=5000 conc=20 prompts=60 --- Peak concurrent requests: 25 Accept length: 5.22 Mean TTFT (ms): 661.43 P90 TTFT (ms): 531.47 P99 TTFT (ms): 5552.02 --- flashinfer input=10000 conc=1 prompts=10 --- Peak concurrent requests: 2 Accept length: 5.23 Mean TTFT (ms): 255.17 P90 TTFT (ms): 289.91 P99 TTFT (ms): 297.77 --- flashinfer input=10000 conc=5 prompts=15 --- Peak concurrent requests: 8 Accept length: 5.25 Mean TTFT (ms): 409.59 P90 TTFT (ms): 471.23 P99 TTFT (ms): 642.26 --- flashinfer input=10000 conc=10 prompts=30 --- Peak concurrent requests: 14 Accept length: 5.27 Mean TTFT (ms): 368.90 P90 TTFT (ms): 449.70 P99 TTFT (ms): 497.91 --- flashinfer input=10000 conc=20 prompts=60 --- Peak concurrent requests: 27 Accept length: 5.30 Mean TTFT (ms): 644.57 P90 TTFT (ms): 891.86 P99 TTFT (ms): 901.37 --- flashinfer input=20000 conc=1 prompts=10 --- Peak concurrent requests: 3 Accept length: 5.30 Mean TTFT (ms): 274.03 P90 TTFT (ms): 297.82 P99 TTFT (ms): 299.33 --- flashinfer input=20000 conc=5 prompts=15 --- Peak concurrent requests: 9 Accept length: 5.31 Mean TTFT (ms): 432.51 P90 TTFT (ms): 523.52 P99 TTFT (ms): 601.07 --- flashinfer input=20000 conc=10 prompts=30 --- Peak concurrent requests: 15 Accept length: 5.33 Mean TTFT (ms): 468.08 P90 TTFT (ms): 723.76 P99 TTFT (ms): 741.19 --- flashinfer input=20000 conc=20 prompts=60 --- Peak concurrent requests: 26 Accept length: 5.36 Mean TTFT (ms): 756.73 P90 TTFT (ms): 1132.48 P99 TTFT (ms): 1803.27 ===================================== INSTANCE: marlin ===================================== -- Prefill warmup (per input shape) -- warmup: input=5000 conc=1 prompts=2 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. warmup: input=10000 conc=1 prompts=2 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. warmup: input=20000 conc=1 prompts=2 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. -- Batch decode warmup (c=20) -- warmup: input=5000 conc=20 prompts=24 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} warmup: input=20000 conc=20 prompts=24 /sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: sglang.bench_serving is deprecated and will be removed in a future release; use sglang.benchmark.serving instead (e.g. python -m sglang.benchmark.serving). warnings.warn( [transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'attention_factor'} -- Benchmark -- --- marlin input=5000 conc=1 prompts=10 --- Peak concurrent requests: 2 Accept length: 5.52 Mean TTFT (ms): 260.38 P90 TTFT (ms): 271.65 P99 TTFT (ms): 274.51 --- marlin input=5000 conc=5 prompts=15 --- Peak concurrent requests: 8 Accept length: 5.50 Mean TTFT (ms): 330.12 P90 TTFT (ms): 481.13 P99 TTFT (ms): 483.15 --- marlin input=5000 conc=10 prompts=30 --- Peak concurrent requests: 15 Accept length: 5.47 Mean TTFT (ms): 306.77 P90 TTFT (ms): 408.24 P99 TTFT (ms): 410.59 --- marlin input=5000 conc=20 prompts=60 --- Peak concurrent requests: 26 Accept length: 5.43 Mean TTFT (ms): 597.49 P90 TTFT (ms): 512.82 P99 TTFT (ms): 4438.23 --- marlin input=10000 conc=1 prompts=10 --- Peak concurrent requests: 2 Accept length: 5.44 Mean TTFT (ms): 265.65 P90 TTFT (ms): 282.26 P99 TTFT (ms): 284.74 --- marlin input=10000 conc=5 prompts=15 --- Peak concurrent requests: 8 Accept length: 5.45 Mean TTFT (ms): 410.25 P90 TTFT (ms): 505.74 P99 TTFT (ms): 507.39 --- marlin input=10000 conc=10 prompts=30 --- Peak concurrent requests: 14 Accept length: 5.46 Mean TTFT (ms): 397.78 P90 TTFT (ms): 507.21 P99 TTFT (ms): 509.92 --- marlin input=10000 conc=20 prompts=60 --- Peak concurrent requests: 27 Accept length: 5.47 Mean TTFT (ms): 548.28 P90 TTFT (ms): 731.57 P99 TTFT (ms): 803.65 --- marlin input=20000 conc=1 prompts=10 --- Peak concurrent requests: 3 Accept length: 5.47 Mean TTFT (ms): 286.49 P90 TTFT (ms): 312.97 P99 TTFT (ms): 322.83 --- marlin input=20000 conc=5 prompts=15 --- Peak concurrent requests: 9 Accept length: 5.47 Mean TTFT (ms): 427.42 P90 TTFT (ms): 535.50 P99 TTFT (ms): 537.50 --- marlin input=20000 conc=10 prompts=30 --- Peak concurrent requests: 15 Accept length: 5.48 Mean TTFT (ms): 441.67 P90 TTFT (ms): 746.80 P99 TTFT (ms): 765.58 --- marlin input=20000 conc=20 prompts=60 --- Peak concurrent requests: 26 Accept length: 5.50 Mean TTFT (ms): 796.88 P90 TTFT (ms): 1594.48 P99 TTFT (ms): 1886.39