Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache

Hello,

I'm a bit obsessed with cache management on local LLMs.

For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes.

That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment:

- Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup)

- Hydrates X stable contexts of Y tokens each in order to completely fill the cache

- Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found

It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work.

My results

This is the result from my A/B test, control (my previous prod) vs my fixed prod.

aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096):

── Retention under pressure ── capacity: 2,023,924 tokens retained contexts: 27/80 retained tokens: 1,052,025 retained % capacity: 51.98% oldest evicted: context #52 (older contexts evicted)

With the dedupe + boundfix patches applied (retention 0):

── Retention under pressure ── capacity: 2,047,043 tokens retained contexts: 77/80 retained tokens: 3,000,048 retained % capacity: 146.56% oldest evicted: context #2 (older contexts evicted)

How this can matter to you

This allows you to exactly know how much tokens your cache actually holds.

For most of us, cache management is a black box; this allows you to get ground truth.

And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes ).

The engine's own advertised number is wrong, and this tool finds the real value.

This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management

It works under one big assumption though: most recent contexts should be preserved as much as possible.

What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines.

How to launch

1. Clone the repo