KV cache might be a bigger problem for local models than parameter count

While discussions often focus on fitting larger models into local hardware based on parameter count, the KV cache presents a significant memory bottleneck for long context inference. Each new token adds key and value states, causing models that initially fit comfortably in VRAM to struggle with 100k or 200k context. This raises the question of optimizing models based on their memory requirements rather than just parameter count.