Applying Sliding Window Attention to pretrained LLMs at inference time [P]

I've been working on a practical implementation of Sliding Window Attention (SWA) for pretrained Hugging Face causal LLMs.

The idea is simple: instead of allowing every generated token to attend to the complete historical KV cache, maintain a bounded cache consisting of:

attention sinks + recent sliding window

I implemented this as a reusable inference layer rather than modifying or retraining the model.

GitHub: https://github.com/oraby8/SWA

The implementation currently includes:

- bounded KV cache

- circular/ring-buffer storage

- attention sinks

- streaming prefill

- chunked attention masking

- autoregressive decoding

- Full Attention vs SWA benchmarking

- TTFT / TPOT / throughput measurements

- KV-cache memory measurements

One interesting result from my Qwen2.5-7B experiment:

Context Full KV SWA-64 16K ~923 MB ~3.5 MB 32K ~1.84 GB ~3.5 MB 64K OOM ~3.5 MB At 16K, SWA-64 also reduced TPOT from ~38.4 ms to ~30.5 ms in this setup.

However, there is an important trade-off: tasks requiring information far outside the active window can degrade. I'm currently investigating how much of this is inherent to SWA versus implementation/model-specific behavior.

I'm sharing the implementation mainly to get feedback from people working on LLM inference, KV-cache optimization, and long-context models.

I'd be particularly interested in:

- Which model architectures should I validate next?

- What failure cases should I benchmark?

- What would make this useful for existing HF inference workflows?

- Are there cache/attention implementation details I may be overlooking?