Applying Sliding Window Attention to pretrained LLMs at inference time [P]

A developer has implemented Sliding Window Attention (SWA) for pretrained Hugging Face causal LLMs at inference time. This implementation significantly reduces memory usage, with SWA-64 using approximately 3.5 MB across 16K, 32K, and 64K context sizes, compared to 923 MB and 1.84 GB for full KV at 16K and 32K respectively. Additionally, SWA-64 decreased TPOT from about 38.4 ms to 30.5 ms at 16K. The developer is seeking feedback and further experiments.