WarpState: experimenting with dual-timescale associative memory + local attention for long-running LLMs

WarpState is an experimental approach for long-running LLMs, combining dual-timescale associative memory with local attention. This method builds upon local attention, adding either one associative memory bank or the more advanced WarpState dual memory. The developer is interested in insights from those familiar with models like Infini-attention, GLA, DeltaNet, RetNet, RWKV, xLSTM, or recurrent-depth models.

This report details an experimental approach that uniquely combines dual-timescale associative memory with local attention, unlike other methods that typically use only one associative memory bank.