[R] Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Complex Kimi Delta Attention (CKDA) enhances the expressivity of KDA by combining a single delta-rule transformation with a channel-wise gate's reflection, enabling 2D rotations. This involves extending KDA's parameter ranges, allowing gates in [-1,1] and the delta-rule coefficient β in [0,2]. CKDA maintains stability and efficiency, achieving state-tracking expressivity comparable to DeltaProduct2. It outperforms Transformers and other linear RNNs in language modeling and shows strong length extrapolation on S3, S4, and periodic audio continuation tasks.