ASRN Adaptive Sparse Recurrence Network [N]

The ASRN (Adaptive Sparse Recurrence Network) is a copy layer designed for language models. It functions by identifying earlier occurrences of the current context through the use of learned hash tables. Once these occurrences are found, the ASRN copies the subsequent information. This method allows for memory usage that scales linearly with the sequence length, offering an efficient approach to handling context in language models.

This network introduces a novel copy layer that uses learned hash tables to find and copy earlier context, unlike previous methods.