[Paper] FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models

FactorEngram is a novel factorized n-gram memory with basis-level contextual gating designed to enhance large language models (LLMs). Unlike existing lookup-based memory systems like Engram, FactorEngram retrieves sparsity-regularized coefficients over a shared dictionary of basis vectors, allowing related patterns to reuse common components. This approach enables context-specific modulation of individual memory components, addressing the limitations of monolithic embeddings. FactorEngram improves language modeling and downstream task performance on 340M- and 1B-parameter Transformer backbones, with optimal insertion before the attention sublayer in middle layers.

Unlike existing lookup-based memory designs that treat each retrieved embedding as a monolithic unit, FactorEngram introduces basis-level contextual gating for selective component readout.