Language Models Can Control Their Own Attention [R]
- 发布
- 09/05 06:07
- 收录
- 09/05 09:00
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
讨论趋势
百分比基于采集到的讨论信号，不代表新增评论数或独立参与人数。曲线仅用于同一话题在不同时段的比较。
声明式注意力（Declarative Attention, DA）是一种新协议，旨在解决语言模型在处理长上下文时，需要扫描整个KV缓存的效率低下问题。通过引导模型在其思维链中声明相关上下文，DA将生成过程分为三种模式，从而使推理引擎能够跳过大部分KV缓存读取。在15个长上下文任务中，对Gemma-4-31B和Qwen-3.6-27B进行的零样本评估显示，DA显著减少了总注意力令牌（52.0%, 31.1%），同时准确性下降幅度适中（1.27pp, 2.75pp）。这项技术为稀疏注意力开辟了新的方向。
Abstract
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: (full context), (a specific region), and (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore. Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) Cite as: arXiv:2609.02737 [cs.CL]
(or arXiv:2609.02737v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.02737 Focus to learn more