CoWindow and MassAlloc Attention: collective causal coverage and distribution-adaptive compute [R]
The authors of CoWindow (CoWA) and MassAlloc (MALA) Attention presented their research on reducing redundant computation in attention mechanisms. Their evaluations on models scaling from 0.6B to 14B, and continued-training experiments at 32B, showed significant FLOPs reduction. Specifically, CoWA decreased total training FLOPs by 28.5% and MALA by 23.1% at 14B with 32K context, while maintaining comparable model capabilities to FullAttn.
This report details the specific FLOPs reduction percentages for CoWA (28.5%) and MALA (23.1%) at 14B with 32K context, unlike general claims of efficiency gains.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Sep 29, 2026, 05:16 UTC
IngestedOffset at this time: UTC+0Sep 29, 2026, 08:00 UTC
- Published
- Sep 29, 2026, 05:16
- Ingested
- Sep 29, 2026, 08:00
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
Discussion trend
The percentage is based on collected discussion signal, not new comments or independent people. The curve only compares the same topic across time.
I'm one of the authors of two recent papers exploring different sources of redundant computation in attention. I'd like to share the ideas and hear feedback from people working on long-context models and attention kernels.
CoWindow Attention (CoWA) distributes distant context across KV heads using complementary windows, while sharing local and prefix-sink windows. Each head attends sparsely, but the union of their visible positions covers the full causal history. The pattern is position-defined and requires no learned router or indexer.
Paper: https://arxiv.org/abs/2609.32704
MassAlloc Attention (MALA) retains full causal QK scoring, then uses attention's own softmax statistics to decide whether to execute subsequent computation for a tile. It reduces low-contribution post-score work, using a common tolerance across training and inference.
Paper: https://arxiv.org/abs/2609.32712
Both support training forward/backward and inference prefill/decoding. At 128K tokens on 8 H100 GPUs with TP=8, attention-operator speedups relative to FullAttn are:
Method Forward Backward Decode CoWA 7.4x 8.6x 3.0x MALA 2.2x 3.0x 1.6x These measurements are for the attention operators, not end-to-end model speedups.
We evaluated scaling from 0.6B to 14B and conducted separate continued-training experiments at 32B. At 14B with 32K context, total training FLOPs decreased by 28.5% for CoWA and 23.1% for MALA, with model capabilities comparable to FullAttn on the reported evaluations.
Two distinctions that matter: collective coverage does not imply identical head-wise interactions or outputs to FullAttn, and MALA still pays for full causal QK scoring. Neither result establishes universal lossless equivalence to dense attention.
I'd be interested in feedback on workloads that might stress collective coverage, or attention distributions where adaptive post-score allocation could be less effective. Happy to discuss implementation and evaluation details.