Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

adamas:-hadamard-sparse-attention-for-efficient-long-context-inference

Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

[Submitted on 21 Oct 2025 (v1), last revised 6 Nov 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce textbf{Adamas}, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top-$k$ candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to $6.5times$ self-attention speedup and $1.7times$ end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025.

Submission history

From: Siyuan Yan [view email]
[v1] Tue, 21 Oct 2025 08:44:47 UTC (1,170 KB)
[v2] Fri, 6 Nov 2026 10:07:01 UTC (2,681 KB)