A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts

a-statistical-theory-of-gated-attention-through-the-lens-of-hierarchical-mixture-of-experts

A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts

[Submitted on 1 Feb 2026 (v1), last revised 14 Nov 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to improve performance, a gated attention model that leverages a gating mechanism within the multi-head self-attention has recently been proposed as a promising alternative. Gated attention has been empirically demonstrated to increase the expressiveness of low-rank mapping in standard attention and even to eliminate the attention sink phenomenon. Despite its efficacy, a clear theoretical understanding of gated attention’s benefits remains lacking in the literature. To close this gap, we rigorously show that each entry in a gated attention matrix or a multi-head self-attention matrix can be written as a hierarchical mixture of experts. By recasting learning as an expert estimation problem, we demonstrate that gated attention is more sample-efficient than multi-head self-attention. In particular, while the former needs only a polynomial number of data points to recover the model parameters, the latter requires exponentially many data points to achieve the same estimation error. Furthermore, our analysis also provides a theoretical justification for why gated attention is more effective at mitigating the attention sink phenomenon and the low-rank bottleneck when a gate is placed at the output of the scaled dot product attention or the value map rather than at other positions in the multi-head self-attention architecture.

Submission history

From: Huy Nguyen [view email]
[v1] Sun, 1 Feb 2026 22:22:13 UTC (114 KB)
[v2] Sat, 14 Nov 2026 17:16:01 UTC (234 KB)