Clustering in pure-attention hardmax transformers and its role in sentiment analysis

clustering-in-pure-attention-hardmax-transformers-and-its-role-in-sentiment-analysis

Clustering in pure-attention hardmax transformers and its role in sentiment analysis

[Submitted on 26 Jun 2024 (v1), last revised 17 Nov 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called textit{leaders}. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context’ by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.

Submission history

From: Albert Alcalde [view email]
[v1] Wed, 26 Jun 2024 16:13:35 UTC (394 KB)
[v2] Wed, 13 May 2026 09:54:58 UTC (749 KB)
[v3] Tue, 17 Nov 2026 15:11:58 UTC (749 KB)