[Submitted on 22 Jul 2026 (v1), last revised 4 Nov 2026 (this version, v3)]
Abstract:Transformers are known to have internal continuous symmetries that leave outputs invariant while altering how activations are distributed across coordinates, and hence how well they quantize. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our quantization strategy requires no specific calibration data, no quantization simulation, and adds a small inference overhead. As a proof of concept, we evaluate on WikiText-2 perplexity and zero-shot reasoning. On Qwen-2.5 0.5B, GaugeQuant reduces perplexity (e.g. $25.62 to 16.76$ under W4A16) and improves zero-shot accuracy. On LLaMA-2 7B, perplexity improves but downstream accuracy does not. The code is available at: this https URL.
Submission history
From: Miguel Pedra Bento [view email]
[v1] Wed, 22 Jul 2026 22:14:23 UTC (86 KB)
[v2] Fri, 24 Jul 2026 12:57:08 UTC (86 KB)
[v3] Wed, 4 Nov 2026 15:58:48 UTC (98 KB)
