SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks

soap-bubbles:-structured-weight-uncertainty-for-neural-networks

SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks

[Submitted on 22 Jun 2026 (v1), last revised 16 Nov 2026 (this version, v4)]

View PDF HTML (experimental)

Abstract:Structured weight-uncertainty can improve many aspects of deep learning, but it remains costly to estimate and difficult to implement. Here, we show that these issues can be addressed by adapting the SOAP optimizer. Our key idea is to run IVON, an existing diagonal-covariance variational method, in the eigenspace of SOAP’s preconditioner and then use the preconditioner to transform the diagonal estimate into a non-diagonal covariance. The resulting method has costs similar to those of SOAP and requires no drastic changes to training pipelines. We call the posteriors obtained in this way SOAP-Bubbles and our new optimizer Eigenspace-VON (EVON). We show that, for logistic regression, EVON can match the full Gaussian covariance and that, for language model pretraining, it yields significantly better results than existing diagonal-covariance methods. Our work makes it easier to estimate more expressive posterior distributions for deep learning at scale.

Submission history

From: Adrian Robert Minut [view email]
[v1] Mon, 22 Jun 2026 13:56:59 UTC (4,445 KB)
[v2] Thu, 9 Jul 2026 05:20:17 UTC (4,444 KB)
[v3] Mon, 9 Nov 2026 10:24:29 UTC (4,360 KB)
[v4] Mon, 16 Nov 2026 10:10:15 UTC (4,360 KB)