[Submitted on 20 Oct 2025 (v1), last revised 12 Oct 2026 (this version, v2)]
Abstract:Mean-field variational inference (MFVI) is a widely used method for approximating high-dimensional probability distributions by product measures. It has been empirically observed that MFVI optimizers often suffer from emph{mode collapse}. For example, when the target measure $pi$ is a mixture $pi = w P_0 + (1 – w) P_1$, the MFVI optimizer tends to place most of its mass near a single component of the mixture. This work provides the first emph{theoretical} explanation of mode collapse in MFVI. Starting from two-component mixtures, we introduce the notion of emph{$varepsilon$-separateness} to quantify the separation of the mixture components. We derive explicit bounds on the mass that any MFVI optimizer assigns to each component, when $P_0$ and $P_1$ are $varepsilon$-separated for sufficiently small $varepsilon$. We further extend this analysis to mixture target distributions with multiple components. Our results reveal that the occurrence of mode collapse crucially depends on the relative position of the components. To address this issue, we propose emph{rotational variational inference (RoVI)}, which augments MFVI with a rotation matrix. The numerical studies support our theoretical findings and illustrate improved mode coverage with RoVI in Bayesian mixture posteriors, including a real-data example.
Submission history
From: Shunan Sheng [view email]
[v1] Mon, 20 Oct 2025 00:36:13 UTC (1,166 KB)
[v2] Mon, 12 Oct 2026 00:40:12 UTC (1,199 KB)
