[Submitted on 3 Oct 2024 (v1), last revised 19 Oct 2026 (this version, v2)]
Abstract:Pre-trained speech models build entangled internal representations, which capture a variety of features (e.g., pitch, loudness, syntax, or semantics of an utterance) in a distributed encoding. This entanglement makes it difficult to track how such representations rely on textual versus acoustic information when used in downstream applications, limiting their interpretability and transparency. In this paper, we explore an approach to post-hoc disentanglement that builds on the Information Bottleneck principle. We take representations from pre-trained speech encoders as input, and separate them into two distinct components in a cascaded fashion: first we learn a vector encoding content (i.e., what can be transcribed as text), and nothing else; second, we learn a vector encoding all complementary acoustic features relevant to a target task. We apply and evaluate our proposed setup to emotion recognition and speaker identification target tasks, quantifying the relative contribution of textual and acoustic features at each model layer. Finally, we use the approach for disentangled feature attribution as it allows us to identify the most salient speech frames from both the textual and acoustic perspectives.
Submission history
From: Hosein Mohebbi [view email]
[v1] Thu, 3 Oct 2024 22:48:04 UTC (536 KB)
[v2] Mon, 19 Oct 2026 17:12:57 UTC (276 KB)
