TIVS: Mapping the Sensitivity Landscape of Visual Token Transformations in MLLMs

tivs:-mapping-the-sensitivity-landscape-of-visual-token-transformations-in-mllms

TIVS: Mapping the Sensitivity Landscape of Visual Token Transformations in MLLMs

[Submitted on 15 Nov 2026 (v1), last revised 17 Nov 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Recent efforts to improve the efficiency of multimodal large language models (MLLMs) have explored visual token pruning, quantization, and feature compression. However, the properties that determine whether transformed visual tokens remain usable by a frozen decoder are not well characterized. We introduce textbf{TIVS (Token Identity Versus Statistics)}, a training-free intervention framework that separately studies sequence permutation, token retention, numerical precision, and statistical replacement under controlled conditions. On 192 POPE questions, three token permutations change TinyLLaVA accuracy by only $-0.17$ percentage points on average, while changing 4.69% of individual predictions. Diagonal and rank-64 Gaussian replacements decrease accuracy by 25.0 and 28.1 points on the same subset. Further experiments with exact moment constraints, quantization error, token retention, and additional visual tasks reveal how strongly the effects depend on the original feature values and the evaluation setting. These findings provide an empirical basis for assessing visual token transformations in frozen MLLMs and designing more reliable multimodal interfaces.

Submission history

From: Yikang Qiao [view email]
[v1] Sun, 15 Nov 2026 03:13:43 UTC (1,018 KB)
[v2] Tue, 17 Nov 2026 03:16:41 UTC (1,071 KB)