[Submitted on 2 Oct 2025 (v1), last revised 23 Oct 2026 (this version, v5)]
Abstract:Evaluation of multimodal reasoning models is typically reduced to a single accuracy score, implicitly treating reasoning as a unitary capability. We introduce MathLens, a benchmark of textbook-style geometry problems that exposes this assumption by operationally decomposing performance into perception, reasoning, and multimodal-specific components. Each problem is derived from a symbolic specification and accompanied by visual diagrams, text-only variants, multimodal questions, and targeted perceptual probes, enabling controlled measurement of each component. Using this decomposition, we show that common training strategies are associated with systematically different capability profiles that are invisible under aggregate accuracy. Across the models we study, reinforcement learning primarily improves perceptual grounding and robustness to diagram variation, while our analysis suggests that textual SFT yields gains through reflective reasoning. In contrast, as perception and reasoning improve, a growing fraction of remaining errors fall outside these components and are categorized as multimodal-specific. These results suggest that apparent progress in multimodal reasoning reflects shifting balances among subskills rather than uniform advancement, motivating evaluation beyond scalar accuracy.
Submission history
From: Jiwan Chung [view email]
[v1] Thu, 2 Oct 2025 06:58:29 UTC (10,710 KB)
[v2] Tue, 7 Oct 2025 15:17:04 UTC (10,710 KB)
[v3] Tue, 3 Feb 2026 08:57:11 UTC (10,984 KB)
[v4] Thu, 7 May 2026 09:13:59 UTC (10,973 KB)
[v5] Fri, 23 Oct 2026 02:22:08 UTC (10,998 KB)
