[Submitted on 28 Jun 2025 (v1), last revised 18 Nov 2026 (this version, v2)]
Abstract:A central challenge for multimodal language models (MLLMs) in real-world tasks is to abstract visual observations into structured spatial states and reason over how those states change under actions over long horizons. Existing multimodal benchmarks often focus on static question answering, where answers can be directly extracted from images, or assume symbolic states are available and bypass the visual-to-spatial abstraction process. We present MARBLE, a diagnostic benchmark for spatially grounded multimodal reasoning and planning across three agent environments: CUBE, MAZE, and BLOX. MARBLE spans 2D and 3D spatial puzzles under geometric constraints, dynamic layouts, and spatial transformations, with difficulty systematically controlled along two axes: observation format and planning horizon. Extensive evaluations show that frontier MLLMs struggle on MARBLE, with the strongest model GPT-5.4 achieving only a 7.3% average success rate across the three environments in the hardest setting. Pairwise ablations on the open-weight models identify visual-to-spatial abstraction as the major bottleneck: when models must infer spatial states from images instead of receiving human-annotated states, success drops from 50.7% to 3.7% for Gemma-4-31B and from 87.9% to 6.5% for Kimi-K2.6. The results indicate that models often fail to construct precise spatial states from visual observations before planning, a task that is straightforward for humans. Overall, MARBLE reveals the gap in reliable visual-to-spatial abstraction for frontier MLLMs and provides a controlled testbed for measuring progress in spatially grounded multimodal reasoning and planning.
Submission history
From: Michael Moor [view email]
[v1] Sat, 28 Jun 2025 19:44:32 UTC (5,879 KB)
[v2] Wed, 18 Nov 2026 17:42:16 UTC (7,646 KB)
