[Submitted on 6 Feb 2025 (v1), last revised 26 Oct 2026 (this version, v5)]
Abstract:Multiple works have emerged to push the boundaries of multi-modal large language models (MLLMs) towards pixel-level understanding. The current trend is to train MLLMs with pixel-level grounding supervision in terms of masks on large-scale labelled data and specialized decoders for the segmentation task. However, we show that such MLLMs when evaluated on recent challenging vision-centric benchmarks, exhibit a weak ability in visual question answering (VQA). Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such pixel-level supervision. In this work, we propose two novel challenging benchmarks with paired evaluation for both VQA and grounding. We demonstrate that simple baselines that are not unified achieve performance that matches or surpasses some of the pixel-level MLLMs. Our paired benchmarks and evaluation enable additional analysis on whether the failure stems from VQA and/or grounding abilities. Furthermore, we propose a paired prompt sensitivity analysis on both the language and visual prompts tailored for the grounding task. Our benchmarking pushes pixel-level MLLMs to improve in both accuracy and robustness, without sacrificing language abilities. More importantly, we study the research question of “When does grounding emerge in MLLMs with respect to the output tokens?” We propose a post-hoc interpretability tool that can be plugged into any MLLM to study the aforementioned question. We show that grounding does not necessarily coincide with the exact referring expression in the output, but can coincide with the object parts, its location, appearance, context or state. Code and datasets are publicly available at this https URL.
Submission history
From: Mennatullah Siam M.S. [view email]
[v1] Thu, 6 Feb 2025 16:29:50 UTC (11,172 KB)
[v2] Sun, 23 Feb 2025 11:01:02 UTC (8,439 KB)
[v3] Mon, 2 Jun 2025 07:35:13 UTC (9,363 KB)
[v4] Sat, 24 Jan 2026 14:57:10 UTC (6,572 KB)
[v5] Mon, 26 Oct 2026 07:05:38 UTC (7,680 KB)
