[Submitted on 2 Apr 2026 (v1), last revised 3 Nov 2026 (this version, v3)]
Authors:Jiahao Meng, Yue Tan, Qi Xu, Haochen Wang, Zhongwei Ren, Weisong Liu, Yuhao Wang, Ray Zhang, Jason Li, Kenny Duan, Yunhai Tong, Ming-Hsuan Yang
Abstract:Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question-answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of cross-verification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data are publicly released.
Submission history
From: Jiahao Meng [view email]
[v1] Thu, 2 Apr 2026 03:29:43 UTC (31,664 KB)
[v2] Wed, 7 Oct 2026 17:06:20 UTC (32,922 KB)
[v3] Tue, 3 Nov 2026 08:21:04 UTC (32,767 KB)
