AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control

agentrobocritic:-understanding-the-capability-of-frontier-agents-on-direct-robot-control

AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control

[Submitted on 19 Nov 2026]

View PDF HTML (experimental)

Abstract:Frontier multimodal language agents such as GPT-6 Astra have demonstrated strong capabilities in controlling robots directly via low-level numerical actions without high-level primitives. However, existing benchmarks report only episode-level success rates, which show neither how these agents succeed nor where they fail. In this paper, we present a systematic action-level analysis of frontier agents on direct robot control across 12 manipulation benchmarks. We introduce AgentRoboCritic, an intent-grounded critique framework in which the policy agent states a local intent alongside each action, and a critic evaluates the action’s outcome and failure mode against this intent. Our action-level analysis demonstrates the promising capabilities of frontier agents on direct robot control, including closed-loop recovery from local execution failures, reliable semantic grounding across diverse scenes, and strong zero-shot manipulation without task-specific adaptation. It also identifies their current limitations, including task-relevant physical execution bottlenecks such as grasping and alignment, and uneven recovery across failure modes. Finally, we validate our critic on a 400-action human-annotated benchmark and release a dataset of 12,844 annotated actions alongside a fine-tuned open-source critic, Muse-Glimmer-30B, whose failure-mode accuracy rises from 32.58% to 74.24%, enabling low-cost action-level analysis.

Submission history

From: Yiyang Du [view email]
[v1] Thu, 19 Nov 2026 18:57:25 UTC (17,816 KB)