[Submitted on 9 Apr 2026 (v1), last revised 14 Nov 2026 (this version, v4)]
Authors:Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang
Abstract:Recent advances in Language Model (LM) agents have increasingly focused on repository-level software engineering (SWE) tasks, leading to various agentic workflows, training methods and failure analysis. Interestingly, rather than directly generating or evaluating edit patches, these approaches first examine and improve intermediate information signals collected during agent exploration, including Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. Although different studies examine different subsets of these signals, no work has systematically compared their relative effectiveness in improving final task success or measured the remaining space for improvement in each signal. To address this gap, this work quantifies the contributions of both oracle and agent-extracted signals to final task success and measures the gap between current agent-found signals and their oracle counterparts. We hope our findings could suggest future research emphasis in SWE agents. We release our source code and oracle-signal dataset as a reusable methodology to benefit future signal analysis of SWE agents.
Submission history
From: Kenan Li [view email]
[v1] Thu, 9 Apr 2026 04:37:24 UTC (639 KB)
[v2] Thu, 28 May 2026 09:34:24 UTC (620 KB)
[v3] Thu, 15 Oct 2026 14:43:29 UTC (527 KB)
[v4] Sat, 14 Nov 2026 12:13:53 UTC (528 KB)
