[Submitted on 13 Oct 2026]
Abstract:Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce textsc{HERO} (textbf{H}arnessing textbf{E}vidence for textbf{R}efinement and textbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, method{} achieves the best offline performance, improving pass@1 by (7.6) points over the initial prompt, while test-time refinement adds another (2.0) points and improves execution efficiency.
Submission history
From: MingYu Lu [view email]
[v1] Tue, 13 Oct 2026 20:49:08 UTC (636 KB)
