[Submitted on 5 Jun 2026 (v1), last revised 16 Nov 2026 (this version, v3)]
Abstract:A growing failure mode in agent evaluation and training is that models can achieve high evaluation scores (e.g., test pass rates) by exploiting shortcuts instead of solving the intended task, making these scores deceptive. This makes evaluation scores unreliable as measures of true task-solving ability. We propose CapCode, a framework for constructing coding datasets with randomized tests whose best achievable non-cheating score is deliberately capped below 100%. This capped-score design gives evaluation scores a clearer interpretation: scores substantially above the cap are implausible and therefore provide evidence of cheating. To prevent cheating, we propose CapReward, a reward design based on the CapCode principle to discourage optimization beyond the cap. Experiments across multiple datasets show that CapCode detects cheating while preserving performance ranking of models, and CapReward reduces cheating behavior, yielding models that better follow the intended task specification.
Submission history
From: Thanawat Lodkaew [view email]
[v1] Fri, 5 Jun 2026 15:20:37 UTC (188 KB)
[v2] Mon, 8 Jun 2026 01:53:05 UTC (188 KB)
[v3] Mon, 16 Nov 2026 08:37:03 UTC (192 KB)
