[Submitted on 20 Jul 2025 (v1), last revised 14 Oct 2026 (this version, v5)]
Abstract:Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models’ reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, textit{the shrinkage of observed support outweighs its expansion}: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in texttt{pass@1}. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an “invisible leash” in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.
Submission history
From: Fang Wu [view email]
[v1] Sun, 20 Jul 2025 07:04:08 UTC (273 KB)
[v2] Fri, 26 Sep 2025 04:52:44 UTC (415 KB)
[v3] Wed, 7 Jan 2026 01:59:36 UTC (413 KB)
[v4] Wed, 4 Feb 2026 00:51:57 UTC (420 KB)
[v5] Wed, 14 Oct 2026 06:13:12 UTC (485 KB)
