[Submitted on 6 Mar 2026 (v1), last revised 21 Oct 2026 (this version, v2)]
Abstract:On-policy agents discard trajectories after each update, including rare high-return episodes. We propose Optimistic Policy Regularization (OPR), which turns those episodes into a closed-form reward bonus: we estimate an elite action prior from top-quartile episodes and add a reviewblue{bounded transform of the log-ratio} of the prior to the policy’s action distribution. OPR needs no auxiliary network, value gate, off-policy correction, or adversarial loop. It is a one-line modification of Proximal Policy Optimization (PPO)’s reward; yet, on 49 Atari games at 10M steps, $5times$ fewer agent decisions than the 200M-frame baselines, it reaches a 148.4% median human-normalized score (HNS; 196.9% interquartile mean): higher scores than published PPO on $36/49$ games ($p{=}0.0014$) and a higher aggregate median than self-imitation learning, which falls well below OPR when frames and backbone are matched. We pair the method with a theory of when it works. The underlying log-ratio has an exact forward-KL interpretation; its pre-advantage placement propagates the signal through Generalized Advantage Estimation; and, under an idealized local update, its first-order effect depends on a prior-advantage alignment whose sign indicates help or harm. Since alignment varies across tasks and runs, we also study an online gating extension. On a ten-game set, it outperforms the backbone by $+17.4$ median HNS on average across seeds. Controlled experiments test temporal propagation and the tradeoffs of state conditioning and gating, supporting OPR as a simple regularizer with an analysis of where to shape.
Submission history
From: Mai Pham [view email]
[v1] Fri, 6 Mar 2026 19:00:07 UTC (2,299 KB)
[v2] Wed, 21 Oct 2026 20:07:35 UTC (1,102 KB)
