[Submitted on 1 Mar 2026 (v1), last revised 7 Nov 2026 (this version, v3)]
Abstract:While reinforcement learning (RL) has been central to the recent success of large language models (LLMs), RL optimization is notoriously unstable, especially when compared to supervised fine-tuning (SFT). In this work, we investigate the stability gap between SFT and RL from a gradient-based perspective, and show that the convexity of the SFT loss with respect to model logits plays a key role in enabling stable training. Our theoretical analysis demonstrates that this property induces favorable gradient directionality during optimization. In contrast, Proximal Policy Optimization (PPO), a widely adopted policy gradient algorithm utilizing a clipped surrogate objective, lacks this stabilizing property. Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework that aligns the learned policy with an optimal target derived from the original RL objective, thereby emulating the stabilizing effects of logits-level convexity. Extensive experiments across multiple model families show that our LCO framework consistently improves training stability and outperforms conventional RL methods on a broad range of benchmarks.
Submission history
From: Hongzhan Chen [view email]
[v1] Sun, 1 Mar 2026 07:40:12 UTC (751 KB)
[v2] Mon, 1 Jun 2026 02:47:36 UTC (751 KB)
[v3] Sat, 7 Nov 2026 02:57:56 UTC (1,543 KB)
