The Post-Training Dilemma: Why We Should Rethink the Sequential SFT-RL Paradigm

the-post-training-dilemma:-why-we-should-rethink-the-sequential-sft-rl-paradigm

The Post-Training Dilemma: Why We Should Rethink the Sequential SFT-RL Paradigm

[Submitted on 12 Jan 2026 (v1), last revised 19 Nov 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:This position paper argues that the sequential supervised fine-tuning followed by reinforcement learning (SFT-then-RL) pipeline—the de facto paradigm for large language model (LLM) post-training—has a fundamental property that cannot be fixed by hyperparameter tuning: SFT and RL optimize over nearly orthogonal parameter subspaces, so RL inevitably degrades the capabilities acquired during the preceding SFT phase. Under the Polyak-Lojasiewicz (PL) condition, which is empirically observed for over-parameterized networks, we show that RL increases the SFT loss by at least $frac{mu}{2}mathrm{dist}(theta_{mathrm{RL}}, Theta^*_{mathrm{SFT}})^2$ for some $mu>0$. Although the KL penalty can constrain distributional shift, it cannot correct the underlying gradient misalignment between the two objectives. Experiments on Qwen3-0.6B across four tasks confirm the theoretical predictions: on MATH-500, Alpaca, HH-RLHF, and SST-2, the SFT loss increases after the RL phase. We identify the underlying mechanism as persistent near-orthogonality between the gradients of the two objectives, paired with opposing landscape dynamics: SFT flattens the loss landscape while RL re-sharpens it. Based on our analysis, we propose three promising research directions: (1) joint optimization of the two objectives with gradient surgery to resolve conflicting updates; (2) landscape-aware scheduling that terminates RL training at the theoretically optimal point; and (3) characterizing the model scale threshold above which SFT becomes unnecessary.

Submission history

From: Xueyan Niu [view email]
[v1] Mon, 12 Jan 2026 10:14:09 UTC (170 KB)
[v2] Wed, 6 May 2026 07:45:21 UTC (176 KB)
[v3] Thu, 19 Nov 2026 01:42:25 UTC (105 KB)