[Submitted on 13 Mar 2026 (v1), last revised 7 Nov 2026 (this version, v3)]
Abstract:Reinforcement learning (RL) has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods. Existing approaches therefore rely on surrogate likelihoods or heuristic approximations, which can introduce bias and obscure the sequential structure of denoising. We formulate diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive an exact, unbiased policy gradient that decomposes over denoising steps and is expressed in terms of intermediate advantages, without requiring explicit evaluation of the sequence likelihood. To obtain a practical and compute-efficient estimator, we (i) select denoising steps for policy updates via an entropy-guided approximation bound, and (ii) estimate intermediate advantages using a one-step denoising reward naturally provided by the diffusion model, avoiding costly multi-step rollouts. Experiments on coding and logical reasoning benchmarks demonstrate state-of-the-art results, with strong competitive performance on mathematical reasoning, outperforming existing RL post-training approaches for DLMs. Code is available at this https URL.
Submission history
From: Fatemeh Doudi [view email]
[v1] Fri, 13 Mar 2026 01:38:44 UTC (2,113 KB)
[v2] Thu, 14 May 2026 15:00:13 UTC (2,112 KB)
[v3] Sat, 7 Nov 2026 16:14:43 UTC (2,169 KB)
