[Submitted on 18 Jan 2026 (v1), last revised 3 Nov 2026 (this version, v5)]
Abstract:Reinforcement learning (RL) has emerged as a paradigm for fine-tuning large-scale generative models, such as diffusion and flow models, to align with complex human preferences and user-specified tasks. A fundamental limitation remains textit{the curse of diversity collapse}, where the objective formulation and optimization landscape inherently collapse the policy to a Dirac delta distribution. To address this challenge, we propose textbf{DRIFT} (textbf{D}ivetextbf{R}sity-textbf{I}ncentivized Reinforcement textbf{F}ine-textbf{T}uning for Versatile Image Generation), an innovative framework that systematically incentivizes output diversity throughout the on-policy fine-tuning process, reconciling strong task alignment with high generation diversity to enhance versatility essential for applications that demand diverse candidate generations. We approach the problem across three representative perspectives: i) textbf{sampling} a reward-concentrated subset that filters out reward outliers to prevent premature collapse; ii) textbf{prompting} with stochastic variations to expand the conditioning space, and iii) textbf{optimization} of the intra-group diversity with a potential-based reward shaping mechanism. Experimental results show that DRIFT exhibits clear Pareto dominance in task alignment and generation diversity, achieving 3.88%$sim$81.42% higher diversity at matched alignment and 14.38%$sim$56.45% higher alignment at matched diversity. Our code is available at textcolor{magenta}{href{this https URL}{this https URL}}.
Submission history
From: Jinmei Liu [view email]
[v1] Sun, 18 Jan 2026 13:25:43 UTC (18,311 KB)
[v2] Wed, 5 Aug 2026 11:59:05 UTC (26,051 KB)
[v3] Tue, 27 Oct 2026 13:20:56 UTC (28,703 KB)
[v4] Wed, 28 Oct 2026 03:32:40 UTC (28,696 KB)
[v5] Tue, 3 Nov 2026 14:44:46 UTC (28,695 KB)
