[Submitted on 7 May 2026 (v1), last revised 20 Oct 2026 (this version, v3)]
Abstract:Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from emph{boundary mismatch} at the flow source or from emph{high-variance} bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using textbf{source-consistent Bellman-coupled paths}: the current path starts from the required base prior at $t{=}0$, reaches the Bellman target at $t{=}1$, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-$t$ marginals to satisfy a distributional Bellman fixed point for all $t$). PCBF couples current and successor return flows through shared base noise and uses a $lambda$-parameterized control-variate target: $lambda{=}0$ recovers an unbiased sample Bellman target, while $lambda{>}0$ trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.
Submission history
From: Hao Yan [view email]
[v1] Thu, 7 May 2026 19:05:01 UTC (8,030 KB)
[v2] Wed, 3 Jun 2026 21:28:18 UTC (8,056 KB)
[v3] Tue, 20 Oct 2026 18:03:55 UTC (8,062 KB)
