[Submitted on 28 May 2025 (v1), last revised 30 Oct 2026 (this version, v4)]
Abstract:Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.
Submission history
From: Mattie Fellows [view email]
[v1] Wed, 28 May 2025 15:07:24 UTC (5,896 KB)
[v2] Thu, 29 May 2025 20:38:03 UTC (5,896 KB)
[v3] Wed, 15 Jul 2026 19:14:08 UTC (6,122 KB)
[v4] Fri, 30 Oct 2026 17:26:53 UTC (6,122 KB)
