Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotients

persistent-transient-policy-evaluation-for-markov-chains-via-minimal-peripheral-quotients

Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotients

[Submitted on 31 Jan 2026 (v1), last revised 7 Nov 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:We study fixed-policy evaluation for finite Markov chains that may be reducible and periodic. Classical evaluation methods with gain and bias decomposition are not always diagnostic: the gain records only invariant Cesàro averages, while persistent phase-dependent behavior is absorbed into the bias together with genuinely transient effects. We identify the real peripheral invariant subspace $mathcal{K}(P)$ of the transition matrix $P$ as the source of this ambiguity. Quotienting by $mathcal{K}(P)$ is the minimal exact quotient that removes all non-decaying modes and makes the remaining dynamics strictly stable. After choosing a gauge projection $Pi$ with kernel $mathcal{K}(P)$, the reward admits a unique decomposition $r = g_Pi^star + (I-P)v_Pi^star$, where $g_Pi^star$ is a persistent regime profile and $v_Pi^star$ is a gauge-fixed transient component. An exact comparison with classical normalized gain and bias shows that the new pair reallocates the same information into a persistent profile and a gauge-fixed representative of the transient quotient. This decomposition reconstructs finite-horizon returns, recovers statewise average reward, admits a transient-cost interpretation, and yields a stable estimator under a generative model.

Submission history

From: Yang Xu [view email]
[v1] Sat, 31 Jan 2026 02:57:01 UTC (88 KB)
[v2] Thu, 7 May 2026 18:44:50 UTC (84 KB)
[v3] Sat, 7 Nov 2026 22:13:56 UTC (135 KB)