[Submitted on 22 Sep 2026 (v1), last revised 10 Oct 2026 (this version, v4)]
Abstract:Fairness audits in production ML typically occur once, at deployment, on a single domain. Both fail in practice: fairness can shift after retraining or a changing user base, and interventions validated on one dataset are rarely tested across the heterogeneous domains an organization deploys. We present FAPE (Fairness Auditing for Production Environments), a four-stage framework evaluating a single post-processing intervention, Fairlearn’s ThresholdOptimizer, across eight domain evaluations: criminal justice, income prediction, legal admissions, credit lending, agricultural lending, a multi-domain benchmark corpus, healthcare, and education. Each is scored on demographic parity and equalized odds difference, plus disparate impact ratio and accuracy cost where computable. Intervention effectiveness tracks baseline disparity magnitude: across model-domain pairs the constraint improved disparity in 9 of 14 high-disparity cases and worsened it in 3 of 4 near-fair ones. Each of the five high-disparity exceptions reverses under one of two measurement checks, a minimum group size or thresholds fit on held-out data. A CUSUM monitor started at deployment, tested on a simulated shift, separates constrained models that never met a 0.1 parity convention from those that met it and later regressed. A single deployment-time audit is therefore an unreliable guide, which argues for baseline-disparity screening and continuous monitoring
Submission history
From: Nithin Raghava Ramachandra Narla [view email]
[v1] Tue, 22 Sep 2026 18:46:00 UTC (1,657 KB)
[v2] Tue, 29 Sep 2026 07:09:28 UTC (1,657 KB)
[v3] Wed, 7 Oct 2026 21:33:20 UTC (1,647 KB)
[v4] Sat, 10 Oct 2026 04:22:05 UTC (1,657 KB)
