Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

are-we-grading-properly?-understanding-failure-modes-in-medical-benchmarks

Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

[Submitted on 10 Sep 2026 (v1), last revised 18 Nov 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to three clinical benchmarks (HealthBench Professional, LiveMedBench, and MentalHealthBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid, although these rates vary severalfold across five detection judges. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form “at least one of / all of the following” as equally weighted children and regrading identical responses shifts scores by up to 15.9 percentage points on affected conversations, with disjunctive bundles inflating scores and conjunctive bundles deflating them. Across six answering models and two common graders, these shifts leave the model ranking unchanged, and their net direction depends on how many bundled criteria are penalties. We also find that RIFT can under-detect bundling on clinical rubrics, by an amount that depends on the detection judge: on the same LiveMedBench criteria, judges flag 3.8-15.9% as non-atomic against an estimated 7-19% that are genuinely bundled.

Submission history

From: Prithvi Dixit [view email]
[v1] Thu, 10 Sep 2026 01:18:39 UTC (701 KB)
[v2] Wed, 18 Nov 2026 20:33:26 UTC (1,113 KB)