[Submitted on 28 May 2026 (v1), last revised 29 Oct 2026 (this version, v2)]
Abstract:When annotators disagree on a label, the disagreement itself carries signal – and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation – whether the model identifies which items elicit disagreement – requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds); the specific thresholds are estimates for this setting, and it is the phenomenon, not the constants, that we expect to generalize. This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities, entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones. The soft label advantage replicates across two architectures (DeBERTa, RoBERTa) and a non-NLI-pretrained baseline; differential saturation additionally appears in an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.
Submission history
From: Guneet Kohli [view email]
[v1] Thu, 28 May 2026 11:46:51 UTC (84 KB)
[v2] Thu, 29 Oct 2026 05:54:23 UTC (98 KB)
