[Submitted on 23 May 2026 (v1), last revised 20 Oct 2026 (this version, v2)]
Abstract:In medical imaging, small sets of expert labels are often combined with zero-shot pseudo-labels from a vision-language model (VLM). When the expert-labeled set is very small, the extra pseudo-labels reduce variance and improve accuracy. Once a classifier trained on expert labels alone reaches the accuracy of the weak labeler, however, adding pseudo-labels lowers test performance. We measure where this crossover occurs on PatchCamelyon histopathology, ISIC 2019 dermoscopy, and NIH ChestX-ray14 radiography using pseudo-labels from BiomedCLIP and PubMedCLIP across six student architectures ($8mathrm{M}$ to $89mathrm{M}$ parameters). Weak labels stop helping after $20$–$50$ expert labels on ISIC 2019, about $100$ on PatchCamelyon, and $250$–$500$ on NIH ChestX-ray14. Above those budgets, adding $10{,}000$ pseudo-labels reduces test macro-AUC by up to $0.14$ points with BiomedCLIP ($0.23$ with PubMedCLIP) on ISIC 2019, $0.10$ on PatchCamelyon, and $0.08$ on NIH ChestX-ray14. Across the six student architectures, the weak-label ceiling stays within a $0.024$ AUC range ($0.841$–$0.865$), indicating that the VLM labeler, rather than student size, limits performance. Comparing VLM pseudo-labels with random label flips at the same error rate shows that systematic VLM mistakes yield $0.09$–$0.20$ lower AUC than random noise on PatchCamelyon and ISIC 2019, but $0.03$–$0.04$ higher AUC on 15-class NIH ChestX-ray14, where per-finding crossovers also span more than a $20times$ range. A simple bias-variance rule estimated from the weak labeler’s validation accuracy and small expert-only runs predicts whether pseudo-labels will help or hurt in all $30$ statistically significant settings ($39/42$ overall).
Submission history
From: Bruce Xu [view email]
[v1] Sat, 23 May 2026 23:15:07 UTC (103 KB)
[v2] Tue, 20 Oct 2026 22:48:22 UTC (134 KB)
