[Submitted on 12 May 2025 (v1), last revised 24 Oct 2026 (this version, v3)]
Abstract:Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Recently, vision-language models (VLMs), pre-trained on massive image-text pairs, have demonstrated remarkable zero-/few-shot performance that often surpasses SSL approaches due to their exceptional generalization capabilities. This gap motivates us to question: how can we effectively harness the powerful generalization capabilities of VLMs into task-specific models? Knowledge distillation (KD) offers a natural framework for transferring VLM capabilities, but we identify that it suffers from gradient conflicts between supervised and distillation losses. To address this challenge, we propose Dual-Head Optimization (DHO), which introduces dual prediction heads for each distinct signal. We observe that DHO mitigates gradient conflicts, enabling improved feature learning compared to single-head KD baselines, with practical benefits of minimal computational overhead and test-time hyperparameter tuning without retraining. Extensive experiments across 15 datasets show that DHO consistently outperforms KD baselines, often outperforming teacher models with smaller student models. DHO also achieves new state-of-the-art performance on both in-distribution ImageNet semi-supervised learning and out-of-distribution generalization across ImageNet variants. Code and model checkpoints are publicly available at this https URL.
Submission history
From: Seongjae Kang [view email]
[v1] Mon, 12 May 2025 15:39:51 UTC (10,480 KB)
[v2] Tue, 30 Sep 2025 14:13:57 UTC (10,276 KB)
[v3] Sat, 24 Oct 2026 06:51:32 UTC (12,164 KB)
