Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

efficiently-aligning-draft-models-via-parameter-and-data-efficient-adaptation

Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

[Submitted on 10 Mar 2026 (v1), last revised 13 Nov 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Speculative decoding accelerates LLM inference but suffers from performance degradation when target models are fine-tuned for specific domains. A naive solution is to retrain draft models for every target model, which is costly and inefficient. To address this, we introduce a practical parameter- and data-efficient framework named Efficient Draft Adaptation, abbreviated as EDA, for efficiently adapting draft models. EDA introduces three innovations: (1) a decoupled architecture that utilizes shared and private components to model the shared and target-specific output distributions separately, enabling parameter-efficient adaptation by updating only the lightweight private component; (2) a data regeneration strategy that utilizes the fine-tuned target model to regenerate training data, thereby improving the alignment between training and speculative decoding, leading to higher average acceptance length; (3) a sample selection mechanism that prioritizes high-value data for efficient adaptation. Our experiments show that EDA effectively restores speculative performance on fine-tuned models, achieving superior average acceptance lengths with significantly reduced training costs compared to full retraining. Code is available at this https URL.

Submission history

From: Luxi Lin [view email]
[v1] Tue, 10 Mar 2026 11:35:58 UTC (1,339 KB)
[v2] Fri, 13 Nov 2026 09:41:09 UTC (1,356 KB)