EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

easyopd:-an-easy-to-use-on-policy-distillation-framework-for-large-language-models

EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

[Submitted on 13 Jul 2026 (v1), last revised 18 Nov 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Conventional language-model distillation often relies on fixed teacher-generated data, which may not cover the states encountered by an evolving student policy. On-policy distillation (OPD) instead collects teacher or evaluator supervision on student-generated rollouts. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend. We present textsc{EasyOPD}, an on-policy distillation framework built on verl, a distributed reinforcement-learning framework for large language models. textsc{EasyOPD} separates user-side configuration, method-specific supervision logic, and verl-based execution. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation. We instantiate representative methods for three OPD settings — cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD. Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectives and task-dependent performance profiles. We release textsc{EasyOPD} with runnable YAML configurations, documentation, and an installable demonstration package and video.

Submission history

From: Jie Sun [view email]
[v1] Mon, 13 Jul 2026 02:27:50 UTC (243 KB)
[v2] Wed, 18 Nov 2026 09:13:52 UTC (243 KB)