On-Policy Distillation and Self-Distillation: A Complete Guide to Training Reasoning Models
This article explains why reasoning models benefit from On-Policy Distillation (OPD), how it differs from offline knowledge distillation, why reverse KL is commonly used, and how On-Policy Self-Distillation extends the approach. It also covers end-to-end training workflows, reasoning-mode alignment, common failure modes, optimization strategies, and evaluation criteria for measuring capability, process quality, distribution alignment, and training cost.
1. Why Reasoning Models Need On-Policy Distillation
Traditional large language model distillation usually relies on offline data. A teacher model first generates question-answer pairs, reasoning traces, or preference data. A student model is then trained with supervised fine-tuning. The student learns from complete trajectories that the teacher has already generated rather than receiving continuous guidance while making its own decisions.
This approach is simple and stable, and it works well for style transfer, knowledge transfer, and basic capability replication. However, it creates a critical problem for models that perform multi-step reasoning: the historical states in the training data are often not states that the student would reach on its own.
For example, a teacher may generate a sequence of correct intermediate steps for a complex problem. A weaker student might choose a different path at the third step, or make an incorrect judgment that the teacher never produced. The student then continues generating from this incorrect state. During offline training, however, it has only seen the correct prefixes from the teacher’s trajectory. It has not learned what to do after making a mistake or how to recover from an off-track state.
When this happens during real inference, errors can accumulate and eventually lead to broken reasoning, an incorrect conclusion, or endless repetition.
This mismatch between the training distribution and the inference distribution is commonly known as exposure bias. It can be summarized as follows:
- During training, the model mainly learns from states generated by the teacher.
- During inference, the model actually visits states generated by itself.
- The two state distributions differ, and the gap is often larger for weaker models.
- Once the model enters a state that is insufficiently covered by the training data, prediction quality can decline rapidly.
The core improvement introduced by On-Policy Distillation is to let the student first generate trajectories using its own policy, after which the teacher provides guidance for those actual student states.
2. What Is On-Policy Distillation?
Let the student model be (S), the teacher model be (T), the input problem be (x), and the student-generated history before position (t) be (y_{<t}). At position (t), the teacher and student produce the following next-token distributions:
[
pT(y_t|x,y{<t}),\qquad pS(y_t|x,y{<t})
]
In offline distillation, the historical sequence (y_{<t}) usually comes from the teacher or from a preconstructed dataset. In OPD, the history is primarily sampled from the student:
[
y_{<t}\sim p_S
]
The teacher then evaluates the same student state and provides a guidance distribution. The student updates its parameters based on that feedback.
OPD is therefore not simply a process of copying teacher answers. It follows a workflow such as:
- Provide a problem or task.
- Use the student to generate one or more trajectories.
- Feed the student-generated prefixes to the teacher.
- Obtain the teacher’s probability distribution, candidate actions, or preference signal for the current state.
- Measure the difference between the teacher and student distributions.
- Update the student so that it better matches the teacher on states the student is likely to visit.
The key question is not what the teacher generated previously, but how the teacher guides a state that the student has already reached.
3. Why Reverse KL Is Commonly Used
Common distribution-distance measures in distillation include forward KL and reverse KL. Let the teacher distribution be (p) and the student distribution be (q). Reverse KL is defined as:
[
D_{\mathrm{KL}}(q|p)
=
\sum_y q(y)\log\frac{q(y)}{p(y)}
]
It can also be written as:
[
D{\mathrm{KL}}(q|p)
=
\mathbb{E}{y\sim q}
[
\log q(y)-\log p(y)
]
]
Because samples are drawn from the student distribution, reverse KL is naturally suited to on-policy training. It provides two important benefits.
1. It emphasizes high-probability behavior
Reverse KL has a strong mode-seeking tendency. When the teacher supports multiple valid answers or reasoning paths, the student does not necessarily need to fit every path equally. Instead, it can concentrate on a more reliable region of the teacher’s distribution.
This is particularly useful for reasoning models. Complex problems often have multiple correct solutions, but the quality of their intermediate steps may vary. If the student is forced to average over all candidate distributions, it may develop a vague, conservative, and incoherent strategy. Reverse KL instead encourages the student to select high-probability behavior from the teacher and form a relatively stable reasoning pattern.
2. It matches the student’s own states
Because the expectation can be estimated under the student distribution, reverse KL directly focuses on content the student is likely to generate. If the student tends to produce an incorrect token at a particular position, training will frequently observe that behavior and use the teacher’s probabilities to correct it.
In other words, reverse KL does not merely tell the student what the teacher did in the original trajectory. It tells the student which choices are more reasonable in the state it is currently visiting. This allows the training signal to cover common student errors rather than being limited to ideal teacher trajectories.
4. OPD vs. Traditional Offline Distillation
The main differences between the two approaches concern trajectory sources, optimization targets, and error handling.
1. Different trajectory sources
Offline distillation generally uses trajectories generated by the teacher, human annotators, or an existing dataset. The student learns from fixed examples.
In OPD, trajectories are generated online by the student. Even incomplete, redundant, or incorrect reasoning processes can become training states.
2. Different optimization targets
Offline supervised fine-tuning typically optimizes the likelihood of target tokens:
[
\mathcal{L}{\mathrm{SFT}}
=
-\sum_t \log p_S(y_t^*|x,y{<t}^*)
]
Here, (y_t^*) is the target token in the dataset.
OPD focuses more on the difference between teacher and student distributions evaluated on student-generated states:
[
\mathcal{L}{\mathrm{OPD}}
=
\sum_t
D\left(
p_S(\cdot|x,y{<t}),
pT(\cdot|x,y{<t})
\right)
]
In this expression, (y_{<t}) comes from student generation rather than from a fixed reference answer.
3. Different approaches to errors
Offline distillation generally shows the student only correct trajectories. As a result, the student has limited experience with its own incorrect states and may not learn how to recover from them.
OPD explicitly includes off-track states in training, allowing the teacher to provide local correction after an error occurs.
This does not mean that OPD can automatically resolve every reasoning error. If the student enters a state that the teacher also cannot evaluate reliably, or if the teacher lacks sufficient ability for the task, distillation quality will still be limited. A more precise description is that OPD enables the teacher to guide paths the student is actually likely to take; it does not guarantee that every path can be corrected.
5. What Is On-Policy Self-Distillation?
When the teacher and student are not two entirely independent models, but rather different versions of the same model at different training stages, parameter states, or inference strategies, the process can be called On-Policy Self-Distillation.
Common forms include:
- Using the pre-training or pre-update model as a relatively stable teacher.
- Using an exponential moving average or other parameter-averaged model as the teacher.
- Generating guidance distributions with a larger reasoning budget or a more deliberate sampling strategy.
- Using high-quality trajectories generated by the model itself after filtering or verification.
- Having the current model learn a more reliable policy based on its own historical behavior.
The key idea is not that the model simply copies itself. Instead, the training process constructs a teacher perspective that is relatively stronger, more stable, or more reliable.
For example, the same model may produce a higher-quality answer when given a larger reasoning budget. That result can serve as a learning target for a shorter-budget policy. Alternatively, the model can generate multiple candidate trajectories, use rules, a verifier, or consistency checks to select more reliable trajectories, and then use them to train itself.
Self-distillation therefore usually contains three stages:
- Generation: The model produces candidate trajectories under its current policy.
- Evaluation: Rewards, rules, verifiers, or consistency mechanisms identify higher-quality behavior.
- Feedback: The selected results are converted into distribution-level guidance, supervised signals, or preference signals.
Compared with an external teacher, self-distillation offers a closed-loop workflow with more controllable costs and makes continuous task-specific iteration easier. However, it has a clear risk: if the filtering mechanism is unreliable, the model may amplify its own mistakes and create a cycle of incorrect self-reinforcement.
6. Designing a Typical Training Workflow
A complete OPD training workflow can be divided into several stages.
Stage 1: Establish Basic Capabilities
Before applying OPD directly, the student usually needs basic language generation, instruction-following, and reasoning-format capabilities. High-quality offline data can first be used for supervised training so that the model understands the task definition, output format, and basic reasoning style.
The goal at this stage is not to make the student fully replicate the teacher. It is to reduce the capability gap enough to prevent the student from generating mostly unusable trajectories.
Stage 2: Sample Student Trajectories Online
For each input problem, use the student to generate one or more trajectories. The sampling strategy should match the task:
- A temperature that is too low reduces exploration.
- A temperature that is too high introduces excessive noise.
- Keeping only the final answer may discard useful intermediate-state information.
- Keeping only the longest trajectories may encourage ineffective repetition.
In practice, training systems may retain complete sequences, termination reasons, answer correctness, and length metadata for later filtering and weighting.
Stage 3: Evaluate Student States with the Teacher
Feed student-generated prefixes into the teacher and obtain guidance distributions at each step. When compute is limited, the teacher can evaluate only selected positions, such as:
- Reasoning branch points.
- Positions where the final answer is formed.
- Positions with high student uncertainty.
- Positions with large teacher-student disagreement.
Selective distillation can reduce teacher inference costs, but it requires reliable trigger rules.
Stage 4: Compute the Distillation Loss
In addition to reverse KL, training can combine several losses:
[
\mathcal{L}
=
\lambda1\mathcal{L}{\mathrm{KL}}
+
\lambda2\mathcal{L}{\mathrm{SFT}}
+
\lambda3\mathcal{L}{\mathrm{answer}}
+
\lambda4\mathcal{L}{\mathrm{preference}}
]
Here:
- (\mathcal{L}_{\mathrm{KL}}) constrains the student distribution to match the teacher distribution.
- (\mathcal{L}_{\mathrm{SFT}}) preserves basic language and formatting capabilities.
- (\mathcal{L}_{\mathrm{answer}}) constrains the correctness of the final answer.
- (\mathcal{L}_{\mathrm{preference}}) encourages the selection of better trajectories.
The weights should be adjusted according to the task. If training optimizes only distribution matching, the student may overfit local teacher behavior while neglecting the final objective. If it optimizes only final-answer correctness, reasoning stability may suffer.
Stage 5: Iterate and Evaluate
After one training round, generate new student trajectories and measure how the distribution has changed. OPD is not a one-time data-generation process but a dynamic loop:
[
\text{Student sampling}
\rightarrow
\text{Teacher guidance}
\rightarrow
\text{Parameter update}
\rightarrow
\text{Resampling}
]
As the student policy changes, the state distribution used for training also changes. Old data therefore cannot fully replace continuous online sampling.
7. The Importance of Reasoning-Mode Consistency
OPD performance depends not only on whether the teacher is more capable, but also on whether the teacher and student use sufficiently compatible reasoning modes.
A reasoning mode may include:
- Whether intermediate reasoning is explicitly expanded.
- Whether the model plans before executing.
- Whether self-checking and backtracking are allowed.
- Whether the model prefers short or long reasoning chains.
- Whether it verifies the result before producing the final answer.
- Whether special tokens, stopping conditions, and output structures are aligned.
Suppose the teacher uses long, step-by-step reasoning with verification, while the student is trained to answer immediately. Aligning token-level distributions may then create conflicts. The student must learn not only the content but also a new output rhythm, structure, and control strategy, which significantly increases training difficulty.
For this reason, reasoning-mode alignment is often performed before more advanced OPD. Possible strategies include:
- Use offline data to teach the student a basic reasoning format.
- Keep the teacher and student output protocols consistent.
- Train reasoning-mode switching as a separate capability.
- Sample and evaluate different reasoning modes separately.
- Avoid combining too many conflicting behavioral objectives in the same training stage.
8. Common Training Problems and Improvements
1. Student trajectories are too poor
If the student is too weak, most generated trajectories may fail to complete the task, and the teacher’s guidance may not produce an effective training signal.
A gradual-improvement strategy can help: first apply basic supervised training and then increase the online training ratio. Another option is to distill only the recoverable portions of student trajectories rather than forcing the model to imitate an entire failed trajectory.
2. Teacher inference is too expensive
The teacher may need a forward pass for every position in every student-generated sequence, which becomes expensive for long contexts.
Possible cost-reduction methods include distilling only candidate positions, refreshing the teacher less frequently, caching teacher outputs, shortening contexts, or evaluating only positions with significant disagreement. At scale, the training system must also plan teacher inference resources, sample parallelism, and storage bandwidth.
3. The model overfits surface formatting
If the objective focuses too heavily on token-level distributions, the student may learn fixed wording and templates without genuinely improving problem-solving ability.
Evaluation should therefore include final accuracy, reasoning length, repetition rate, error-recovery ability, and cross-task generalization. A declining distillation loss alone is not sufficient evidence of improvement.
4. Incorrect self-reinforcement
In self-distillation, the model may treat its own incorrect answer as a teacher signal, causing errors to accumulate.
Potential safeguards include externally verifiable rules, answer validators, multi-trajectory consistency checks, human spot checks, and periodic rollback to a more stable model version. For high-risk tasks, the system should maintain explicit quality thresholds and avoid adding every online sample to the training set without filtering.
9. How to Evaluate Whether OPD Actually Works
OPD should be evaluated at four levels.
Capability
Measure task accuracy, success rates on complex problems, cross-domain generalization, and few-shot adaptation.
Process
Analyze whether reasoning chains are coherent, whether the model enters repetitive loops, whether it can detect and correct earlier mistakes, and whether its behavior remains stable across different samples.
Distribution
Compare student behavior before and after training, including high-probability candidates, entropy, branch selection, and decision consistency at critical positions.
Cost
Track teacher invocation counts, average sequence length, training throughput, memory usage, and cost per sample. Capability gains from OPD must be balanced against computational overhead before the method can be used at scale.
10. Conclusion
The core value of On-Policy Distillation is that it changes distillation from learning ideal trajectories already produced by the teacher to receiving guidance on states the student is actually likely to visit. Student-side online sampling helps reduce exposure bias, distribution-level supervision improves behavior transfer, and the mode-seeking property of reverse KL makes it easier for the student to develop a stable and focused reasoning strategy.
On-Policy Self-Distillation goes a step further by incorporating teacher-like capabilities into a closed training loop. Different model versions, reasoning budgets, or quality-filtering strategies can provide relatively reliable guidance. This reduces dependence on an external teacher, but it also makes preventing incorrect self-reinforcement and quality drift especially important.
In engineering practice, OPD is not a simple replacement for supervised fine-tuning. It is a training system that combines foundational capabilities, online sampling, teacher evaluation, loss design, quality filtering, and continuous assessment. OPD is most effective when the teacher and student have reasonably aligned reasoning modes, the student has sufficient initial capability, and online trajectories are filtered carefully. Under those conditions, it can improve reasoning transfer, error recovery, and the integration of capabilities within a model.
