KL Divergence Direction in On-Policy Self-Distillation: Forward vs. Reverse KL in Theory and Practice
KL divergence direction is often overlooked in LLM alignment and logits distillation, yet it directly determines whether a student model covers all high-probability modes of the teacher or collapses onto a few. This article explains the differences between Forward KL and Reverse KL, why Reverse KL naturally fits on-policy self-distillation where training samples come from the student itself, and how token-level versus sequence-level estimation changes the trade-offs. It includes a practical algorithm walkthrough with pseudocode, selection guidelines, and diagnostic metrics to help you stabilize training while avoiding mode collapse.
KL Divergence Direction in On-Policy Self-Distillation: Forward vs. Reverse KL in Theory and Practice
In LLM alignment, model distillation, and logits distillation during SFT, the directionality of KL divergence is frequently ignored, yet it directly determines student model behavior: should the student cover all high-probability modes of the teacher distribution, or focus only on the few regions the teacher considers most certain? This matters even more in On-Policy Self-Distillation, where training samples are generated by the student model itself rather than drawn from a fixed offline dataset. Here, the KL direction affects not only the optimization objective but also the gradient estimation method, sampling efficiency, and the risk of mode collapse. This article covers the definitions, geometric intuition, and on-policy practice, and provides an implementable algorithm workflow along with selection guidelines.
1. Background: What Is On-Policy Self-Distillation
Let the teacher model distribution be (P) and the student model distribution be (Q). Conventional distillation minimizes some distribution divergence (D(P,Q)) so the student approaches the teacher. The key property of on-policy distillation is that training trajectories are sampled from the student’s current policy (Q), rather than relying entirely on teacher sampling or a fixed dataset. Self-Distillation further relaxes where the teacher comes from: the teacher can be a different version of the same model, an EMA of the weights, a version with additional context or prompts, a version corrected through rejection sampling, or an inference path with a larger compute budget.
This setup has two clear advantages:
- Mitigates exposure bias: The student receives teacher corrections on states it would actually reach, so the training distribution is closer to the inference distribution.
- More controllable sampling cost: There is no need to frequently generate large amounts of trajectories from the teacher; student samples alone can form the training data.
But it also introduces new challenges: samples come from (Q), so the loss estimate is naturally better suited to Reverse KL. If you mistakenly use a sampled estimate of Forward KL, you may encounter high-variance importance weights; if you rely solely on Reverse KL for its stable focusing behavior, you risk mode collapse. Therefore, you must first clarify who (P) and (Q) are before discussing the KL direction.
2. Core Differences Between Forward KL and Reverse KL
1. Forward KL: Covering All High-Probability Teacher Regions
Forward KL is defined as:
[
D_{\mathrm{KL}}(P\Vert Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}
]
Minimizing (D_{\mathrm{KL}}(P\Vert Q)) is equivalent to maximizing (\sum_x P(x)\log Q(x)), i.e., the standard cross-entropy loss. Its expectation is computed under the teacher distribution (P).
Behaviorally, Forward KL is mean-seeking / mode-covering: if the teacher assigns high probability somewhere and the student assigns very low probability there, the penalty is large. The student is therefore forced to cover every high-probability mode the teacher considers important. If the teacher is a multimodal distribution, the student may be pushed to allocate mass to every mode, and even retain some probability in low-probability regions to avoid missing anything.
In logits distillation, when the teacher provides the full soft label distribution, the commonly used soft cross-entropy essentially corresponds to the Forward KL direction. Standard SFT with hard labels can also be seen as forward cross-entropy under an extreme one-hot teacher.
2. Reverse KL: Focusing on High-Probability Teacher Regions
Reverse KL is defined as:
[
D_{\mathrm{KL}}(Q\Vert P)=\sum_x Q(x)\log\frac{Q(x)}{P(x)}
]
Note that the weighting is on (Q). If the student assigns high probability to regions where the teacher assigns low probability, the penalty is severe; conversely, if the teacher is highly probable somewhere but the student rarely samples there, the contribution is negligible as long as (Q(x)\approx 0).
Behaviorally, Reverse KL is mode-seeking: the student tends to concentrate on one or a few high-probability modes of the teacher rather than covering all of them. It essentially enforces “don’t go where the teacher considers impossible,” but does not guarantee “learn every region the teacher considers important.” With a finite vocabulary and smoothing, be careful to avoid numerical issues when (Q(x)=0) while (P(x)>0).
3. Geometric Intuition and Gradient Differences
From the support perspective, Forward KL is extremely sensitive to regions where “the teacher has probability but the student is near zero,” and tends to expand the student’s support. Reverse KL is extremely sensitive to regions where “the student has probability but the teacher is near zero,” and tends to shrink the student’s support.
From the gradient perspective, Forward KL gradients are weighted by teacher probabilities, so all teacher high-probability regions pull on the student. Reverse KL gradients are weighted by student probabilities, so only regions the student currently samples get corrected. In on-policy settings, Reverse KL therefore pairs naturally with the “student samples, student-weighted” Monte Carlo estimator; Forward KL requires teacher sampling or a full-vocabulary sum, otherwise it needs importance weights (P/Q), which can have very high variance.
3. Why the KL Direction Is Critical in On-Policy Settings
1. Reverse KL Naturally Matches Student Sampling
In on-policy training, samples follow (y\sim Q). The expectation of Reverse KL is exactly:
[
D{\mathrm{KL}}(Q\Vert P)=\mathbb{E}{y\sim Q}\left[\log Q(y)-\log P(y)\right]
]
So you can estimate it directly from student-sampled trajectories, with single-sample or multi-sample estimators, without sampling from the teacher. For token-level distributions, you can compute (\log Q_t(y_t)-\log P_t(y_t)) for the tokens the student sampled, or use a full-vocabulary sum for a more precise Reverse KL. This makes Reverse KL easy to implement in on-policy distillation with relatively controllable gradient variance.
2. Forward KL Is Harder to Estimate in On-Policy Settings
The expectation of Forward KL is under (P):
[
D{\mathrm{KL}}(P\Vert Q)=\mathbb{E}{y\sim P}\left[\log P(y)-\log Q(y)\right]
]
If you only have student samples (y\sim Q), you need importance sampling:
[
\mathbb{E}_{y\sim Q}\left[\frac{P(y)}{Q(y)}\left(\log P(y)-\log Q(y)\right)\right]
]
When the student underestimates a teacher high-probability region, (P/Q) can become very large, causing high variance and even training instability. Forward KL is therefore better suited to token-level distillation where the teacher can provide the full vocabulary distribution for full-vocabulary computation; directly estimating Forward KL from purely sequence-level on-policy sampling is not ideal.
3. Specific Risks in Self-Distillation
In Self-Distillation, the teacher and student share a common origin, so the distribution gap is usually small and the effect of KL direction may seem weakened. In practice, three risks remain:
- A teacher with extra context can be multimodal: If the teacher produces multiple plausible modes due to prompts, retrieval, or reasoning paths, Reverse KL may only learn one of them.
- Student sampling bias is self-reinforcing: Reverse KL only corrects regions the student samples; the less the student explores a region, the harder it is to fix.
- Teacher uncertainty can be over-sharpened: If the teacher distribution is relatively flat, Reverse KL may push the student to converge too early, reducing diversity.
Therefore, Self-Distillation should not mechanically pick a single direction. The choice should depend on whether the task requires multimodal coverage, whether the teacher provides the full distribution, and whether the sampling budget is sufficient.
4. Token-Level vs. Sequence-Level KL in Practice
Token-level KL compares vocabulary distributions at each position, offering high information density, and suits scenarios where the teacher can output logits. Forward KL requires a full-vocabulary sum, which is computationally heavier but stable; Reverse KL can use either the full vocabulary or an estimate based on the tokens the student sampled. The advantage of token-level losses is fine-grained credit assignment; the drawback is that they may miss long-sequence consistency.
Sequence-level KL compares the distribution over entire trajectories, which is closer to the generation objective, but estimates have high variance and sparse credit assignment. After sampling a full trajectory on-policy, sequence-level Reverse KL estimation is natural; sequence-level Forward KL estimation requires teacher-side trajectories or importance weights. In practice, length normalization, discounted token-level accumulation, and mixing sequence-level with token-level losses are commonly used to balance stability and consistency.
Temperature also matters. A high-temperature teacher distribution is smoother, reducing Reverse KL’s mode collapse risk; a low-temperature teacher is sharper, making it easier for the student to focus on a few modes. Teacher and student temperatures should be explicitly recorded and kept consistent or proportionally calibrated. Top-k and top-p truncation change the support of the distribution: Forward KL is sensitive to teacher-side truncation, while Reverse KL is insensitive to the teacher’s low-probability tail but may ignore plausible modes that were truncated.
5. Algorithm Workflow and Pseudocode
Below is a general On-Policy Self-Distillation workflow. The teacher’s parameters are gradient-stopped, the student samples trajectories, the teacher provides a distribution at each position or scores the sampled tokens, and the loss is computed according to the chosen KL direction.
Inputs: student Q, teacher P (EMA / context-augmented / expert-corrected version),sampling temperature T, KL weights β_fwd and β_rev, entropy weight λRepeat:1. Sample input x from the dataset2. Student samples a trajectory y = (y_1,...,y_L) ~ Q(·|x) at temperature T3. Teacher computes distribution P_t(·|x,y_<t) at each position t, with gradients stopped4. Student computes the corresponding distribution Q_t(·|x,y_<t)5. Compute the loss:L_rev = mean_t sum_v Q_t(v) * (log Q_t(v) - log P_t(v))or sampled estimate L_rev_sample = mean_t (log Q_t(y_t) - log P_t(y_t))L_fwd = mean_t sum_v P_t(v) * (log P_t(v) - log Q_t(v))L_ent = -mean_t sum_v Q_t(v) * log Q_t(v)L = β_rev * L_rev + β_fwd * L_fwd - λ * L_ent6. Backpropagate and update the student Q7. Optional: update the teacher P via EMAUntil convergence
If memory or vocabulary computation is limited, use the Reverse KL estimate based on student-sampled tokens, and periodically add full-vocabulary Forward KL or full-vocabulary Reverse KL as a calibration step. If the teacher can only score complete sequences and cannot provide token-level distributions, switch to a sequence-level preference or rejection-sampling objective—do not pretend you have token-level KL.
6. Selection Guidelines and Diagnostic Metrics
Use the following principles when choosing a KL direction:
- Need multimodal coverage and must not miss important teacher modes: Prefer Forward KL, or a Forward/Reverse KL mixture; the prerequisite is that the teacher can provide the full distribution or sufficient sampling.
- Need stable focusing and want to keep the student out of teacher low-probability regions: Prefer Reverse KL, combined with entropy regularization, temperature smoothing, and exploratory sampling.
- Purely on-policy student sampling with a teacher that only scores sampled trajectories: Reverse KL is more natural; if you want Forward KL, use full-vocabulary computation or importance sampling and monitor variance.
- Self-Distillation with a same-origin teacher: Reverse KL as the primary objective is recommended, with periodic Forward KL coverage terms or diversity constraints to avoid mode collapse.
At minimum, monitor these diagnostics: the coverage gap ratio (teacher high-probability but student low-probability), the out-of-bounds ratio (student high-probability but teacher low-probability), student entropy and sample diversity, KL estimation variance, and the degree of distribution sharpening across temperatures. Only by considering the KL direction together with the sampling source, teacher characteristics, and task multimodality can On-Policy Self-Distillation remain stable without excessive collapse.
