On-Policy Distillation in Practice: A Guide to KL Selection and Multi-Teacher Orchestration
On-Policy Distillation (OPD) trains a student model on its own sampled responses, with a fixed-parameter teacher supplying dense, position-by-position supervision. This guide explains how OPD aligns the training and inference distributions to reduce exposure bias, when to choose sampled-token versus full-vocabulary supervision, and why forward KL and reverse KL behave differently (mode coverage versus mode seeking). It also provides a minimal PyTorch training loop and loss implementation, a three-layer pattern for orchestrating multiple specialized teachers, a deployment checklist, and answers to common FAQs. Use it to decide which OPD variants and trade-offs fit your post-training pipeline.
1. The Short Answer: What On-Policy Distillation Is
In one sentence: On-Policy Distillation (OPD) means letting the student sample and generate its own responses for the current prompt, then having a fixed-parameter teacher provide position-by-position supervision on these student-generated sequences. Its value comes from two things: aligning the training distribution with the inference distribution (which mitigates exposure bias), and giving the teacher dense supervision (more efficient than sparse rewards).
It’s worth clarifying the boundaries. In OPD, “On-Policy” specifically means the student self-samples while the teacher’s parameters stay fixed and only provide supervision. It does not cover cases where the teacher updates during training. It is also not DPO (contrastive learning on preference pairs) or RLHF (reward model plus policy gradient). These three are often used interchangeably, but their data sources, supervision forms, and optimization objectives all differ, so be sure to distinguish them in practice.
2. On-Policy vs. Off-Policy: Is the Training Distribution Aligned
The prerequisite for understanding OPD is being clear about “where the data comes from.”
- Off-Policy (offline) distillation: the student learns on a fixed dataset whose samples come from ground-truth labels, or from teacher outputs that were generated in advance and never change afterward.
- On-Policy distillation: the student first rolls out to generate its own responses, and the teacher then provides position-by-position supervision on these “student-produced sequences.”
The difference looks like it’s only about the data source, but it actually determines whether the training and inference distributions match. Offline distillation suffers from typical exposure bias: during training the student only ever sees teacher-style context, but during inference it must keep generating from its own possibly-erroneous prefixes. Once it drifts away from the training distribution, the errors accumulate and amplify along the autoregressive chain. On-Policy naturally aligns the training and inference distributions: the student is corrected precisely in the state it will actually grow into. This is its most fundamental advantage over offline distillation.
3. Two Supervision Granularities: Sampled Token vs. Full Vocabulary
Within the On-Policy framework, there are two routes for how the teacher “gives the signal”:
- Sampled-token distillation: compare the teacher’s and student’s log-probabilities only on the single token the student actually sampled. This is essentially sparse, policy-gradient-style feedback. It’s cheap and naturally fits RL-style optimization, but it compresses the richness of the teacher distribution into a scalar.
- Full-vocabulary distillation: at each position, provide the probability distribution over the entire vocabulary, and have the student approximate the whole distribution. This is dense supervision with higher sample efficiency and more stable convergence; the cost is retaining and comparing logits at the [B, T, V] scale, which demands more memory and a heavier teacher forward pass.
Should you prioritize full vocabulary when you can? It depends on your memory budget and teacher throughput: when compute allows, prefer full vocabulary and trade denser supervision for fewer iterations; when memory is tight or you need to integrate quickly with an existing RL pipeline, sampled-token is the lighter choice.
| Dimension | Sampled-Token Distillation | Full-Vocabulary Distillation |
|---|---|---|
| Supervision signal | Single-token log-prob difference (sparse) | Whole-vocabulary probability distribution (dense) |
| Memory cost | Low (only gathered logits needed) | High ([B, T, V] logits required) |
| Teacher forward | One forward pass | One forward pass, but heavy comparison overhead |
| Convergence | Needs more steps, noisier | Sample-efficient, more stable convergence |
| Best fit | Tight memory, fast RL integration | Ample compute, aiming for stable reproduction |
| Common failure | Signal too sparse, slow convergence | OOM; requires vocabulary chunking |
4. Forward KL vs. Reverse KL: Mode Coverage vs. Mode Seeking
Distribution alignment is characterized by KL divergence, and the direction matters enormously:
- Forward KL (D_KL(teacher‖student)): favors mode coverage, forcing the student to cover all the teacher’s possible outputs. When the student’s capacity is smaller than the teacher’s, it spreads probability thin across multiple modes and produces blurry, averaged-out outputs.
- Reverse KL (D_KL(student‖teacher)): favors mode seeking, concentrating the student’s probability mass into the teacher’s high-confidence regions. For a capacity-limited student whose goal is stable reproduction of the teacher’s core behavior, reverse KL usually converges to a cleaner, more usable distribution.
This is also why many full-vocabulary OPD practices today choose reverse KL: the student doesn’t need to replicate all of the teacher’s diversity, only to reliably land in the regions the teacher considers “correct.” The direction choice is still best confirmed with a small validation set rather than copied wholesale.
| Dimension | Forward KL D_KL(teacher‖student) | Reverse KL D_KL(student‖teacher) |
|---|---|---|
| Behavioral tendency | Mode coverage | Mode seeking |
| Output characteristics | Broad coverage, but may average/blur | Concentrated, sharp, clean |
| Best fit | Preserving teacher diversity | Stable reproduction of core behavior |
| Memory/compute | Comparable | Comparable |
| Common failure | Blurry outputs when capacity is insufficient | Premature convergence, drops long-tail correct modes |
5. A Ready-to-Use Training Loop and Loss Implementation
Below is the core loss and a minimal training-loop skeleton (PyTorch style). Key points: the teacher forward pass must run under no_grad and be detached; response_mask supervises only the assistant response; reverse KL is weighted by the student’s probabilities; and the sampled-token version is a Monte Carlo (sampling) estimate of reverse KL—it takes (log p_s - log p_t) on the sampled token, whose expectation equals reverse KL, so keep the sign and sampling distribution consistent.
import torchimport torch.nn.functional as Fdef full_vocab_reverse_kl(student_logits, teacher_logits, mask):# student_logits, teacher_logits: [B, T, V]; mask: [B, T], 1 only at response positionsstudent_logp = F.log_softmax(student_logits, dim=-1)teacher_logp = F.log_softmax(teacher_logits, dim=-1)# reverse KL: sum_v p_s * (log p_s - log p_t)kl = (student_logp.exp() * (student_logp - teacher_logp)).sum(dim=-1) # [B, T]return (kl * mask).sum() / mask.sum().clamp(min=1.0)def sampled_token_reverse_kl(student_logits, teacher_logits, sampled_ids, mask):# sampling estimate of reverse KL: take (log p_s - log p_t) on the token the student sampledstudent_logp = F.log_softmax(student_logits, dim=-1)teacher_logp = F.log_softmax(teacher_logits, dim=-1)s = student_logp.gather(-1, sampled_ids.unsqueeze(-1)).squeeze(-1)t = teacher_logp.gather(-1, sampled_ids.unsqueeze(-1)).squeeze(-1)kl = (s - t) * maskreturn kl.sum() / mask.sum().clamp(min=1.0)@torch.no_grad()def teacher_forward(teacher, input_ids):# teacher parameters are fixed; run under no_grad throughout; already detached on return to avoid building a graphreturn teacher(input_ids).logitsdef train_step(student, teacher, prompt_ids, gen_cfg, opt, tokenizer,max_new=256, grad_accum=4):opt.zero_grad()for _ in range(grad_accum):# 1) student rollout: self-sample a response (no_grad)student.eval()with torch.no_grad():seq = student.generate(prompt_ids, **gen_cfg, max_new_tokens=max_new)student.train()# 2) build response_mask: mask out prompt and padding, supervise only the assistant responseprompt_len = prompt_ids.shape[1]attn = (seq != tokenizer.pad_token_id).long()response_mask = attn.clone()response_mask[:, :prompt_len] = 0# 3) teacher forward (no_grad, fixed supervision)teacher_logits = teacher_forward(teacher, seq)# 4) student forward (needs gradient)student_logits = student(seq).logits# 5) shift alignment: use logits at position t-1 to predict position ts_lp = student_logits[:, :-1]t_lp = teacher_logits[:, :-1]m = response_mask[:, 1:]loss = full_vocab_reverse_kl(s_lp, t_lp, m) / grad_accum# 6) accumulate and backproploss.backward()# 7) gradient clipping + steptorch.nn.utils.clip_grad_norm_(student.parameters(), 1.0)opt.step()return loss.detach() * grad_accum
Engineering details: mask masks out padding and the prompt segment; reverse KL must be weighted by the student’s probabilities (student_logp.exp()); in the full-vocabulary approach the [B, T, V] logits are the main source of memory pressure, so you can accumulate KL chunked over the vocabulary dimension to lower the peak; in real training, use grad_accum to accumulate several micro-batches before stepping, and optionally layer on AMP mixed precision to save further memory.
6. Multi-Teacher Distillation and Expert-Model Orchestration
When capabilities need to span multiple heterogeneous domains, a single teacher is hard-pressed to do it all. You can introduce multi-teacher orchestration, typically structured in three layers:
- Specialized teacher layer: train a dedicated teacher for each domain—math, code, agents, instruction following—each refined through SFT + RL, with a higher domain ceiling.
- Routing/aggregation layer: decide which teacher to use for each prompt, or how to aggregate multiple teacher distributions.
- Unified student layer: the student self-samples rollouts and receives token-by-token full-vocabulary reverse-KL supervision from the routed teacher, converging multi-domain capability into a single set of weights.
| Routing strategy | Trigger condition | Advantages | Risks |
|---|---|---|---|
| Hard routing (by domain classification) | Prompt’s domain is clear | Pure signal, simple to implement | Misclassification directly pollutes supervision |
| Confidence weighting | Teachers disagree strongly on the same sequence | Smooth, robust to single-point errors | Weight calibration is costly |
| Token-level mixing | Response switches across domains | Fine-grained alignment | Heavy compute, complex to implement |
Compared with a “one-shot big mixed RL,” a “divide-and-conquer, then aggregate” approach breaks the complex reward calibration into several controllable sub-problems, giving each specialized teacher a purer internal signal; at the student stage, dense teacher distributions replace sparse rewards, making training more stable. On the compute advantage: some public practices (whose conclusions vary with model scale, teacher count, and evaluation methodology) suggest OPD can, for the same effect target, achieve comparable results at lower compute than full RL; the specific gains should be validated against your own evaluation set and compute budget, and others’ numbers shouldn’t be copied directly.
7. Deployment Checklist and Common Trade-offs
Before adopting OPD, work through this list item by item:
- Fix the teacher version: record the version number/hash of the teacher weights and don’t update them for the entire training run.
- Routing fallback: when routing confidence is below a threshold, fall back to a general or default teacher to avoid erroneous supervision.
- Vocabulary chunk size: set a chunk size (e.g., 8k/16k) and validate the balance point between peak memory and throughput.
- Validation-set KL comparison: run small samples for both forward and reverse KL, and pick the direction using validation-set metrics.
- RL mixing strategy: define the data ratio and switch point where the specialization stage uses RL to polish teachers and the aggregation stage uses OPD to shape the student.
- Data and compliance: confirm the teacher model’s license and commercial terms, the source and copyright of training data, and the desensitization of privacy information in logs.
- Training rollback: keep checkpoints and early-stopping criteria so you can revert to the last stable version if routing or a teacher misbehaves.
FAQ
- How is On-Policy Distillation different from DPO and RLHF? OPD uses student self-sampling plus dense teacher-distribution supervision; DPO does contrastive optimization on preference pairs; RLHF uses a reward model plus policy gradient. All three differ in optimization objective and data source.
- How do I choose the KL direction? Use forward KL to preserve diversity coverage, and reverse KL to converge stably to high-confidence behavior; confirm the final choice with a validation-set comparison.
- What are the common mistakes? Forgetting
no_gradon the teacher, which builds a graph or leaks gradients; forgetting to mask the prompt inresponse_mask; forgetting to shift by one position for alignment; running full vocabulary without vocabulary chunking and hitting OOM directly. - What if I’m short on memory? Reduce batch/sequence length, accumulate KL chunked over the vocabulary dimension, enable AMP, and use gradient accumulation instead of large batches.
Conclusion
The value of On-Policy Distillation lies in two things working together—training-distribution alignment plus dense teacher supervision—to simultaneously ease the exposure bias of offline distillation and the sample inefficiency of pure RL. From sampled-token to full-vocabulary, from forward to reverse KL, from a single teacher to multi-teacher orchestration, this family of methods offers a set of composable knobs that turn post-training from “a mixed RL that’s hard to tune” into “a distillation pipeline you can divide, reuse, and scale.” For teams preparing to improve their post-training process, understanding these variants and trade-offs is more valuable in the long run than memorizing any single configuration.
