On-Policy Distillation in Practice: A Guide to KL Selection and Multi-Teacher Orchestration

    On-Policy Distillation (OPD) trains a student model on its own sampled responses, with a fixed-parameter teacher supplying dense, position-by-position supervision. This guide explains how OPD aligns the training and inference distributions to reduce exposure bias, when to choose sampled-token versus full-vocabulary supervision, and why forward KL and reverse KL behave differently (mode coverage versus mode seeking). It also provides a minimal PyTorch training loop and loss implementation, a three-layer pattern for orchestrating multiple specialized teachers, a deployment checklist, and answers to common FAQs. Use it to decide which OPD variants and trade-offs fit your post-training pipeline.

    1. The Short Answer: What On-Policy Distillation Is

    In one sentence: On-Policy Distillation (OPD) means letting the student sample and generate its own responses for the current prompt, then having a fixed-parameter teacher provide position-by-position supervision on these student-generated sequences. Its value comes from two things: aligning the training distribution with the inference distribution (which mitigates exposure bias), and giving the teacher dense supervision (more efficient than sparse rewards).

    It’s worth clarifying the boundaries. In OPD, “On-Policy” specifically means the student self-samples while the teacher’s parameters stay fixed and only provide supervision. It does not cover cases where the teacher updates during training. It is also not DPO (contrastive learning on preference pairs) or RLHF (reward model plus policy gradient). These three are often used interchangeably, but their data sources, supervision forms, and optimization objectives all differ, so be sure to distinguish them in practice.

    2. On-Policy vs. Off-Policy: Is the Training Distribution Aligned

    The prerequisite for understanding OPD is being clear about “where the data comes from.”

    • Off-Policy (offline) distillation: the student learns on a fixed dataset whose samples come from ground-truth labels, or from teacher outputs that were generated in advance and never change afterward.
    • On-Policy distillation: the student first rolls out to generate its own responses, and the teacher then provides position-by-position supervision on these “student-produced sequences.”

    The difference looks like it’s only about the data source, but it actually determines whether the training and inference distributions match. Offline distillation suffers from typical exposure bias: during training the student only ever sees teacher-style context, but during inference it must keep generating from its own possibly-erroneous prefixes. Once it drifts away from the training distribution, the errors accumulate and amplify along the autoregressive chain. On-Policy naturally aligns the training and inference distributions: the student is corrected precisely in the state it will actually grow into. This is its most fundamental advantage over offline distillation.

    3. Two Supervision Granularities: Sampled Token vs. Full Vocabulary

    Within the On-Policy framework, there are two routes for how the teacher “gives the signal”:

    • Sampled-token distillation: compare the teacher’s and student’s log-probabilities only on the single token the student actually sampled. This is essentially sparse, policy-gradient-style feedback. It’s cheap and naturally fits RL-style optimization, but it compresses the richness of the teacher distribution into a scalar.
    • Full-vocabulary distillation: at each position, provide the probability distribution over the entire vocabulary, and have the student approximate the whole distribution. This is dense supervision with higher sample efficiency and more stable convergence; the cost is retaining and comparing logits at the [B, T, V] scale, which demands more memory and a heavier teacher forward pass.

    Should you prioritize full vocabulary when you can? It depends on your memory budget and teacher throughput: when compute allows, prefer full vocabulary and trade denser supervision for fewer iterations; when memory is tight or you need to integrate quickly with an existing RL pipeline, sampled-token is the lighter choice.

    Dimension Sampled-Token Distillation Full-Vocabulary Distillation
    Supervision signal Single-token log-prob difference (sparse) Whole-vocabulary probability distribution (dense)
    Memory cost Low (only gathered logits needed) High ([B, T, V] logits required)
    Teacher forward One forward pass One forward pass, but heavy comparison overhead
    Convergence Needs more steps, noisier Sample-efficient, more stable convergence
    Best fit Tight memory, fast RL integration Ample compute, aiming for stable reproduction
    Common failure Signal too sparse, slow convergence OOM; requires vocabulary chunking

    4. Forward KL vs. Reverse KL: Mode Coverage vs. Mode Seeking

    Distribution alignment is characterized by KL divergence, and the direction matters enormously:

    • Forward KL (D_KL(teacher‖student)): favors mode coverage, forcing the student to cover all the teacher’s possible outputs. When the student’s capacity is smaller than the teacher’s, it spreads probability thin across multiple modes and produces blurry, averaged-out outputs.
    • Reverse KL (D_KL(student‖teacher)): favors mode seeking, concentrating the student’s probability mass into the teacher’s high-confidence regions. For a capacity-limited student whose goal is stable reproduction of the teacher’s core behavior, reverse KL usually converges to a cleaner, more usable distribution.

    This is also why many full-vocabulary OPD practices today choose reverse KL: the student doesn’t need to replicate all of the teacher’s diversity, only to reliably land in the regions the teacher considers “correct.” The direction choice is still best confirmed with a small validation set rather than copied wholesale.

    Dimension Forward KL D_KL(teacher‖student) Reverse KL D_KL(student‖teacher)
    Behavioral tendency Mode coverage Mode seeking
    Output characteristics Broad coverage, but may average/blur Concentrated, sharp, clean
    Best fit Preserving teacher diversity Stable reproduction of core behavior
    Memory/compute Comparable Comparable
    Common failure Blurry outputs when capacity is insufficient Premature convergence, drops long-tail correct modes

    5. A Ready-to-Use Training Loop and Loss Implementation

    Below is the core loss and a minimal training-loop skeleton (PyTorch style). Key points: the teacher forward pass must run under no_grad and be detached; response_mask supervises only the assistant response; reverse KL is weighted by the student’s probabilities; and the sampled-token version is a Monte Carlo (sampling) estimate of reverse KL—it takes (log p_s - log p_t) on the sampled token, whose expectation equals reverse KL, so keep the sign and sampling distribution consistent.

    1. import torch
    2. import torch.nn.functional as F
    3. def full_vocab_reverse_kl(student_logits, teacher_logits, mask):
    4. # student_logits, teacher_logits: [B, T, V]; mask: [B, T], 1 only at response positions
    5. student_logp = F.log_softmax(student_logits, dim=-1)
    6. teacher_logp = F.log_softmax(teacher_logits, dim=-1)
    7. # reverse KL: sum_v p_s * (log p_s - log p_t)
    8. kl = (student_logp.exp() * (student_logp - teacher_logp)).sum(dim=-1) # [B, T]
    9. return (kl * mask).sum() / mask.sum().clamp(min=1.0)
    10. def sampled_token_reverse_kl(student_logits, teacher_logits, sampled_ids, mask):
    11. # sampling estimate of reverse KL: take (log p_s - log p_t) on the token the student sampled
    12. student_logp = F.log_softmax(student_logits, dim=-1)
    13. teacher_logp = F.log_softmax(teacher_logits, dim=-1)
    14. s = student_logp.gather(-1, sampled_ids.unsqueeze(-1)).squeeze(-1)
    15. t = teacher_logp.gather(-1, sampled_ids.unsqueeze(-1)).squeeze(-1)
    16. kl = (s - t) * mask
    17. return kl.sum() / mask.sum().clamp(min=1.0)
    18. @torch.no_grad()
    19. def teacher_forward(teacher, input_ids):
    20. # teacher parameters are fixed; run under no_grad throughout; already detached on return to avoid building a graph
    21. return teacher(input_ids).logits
    22. def train_step(student, teacher, prompt_ids, gen_cfg, opt, tokenizer,
    23. max_new=256, grad_accum=4):
    24. opt.zero_grad()
    25. for _ in range(grad_accum):
    26. # 1) student rollout: self-sample a response (no_grad)
    27. student.eval()
    28. with torch.no_grad():
    29. seq = student.generate(prompt_ids, **gen_cfg, max_new_tokens=max_new)
    30. student.train()
    31. # 2) build response_mask: mask out prompt and padding, supervise only the assistant response
    32. prompt_len = prompt_ids.shape[1]
    33. attn = (seq != tokenizer.pad_token_id).long()
    34. response_mask = attn.clone()
    35. response_mask[:, :prompt_len] = 0
    36. # 3) teacher forward (no_grad, fixed supervision)
    37. teacher_logits = teacher_forward(teacher, seq)
    38. # 4) student forward (needs gradient)
    39. student_logits = student(seq).logits
    40. # 5) shift alignment: use logits at position t-1 to predict position t
    41. s_lp = student_logits[:, :-1]
    42. t_lp = teacher_logits[:, :-1]
    43. m = response_mask[:, 1:]
    44. loss = full_vocab_reverse_kl(s_lp, t_lp, m) / grad_accum
    45. # 6) accumulate and backprop
    46. loss.backward()
    47. # 7) gradient clipping + step
    48. torch.nn.utils.clip_grad_norm_(student.parameters(), 1.0)
    49. opt.step()
    50. return loss.detach() * grad_accum

    Engineering details: mask masks out padding and the prompt segment; reverse KL must be weighted by the student’s probabilities (student_logp.exp()); in the full-vocabulary approach the [B, T, V] logits are the main source of memory pressure, so you can accumulate KL chunked over the vocabulary dimension to lower the peak; in real training, use grad_accum to accumulate several micro-batches before stepping, and optionally layer on AMP mixed precision to save further memory.

    6. Multi-Teacher Distillation and Expert-Model Orchestration

    When capabilities need to span multiple heterogeneous domains, a single teacher is hard-pressed to do it all. You can introduce multi-teacher orchestration, typically structured in three layers:

    1. Specialized teacher layer: train a dedicated teacher for each domain—math, code, agents, instruction following—each refined through SFT + RL, with a higher domain ceiling.
    2. Routing/aggregation layer: decide which teacher to use for each prompt, or how to aggregate multiple teacher distributions.
    3. Unified student layer: the student self-samples rollouts and receives token-by-token full-vocabulary reverse-KL supervision from the routed teacher, converging multi-domain capability into a single set of weights.
    Routing strategy Trigger condition Advantages Risks
    Hard routing (by domain classification) Prompt’s domain is clear Pure signal, simple to implement Misclassification directly pollutes supervision
    Confidence weighting Teachers disagree strongly on the same sequence Smooth, robust to single-point errors Weight calibration is costly
    Token-level mixing Response switches across domains Fine-grained alignment Heavy compute, complex to implement

    Compared with a “one-shot big mixed RL,” a “divide-and-conquer, then aggregate” approach breaks the complex reward calibration into several controllable sub-problems, giving each specialized teacher a purer internal signal; at the student stage, dense teacher distributions replace sparse rewards, making training more stable. On the compute advantage: some public practices (whose conclusions vary with model scale, teacher count, and evaluation methodology) suggest OPD can, for the same effect target, achieve comparable results at lower compute than full RL; the specific gains should be validated against your own evaluation set and compute budget, and others’ numbers shouldn’t be copied directly.

    7. Deployment Checklist and Common Trade-offs

    Before adopting OPD, work through this list item by item:

    • Fix the teacher version: record the version number/hash of the teacher weights and don’t update them for the entire training run.
    • Routing fallback: when routing confidence is below a threshold, fall back to a general or default teacher to avoid erroneous supervision.
    • Vocabulary chunk size: set a chunk size (e.g., 8k/16k) and validate the balance point between peak memory and throughput.
    • Validation-set KL comparison: run small samples for both forward and reverse KL, and pick the direction using validation-set metrics.
    • RL mixing strategy: define the data ratio and switch point where the specialization stage uses RL to polish teachers and the aggregation stage uses OPD to shape the student.
    • Data and compliance: confirm the teacher model’s license and commercial terms, the source and copyright of training data, and the desensitization of privacy information in logs.
    • Training rollback: keep checkpoints and early-stopping criteria so you can revert to the last stable version if routing or a teacher misbehaves.

    FAQ

    • How is On-Policy Distillation different from DPO and RLHF? OPD uses student self-sampling plus dense teacher-distribution supervision; DPO does contrastive optimization on preference pairs; RLHF uses a reward model plus policy gradient. All three differ in optimization objective and data source.
    • How do I choose the KL direction? Use forward KL to preserve diversity coverage, and reverse KL to converge stably to high-confidence behavior; confirm the final choice with a validation-set comparison.
    • What are the common mistakes? Forgetting no_grad on the teacher, which builds a graph or leaks gradients; forgetting to mask the prompt in response_mask; forgetting to shift by one position for alignment; running full vocabulary without vocabulary chunking and hitting OOM directly.
    • What if I’m short on memory? Reduce batch/sequence length, accumulate KL chunked over the vocabulary dimension, enable AMP, and use gradient accumulation instead of large batches.

    Conclusion

    The value of On-Policy Distillation lies in two things working together—training-distribution alignment plus dense teacher supervision—to simultaneously ease the exposure bias of offline distillation and the sample inefficiency of pure RL. From sampled-token to full-vocabulary, from forward to reverse KL, from a single teacher to multi-teacher orchestration, this family of methods offers a set of composable knobs that turn post-training from “a mixed RL that’s hard to tune” into “a distillation pipeline you can divide, reuse, and scale.” For teams preparing to improve their post-training process, understanding these variants and trade-offs is more valuable in the long run than memorizing any single configuration.