Large Language Model Knowledge Distillation: Teacher Trajectories and Logit Distillation Explained
This article explains knowledge distillation for large language models, with a focus on teacher trajectories and two common training routes: logit distillation and sequence sampling. It compares their information density, training cost, memory footprint, and applicability, and discusses practical engineering concerns such as temperature tuning, data quality control, training stability, and capability evaluation. The goal is to help readers choose a suitable distillation approach when transferring reasoning capabilities from a large teacher model to a smaller student model.
Introduction: Why Knowledge Distillation Matters
In the era of large models, improving model capability often comes with a sharp increase in parameter count. A strong teacher model with reasoning ability may have tens or even hundreds of billions of parameters, making it expensive to run and difficult to deploy. In real business scenarios, teams usually want a smaller student model that is faster, cheaper to serve, and easier to deploy while still preserving as much of the teacher model’s capability distribution as possible. Knowledge distillation is a key technique for addressing this tradeoff.
The core idea is simple: let a smaller student model imitate the output behavior of a larger teacher model. Unlike traditional hard-label training, which only learns the correct answer, distillation also uses the soft information contained in the teacher’s output, especially the probability distribution over candidate tokens. These distributions often carry transferable knowledge beyond the single correct label.
This article focuses on the training paradigm of teacher trajectories and compares two mainstream technical routes: logit distillation and sequence sampling.
1. What Are Teacher Trajectories?
Teacher trajectories refer to the complete token sequences generated by a teacher model when solving a task. These sequences include not only the final answer but also intermediate reasoning steps, often referred to as the “thinking process” or chain of thought.
Unlike traditional supervised data that only contains input-final-output pairs, teacher trajectories preserve process information. This is especially important when training reasoning-oriented models. A student model that only imitates final answers may struggle to learn how to derive those answers step by step. Complete trajectory data can explicitly convey intermediate logic.
In practice, there are two common ways to use this information:
- Use the teacher’s full next-token distribution at every step, commonly known as logit distillation.
- Sample the given sequence produced by the teacher, often referred to as sequence sampling or hard-label imitation.
These two approaches differ in information density, training cost, and effectiveness. The following sections explain them in detail.
2. Route One: Logit Distillation
Logit distillation is the most classic and information-dense distillation method. Its key idea is not only to tell the student what the correct answer is, but also to tell the student how likely the teacher considers each candidate token to be.
2.1 Core Principle
For a sequence of length T, the teacher model outputs a next-token probability distribution over the entire vocabulary at every step t:
p_teacher(· | x, y_<t) ∈ ℝ^V
Here, V is the vocabulary size. The student model also outputs a distribution, denoted as p_student. The distillation objective is to make these two distributions as close as possible, commonly using KL divergence:
import torchimport torch.nn.functional as Fdef logit_distillation_loss(student_logits, teacher_logits, temperature=2.0):"""student_logits / teacher_logits: [batch, seq_len, vocab_size]temperature: temperature coefficient used to smooth the distributions"""# Temperature scaling softens the probability distribution and exposes more dark knowledgestudent_soft = F.log_softmax(student_logits / temperature, dim=-1)teacher_soft = F.softmax(teacher_logits / temperature, dim=-1)# Align the soft labels at every position using KL divergencekl = F.kl_div(student_soft, teacher_soft, reduction="batchmean")# Multiply by T^2 to balance gradient scales, a classic distillation practicereturn kl * (temperature ** 2)
2.2 The Role of Temperature
The temperature coefficient T is a key hyperparameter in distillation. When T = 1, the distribution is the sharpest, and probability mass is concentrated on a few high-confidence tokens. When T increases, the distribution becomes smoother, and tokens that originally had small but nonzero probabilities, often called dark knowledge, are amplified.
The relative relationships among these low-probability tokens can encode the teacher model’s world knowledge. For example, when predicting the word “cat,” a teacher model may assign nonzero probabilities to words such as “dog” and “rabbit.” The relative ranking of these second-best candidates can convey semantic similarity. The temperature coefficient acts like a microscope that reveals this information.
2.3 Advantages and Costs
The main advantage of logit distillation is its high information density. It provides full-vocabulary supervision at every step, leading to fast convergence and strong transfer effects. However, the costs are also significant:
- Higher computation and memory overhead: Both the teacher and student distributions must be retained for every step.
- Stricter alignment requirements: The teacher and student vocabularies must be identical, or vocabulary mapping must be performed.
- Teacher inference requirements: Online distillation requires loading both models at the same time. Offline distillation requires caching massive distribution tensors in advance, which creates substantial storage costs.
3. Route Two: Sequence Sampling
Sequence sampling is a lighter-weight distillation method. The teacher model first generates complete reasoning trajectories. Then the student model is trained on these generated sequences using standard supervised learning, namely next-token prediction with hard labels and cross-entropy loss.
3.1 Core Principle
def sequence_sampling_loss(student_logits, target_tokens):"""target_tokens: full sequence generated by the teacher, including intermediate reasoningShape [batch, seq_len], used as hard labels"""return F.cross_entropy(student_logits.reshape(-1, student_logits.size(-1)),target_tokens.reshape(-1),ignore_index=-100 # Ignore padding)
At this point, the method does not align the probability distribution at every step. Instead, it asks the student model to fit the specific trajectory sampled by the teacher. In essence, this is supervised fine-tuning with teacher-generated data.
3.2 Advantages and Costs
The advantages include:
- Simplicity: There is no need to handle distribution alignment, and the standard supervised fine-tuning pipeline can be reused directly.
- Memory friendliness: It does not require retaining the teacher’s full distribution at every step. The teacher can usually generate data offline in batches.
- Flexibility: Teacher trajectories can be filtered, deduplicated, weighted, or manually reviewed to improve data quality.
The main cost is information loss. A sampled sequence only retains one point estimate from the teacher’s distribution and loses the relative probabilities of other candidate tokens. As a result, convergence may be slower, and the upper bound of transfer may be slightly lower than logit distillation. However, in reasoning capability transfer scenarios, high-quality trajectory data can often narrow this gap.
4. Comparison and Selection
| Dimension | Logit Distillation | Sequence Sampling |
|---|---|---|
| Information density | High, full-vocabulary distributions | Medium, a single sampled sequence |
| Training cost | High, distributions must be retained or transferred | Low, standard supervised fine-tuning workflow |
| Memory overhead | Large | Small |
| Vocabulary requirement | Vocabularies must match or require mapping | No strict constraint |
| Data controllability | Weak, distributions are difficult to filter | Strong, trajectories can be filtered, reviewed, or reweighted |
| Typical use case | High-fidelity alignment of teacher behavior | Large-scale, cost-sensitive, reasoning trajectory transfer |
In practice, the following selection guidance is useful:
- If the goal is high-fidelity reproduction of teacher behavior and sufficient compute resources are available, logit distillation can be considered first.
- If the goal is to transfer reasoning ability from a stronger teacher model and high-quality teacher trajectory data is available, sequence sampling is often more cost-effective.
- The two methods can also be combined. For example, hard-label supervision can be applied on trajectories while adding a logit alignment loss, balancing stability and effectiveness.
5. Engineering Practice
5.1 Prioritize Data Quality
The effectiveness of distillation depends heavily on the quality of teacher data. Teachers may occasionally produce incorrect reasoning or hallucinated steps. If such outputs are passed directly to the student model without filtering, noise may be amplified.
In practice, techniques such as correctness verification for tasks with verifiable answers, consistency filtering, and multi-sampling voting can be used to improve trajectory quality.
5.2 Balance Temperature and Loss
The temperature coefficient should be tuned according to the task. If the temperature is too low, dark knowledge is ignored and the method degenerates into hard-label learning. If the temperature is too high, the distribution becomes overly smooth and loses discriminative power.
A common practice is to perform grid search over values between 1.5 and 4.
5.3 Training Stability
Several practical considerations can improve training stability:
- Distillation loss is usually combined with the task’s own loss, such as cross-entropy, to prevent the student model from focusing only on distribution matching and losing attention to correct answers.
- In online distillation, the teacher model’s inference resource usage should be monitored. Asynchronous or time-shifted scheduling can help reduce peak pressure.
- In offline distillation, distribution cache storage and access should be planned carefully. Tiered caching can be combined with general-purpose object storage services.
5.4 Capability Evaluation
Distillation should not be evaluated based only on training loss. The student model’s actual performance on downstream tasks must also be measured.
It is recommended to build an evaluation set that covers the target capabilities and compare the student model with both the teacher model and an undistilled baseline. This helps confirm that knowledge has been effectively transferred rather than simply degraded.
6. Conclusion
Teacher trajectory training provides a clear path for capability transfer in large language models. Logit distillation pursues high-fidelity alignment through higher information density, while sequence sampling enables flexible reasoning capability transfer at lower cost. The two methods are not mutually exclusive; they can be combined flexibly according to compute budget, data scale, and task goals.
For teams that want to deploy model compression and capability transfer at scale, understanding the essential differences between these two routes, as well as mastering key practices such as temperature tuning and data quality control, is fundamental to building an efficient and stable distillation pipeline. On general-purpose cloud platforms such as Baidu AI Cloud, developers can use common training and storage capabilities to support the full pipeline from teacher-data generation to student-model training, turning strong-model capabilities into scalable and deployable services.
