The “Distillation” Argument: What Knowledge Distillation Can and Cannot Transfer in LLMs

    This article takes a technical look at the common claim that a language model’s capability comes entirely from knowledge distillation, so cutting off the source would make its progress stall. It explains how distillation works through the teacher-student framework, and what its output-level transfer channel can and cannot pass on. Distillation readily transfers surface-level style, while distribution modeling, multi-step reasoning, generalization, and engineering capability cannot simply be copied. These capabilities are shaped by data engineering, training engineering, and alignment and evaluation work. A better way to respond to the argument is also offered.

    An Unequal Balance of Attack and Defense

    In discussions of large language models, one claim keeps resurfacing: one side’s capability “comes entirely from distillation,” and its progress will stall once the source is cut off. The claim spreads efficiently for a subtle reason. The people making it only need to repeat the word “distillation,” while anyone rebutting it has to put together a long argument to prove their point. Attack and defense are inherently unequal in cost, which makes the “distillation argument” especially persistent in public debate.

    To respond to it rationally, we first need to go back to the technology itself and answer two questions: what exactly is knowledge distillation, and what can and cannot be learned through it?

    What Knowledge Distillation Really Is

    Knowledge Distillation (KD) is a mature technique in deep learning for compressing models and transferring their capabilities. It is usually described through a teacher-student framework: the “soft labels” produced by a larger, stronger teacher model serve as an additional supervision signal that guides a smaller student model.

    Ordinary training relies only on the “correct answers” (hard labels). Soft labels, by contrast, carry the relative probability distribution across classes, which captures the teacher model’s “understanding” of a sample. A common approach is to have the student model fit both the ground-truth labels and the teacher’s output distribution, with a temperature coefficient controlling how smooth that distribution is. The following illustrative pseudocode shows the core of the loss function:

    1. # Use KL divergence to measure the difference between student and teacher output distributions
    2. def distillation_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):
    3. # T: temperature coefficient, amplifies the information carried by soft labels
    4. soft_teacher = softmax(teacher_logits / T)
    5. soft_student = log_softmax(student_logits / T)
    6. kd = kl_divergence(soft_teacher, soft_student) * (T ** 2)
    7. # Continue learning from the ground-truth labels at the same time
    8. ce = cross_entropy(student_logits, labels)
    9. return alpha * kd + (1 - alpha) * ce

    This logic reveals what distillation is at its core: a knowledge transfer channel that operates at the output level. A student model can efficiently inherit the teacher’s tendency to produce certain outputs. What travels through this channel, however, is a “tendency in the results.” It is not the full process by which the teacher model internally learned that tendency.

    What Distillation Does Well: High-Frequency Surface Features

    In language models, some features are extremely easy to learn and imitate. We can call them “high-frequency surface features.” They include specific interjections, sentence structures, word preferences, and punctuation habits, along with certain set openings and catchphrases.

    These features are easy to pick up because they appear frequently in text, follow stable patterns, and are only weakly tied to specific meanings. Models naturally tend to capture statistically prominent signals first when they learn. As a result, if training or distillation data contains features like these, a student model can quickly take on a similar writing style.

    This creates a key cognitive trap: humans are very good at recognizing stylistic similarities. When a model produces a familiar “tone,” it is easy to jump to the intuitive conclusion that it “must have been copied.” Such intuition, however, mistakes the ease of surface-level imitation for equivalence of overall capability.

    What Distillation Struggles to Transfer: Underlying Reasoning and Distributions

    What really sets a language model’s ceiling is not whether it can imitate a certain tone. It is the underlying capabilities it acquires during pretraining, including:

    • Modeling the distributions of language and the world: through massive corpora, a base model learns the statistical patterns of language and how they connect to knowledge about the world. This “distribution” is the foundation of the model’s capability. It is shaped by huge amounts of data, computing power, and a carefully designed training pipeline working together.
    • Logical reasoning and long-range dependencies: capabilities such as multi-step reasoning, mathematical derivation, and code generation rely on complex associations built into the model’s deeper layers. They are hard to acquire merely by imitating output distributions. A student model may remember that “this kind of question is usually answered this way,” yet struggle to reproduce the internal mechanism behind “why it is answered this way.”
    • Generalization and robustness: when faced with inputs beyond the training distribution, a strong model can still generalize reasonably. This ability is rooted in the quality of the base model’s representations, and distillation cannot transfer it directly.

    In other words, distillation is like a high-quality reference answer. It helps a student model quickly align on certain outputs, but it cannot take the place of laying one’s own foundation and building one’s own structure. Through distillation alone, a model can learn what it looks like to be strong, but not why a model is strong.

    Why Engineering Capability Is the Real Dividing Line

    Zoom out to the whole industry, and a curious pattern emerges. Some teams are not short of computing power, and in theory they can carry out knowledge transfer in various forms. Yet they still struggle to build competitive models. The reason often has little to do with resources and more to do with engineering capability.

    Large models are systems built on a solid engineering foundation, including but not limited to:

    • Data engineering: collecting, cleaning, and deduplicating text, balancing data mixtures, and evaluating data quality. This work directly shapes the health of the base distribution.
    • Training engineering: the stability of distributed training, mixed-precision strategies, parallel partitioning, fault recovery, and optimizing training efficiency.
    • Alignment and evaluation engineering: instruction fine-tuning, alignment with human preferences, and building reliable evaluation systems. These help teams identify problems and iterate on improvements.

    These stages depend on one another, and a weakness in any one of them can limit the final results. They also explain why competition in model development is concentrated among a relatively small group of players. The barrier is not whether you can distill, but whether you can get an entire engineering system up and running, and running well. Engineering capability is a hard constraint and the real source of long-term competitiveness.

    Responding to the “Distillation Argument” Rationally

    Let’s return to the original claim: capability “comes entirely from distillation,” so limiting the source will cause progress to stall. From a technical standpoint, this inference has two fundamental flaws:

    1. Misattribution: it treats transferable surface features as equivalent to the underlying capabilities and engineering systems that determine a model’s ceiling. Surface features are easy to imitate, but the foundation is hard to move.
    2. A logical leap: it assumes that capability growth is a one-way dependence. In fact, modeling underlying distributions, reasoning capability, and engineering iteration all grow from within. They cannot be copied or destroyed by cutting off an external output channel.

    A more accurate conclusion, and one that better reflects engineering reality, is this. A model’s continued progress comes from researchers constantly refining their data, training, and alignment systems under resource constraints. It also comes from the long-term accumulation of engineering capability. That capability is homegrown and can be iterated on. It does not stop just because the external environment changes.

    Oversimplifying “distillation” boils a complex systems engineering problem down to a catchy slogan that does not hold up to scrutiny. Recognizing this is the first step toward making rational technical judgments, and the most important one.