AI Model Distillation: How to Evaluate the Claim That China’s AI Depends on It Entirely
Knowledge distillation can transfer useful capabilities between AI models, but similar outputs do not prove that one model owes all of its capabilities to another. This article explains what distillation can and cannot transfer, how to assess claims of model copying, and why access restrictions can slow some development paths without necessarily bringing an AI industry to a halt.
When someone argues that “China’s AI relies entirely on distillation, so slowing US AI development would bring Chinese AI progress to a halt,” the strongest response is not to insist that no distillation took place. It is to separate several questions the claim bundles together: Where did the training data come from? What did the model learn from it? What engineering work made those capabilities possible? And how much does the broader AI ecosystem depend on access to an external model?
The claim is easy to repeat because it compresses a complicated process into a single term. Training a model, governing its data, optimizing inference, evaluating capabilities, and deploying it in products all take specialized work. “Distillation,” by contrast, suggests that one model can cheaply copy everything another model does. That intuition conflates different training methods and overstates what imitation alone can achieve.
1. Define what “distillation” means
In machine learning, knowledge distillation is a training method in which a student model learns from a more capable teacher model. The teacher can provide soft labels, probability distributions, or example outputs. The goal is often to transfer some of the teacher’s capabilities to a model that costs less to run.
In discussions about large language models, “distillation” can refer to different training methods and different scopes of transfer:
- Output-based imitation. The student learns from answers, classifications, code examples, or worked solutions generated by the teacher. It can pick up the teacher’s writing habits and performance on common tasks, but what it learns depends on data quality, task coverage, and its own architecture.
- Soft-label learning. Instead of seeing only the final answer, the student learns how the teacher distributes probability across candidate answers. Those relative probabilities provide more information than a single target answer, but they require access to richer training signals than collected text alone provides.
- Task-specific transfer. A teacher can generate training examples that help a student become useful at summarization, classification, information extraction, or format conversion. Strong performance on one of these tasks does not establish that the student has replicated the teacher as a whole.
- Broad capability imitation. Trying to match a teacher across math, coding, long-context understanding, and tool use requires large, diverse, high-quality datasets and a stable training process. It is substantially harder than reproducing a recognizable writing style.
“Distillation was used” and “all capabilities came from distillation” are different claims. The first describes a training choice. The second attributes the model’s performance to that choice and requires much stronger evidence.
2. Similar style is not evidence of identical capabilities
Large language models readily learn recurring patterns. Stock phrases, heading conventions, paragraph structure, refusal templates, and tone are easy to observe and imitate. If many training examples share the same format, a model may learn to:
- Lead with the conclusion, then list the reasons.
- Use familiar headings or add routine disclaimers.
- Explain code in a recognizable style.
- End answers by restating the main point.
These similarities show overlap in observable output patterns, but they do not establish where the patterns came from or whether the models share the same underlying capabilities. Readers are good at recognizing familiar phrasing and structure. Identifying a style, however, is not the same as identifying the source of a capability.
It helps to distinguish three layers of model behavior:
- Surface expression: Vocabulary, tone, formatting, and style.
- Task strategy: How the model breaks down a problem, selects steps, and organizes information.
- Underlying capability: Abstract reasoning, modeling factual relationships, working with executable code, handling long-range dependencies, and generalizing to unfamiliar inputs.
Surface expression is the easiest to imitate. Learning task strategies takes more training. Underlying capabilities depend heavily on the base model, data quality, optimization methods, and engineering implementation. A familiar answer format, by itself, tells you little about where those capabilities came from.
3. Why output distillation alone is unlikely to reproduce every capability
A student can learn to give the same correct answer as its teacher without learning a method that still works when the problem changes. Four limitations explain why.
Outputs expose only part of the teacher’s computation
A teacher model’s final text reflects its internal computation but does not expose all of it. Internal states, attention patterns, learned representations, and intermediate computations are generally not visible to someone who sees only the output. A student may learn how a familiar question is usually answered without being able to derive the answer when the assumptions change.
Training examples cover a limited range of tasks
A finite set of teacher-generated examples can cover only so much of the task distribution. A student trained mainly on common questions may score well on familiar benchmarks yet struggle with rare combinations, counterfactual questions, cross-domain problems, or multiple interacting constraints. A handful of matching demonstrations is therefore weak evidence of a complete capability transfer.
Teacher errors can carry over
Teachers make factual errors, skip logical steps, favor particular formats, and sometimes give inconsistent answers. A student can learn those defects along with useful behavior. Repeated rounds of generation and retraining can compound errors and degrade performance. Distillation is therefore not a lossless copy: the generated data still needs filtering, verification, deduplication, and correction.
The student’s base model constrains what it can learn
The same teacher-generated dataset can produce very different results in different base models. Parameter count, architecture, vocabulary, training stability, data mix, optimizer configuration, and parallel training infrastructure all affect the outcome. A student with insufficient representational capacity may pick up surface patterns from high-quality examples without reliably acquiring the more complex skills those examples demonstrate.
4. Model competitiveness depends on the full engineering pipeline
Access to a capable teacher can help, but it does not replace the work needed to train, serve, evaluate, and deploy a model.
- Data engineering. Teams collect, clean, deduplicate, and assess data; manage copyright and compliance; balance coverage across domains; and check for contamination. How high-quality data is selected and organized can matter more than raw volume.
- Pretraining. Large-scale training requires distributed parallelism, GPU memory management, efficient communication, stable optimization, checkpoint recovery, and fault tolerance. Buying compute does not solve these problems on its own.
- Post-training and alignment. Supervised fine-tuning, preference optimization, task-specific training, safety policies, and tool-use training all affect whether a model follows instructions, respects output formats, and completes complex tasks reliably.
- Inference optimization. Production systems must balance latency, memory usage, concurrency, cost, context length, and recovery from failures. Quantization, caching, batching, routing, and serving orchestration all contribute to how usable a model is in practice.
- Evaluation and iteration. No single leaderboard captures real-world performance. Internal evaluations can cover general and specialist tasks, long-context work, coding, factual accuracy, reliability, and safety. Feedback from production helps teams decide what to fix next.
- Product integration. Deploying AI in search, office software, customer support, content moderation, research and development, or data analysis requires access controls, knowledge bases, logging, tool integration, and well-designed human workflows. Those systems and skills do not disappear if access to an external model is restricted.
Model competition spans research, data governance, systems engineering, and practical deployment. Distillation can contribute to that work, but it cannot substitute for the entire pipeline.
5. How to assess distillation claims with evidence
Make the claim specific
What does “distillation” mean in this case? Does it mean using teacher-generated training data, reproducing an output style, or extracting model capabilities without authorization? These are different claims and call for different evidence. A claim that does not distinguish among them is hard to test.
Separate the training data from the resulting capability
Even if teacher-generated data was used, ask what share of the training data it represented, which tasks it covered, and whether it was manually screened or independently verified. Then ask how the student performs on new tasks for which it had no teacher demonstrations. Evidence that a dataset was used does not, by itself, show how much of the final model’s performance it explains.
Test generalization beyond familiar examples
Evaluate the model on unfamiliar problem types, different phrasings, more complex constraints, and combinations of tasks from different domains. If two models behave similarly on common questions but one consistently fails when the problem changes, the similarity is more consistent with narrow imitation than with broad capability transfer.
Compare error patterns, not just correct answers
Two models arriving at the same correct answer does not establish copying. Different errors do not rule out any influence, either. More informative signals include highly correlated errors, recurring similarities in visible solution steps, and unusually close agreement on rare inputs. Check whether those patterns persist across different prompts and sampling temperatures. Such comparisons can strengthen or weaken an attribution claim, but they do not independently establish how a model was trained.
Look at whether the team can keep improving the model
A team with its own engineering capabilities should be able to improve its data, training process, serving infrastructure, and product experience over time. If progress stalls across the board when access to a teacher changes, dependence is a reasonable question. If the team continues to improve through internally collected data, synthetic data, human feedback, and lessons from deployed applications, an explanation based entirely on external distillation is less convincing.
6. Restrictions can slow some development paths without stopping AI progress
Restrictions on advanced models, compute, or API access can raise development costs or make it harder to obtain high-quality demonstrations in the short term. But those effects do not imply that an entire AI industry will stop progressing.
- Teams can continue training base models on public and proprietary data.
- Researchers can improve architectures, optimization methods, and training efficiency.
- Companies can collect valuable data from real business workflows.
- Smaller, specialized, and on-device models can continue to develop independently.
- Systems engineering can improve practical performance on the same hardware.
- Multi-model workflows, retrieval augmentation, and tool use can compensate for some limitations of individual models.
The key point is that AI development has more than one path. Restricting one source of training signals does not remove the gains available from better data selection, more efficient training, architectures better suited to target tasks, or feedback from real deployments.
Conclusion: Turn the distillation debate into a testable question
Knowledge distillation can lower the cost of transferring some capabilities. It is particularly well suited to reproducing writing styles and common task patterns. But it does not automatically replace base-model development, nor does it guarantee that complex reasoning, generalization, or the capacity for continued improvement will carry over.
The strongest response to the “all distillation” claim is not to try to prove that no distillation occurred. It is to ask three questions: What kind of distillation is being claimed? Which capabilities did it transfer? What other data, algorithms, and engineering systems contributed to the result?
Once those questions are defined and tested, the limits of the claim become clearer. Access to another model’s outputs can matter, but sustained progress also depends on the ability to build datasets, train models, optimize systems, and make them work in real applications.
