LLM Distillation: How to Rationally Evaluate Model Progress

    This article examines the claim that overseas "distillation crackdowns" caused leading Chinese LLMs to collectively stall, and explains why that narrative does not hold up to scrutiny. It clarifies what knowledge distillation can and cannot copy, why API-side protections raise the cost of data extraction rather than cutting off access entirely, and why release cadence and community word-of-mouth are poor substitutes for measurable benchmarks. It then lays out a practical framework for judging real model progress—capability, efficiency, stability, and business metrics; self-built evaluation sets with bootstrap confidence intervals; LLM-as-a-judge calibration; ROI calculation; and gray release with rollback—so developers and teams can turn the "who copied whom" debate into reproducible engineering decisions.

    Intro: Can a “Timeline” Prove “Collective Stalling”?

    Recently, a tech blogger put forward a chain of reasoning:

    • February this year: An overseas model provider disclosed that it had found someone using roughly 24,000 violating accounts to make more than 16 million cumulative calls, apparently aiming to bulk-“extract” the reasoning and coding abilities of its model.
    • October this year: The same provider announced upgraded protections that reduced the reasoning content that can be extracted by about 90%~99%.
    • Afterward: In the blogger’s view, the flagship update cadence of leading Chinese models slowed. Small incremental steps were dismissed as “not a big breakthrough,” and feedback on preview releases was “somewhat flat.”

    From this, the conclusion followed: after losing the Claude-style “standard answers,” Chinese models “collectively stalled,” and their pivot to other directions also failed.

    More telling is the blogger’s own contradictory wording. Just two months earlier, the same person publicly argued against simply attributing Chinese model progress to “distilling overseas models,” and praised a particular version of a Chinese model as “the best open model in the world.” One month later, that same model was cited as a textbook case of “no breakthrough.”

    The lesson: in community discourse, narrative coherence is not the same as evidence reliability. Rather than argue over conclusions, it is more productive to break the proposition apart and return to technical and engineering facts layer by layer:

    1. What is “distillation” technically? What can it copy, and what can’t it?
    2. Does protection on the API side actually “close the door”?
    3. What kind of metrics let you tell whether a model is “standing still” or “steadily taking small, fast steps”?

    1. Cutting Through the Noise: What Can Model Distillation Actually Distill?

    1.1 Distillation: A Mature General-Purpose Technique

    Knowledge Distillation was systematically introduced by Hinton et al. in 2015, and has long been a classic method for model compression and knowledge transfer. Illustrative code:

    1. import torch
    2. import torch.nn.functional as F
    3. def distill_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):
    4. """
    5. Soft-target distillation loss = alpha * KL(student || teacher) + (1 - alpha) * CE(student, labels)
    6. T: temperature, amplifies the relative differences among logits to expose the teacher's knowledge distribution
    7. alpha: weight between soft and hard targets, commonly 0.5~0.9
    8. """
    9. s = F.log_softmax(student_logits / T, dim=-1)
    10. t = F.softmax(teacher_logits / T, dim=-1)
    11. # T^2 compensates for the change in gradient magnitude caused by temperature scaling
    12. loss_soft = F.kl_div(s, t, reduction="batchmean") * (T * T)
    13. loss_hard = F.cross_entropy(student_logits, labels)
    14. return alpha * loss_soft + (1 - alpha) * loss_hard

    Distillation falls into two broad categories:

    • Soft-label distillation: The student model learns the teacher’s full probability distribution—“soft labels.” The teacher assigns a probability to every candidate, encoding information such as “the teacher considers a second-best answer reasonable too.” What transfers here is “dark knowledge.”
    • Sequence-level distillation: The teacher’s generated content is used directly as training data. The typical use case is turning answers—or answers with chains of thought—produced by a strong model into supervised fine-tuning data. For APIs that only return text, this second form is what the industry actually debates.

    1.2 Two Concept Slips in the Argument

    First slip: swapping “distillation” for “plagiarism.” In its standard definition, distillation involves a student model and a teacher model, with the goal of compressing and transferring capability into a smaller model. Companies often use their own in-house teacher, or data they are licensed to use. That is fundamentally different from “using large numbers of accounts to bulk-extract a service bound by terms,” which is what actually qualifies as a “distillation attack.” Whether something is legitimate depends on data licensing and terms of service—not on whether the word “distillation” is used.

    Second slip: equating “distillation data” with “R&D capability.” The AI industry relies heavily on open source and public research, and this is a widely accepted norm. When evaluating a model team, the right things to look at are its systems-engineering ability and differentiated innovation, for example:

    • Data mixing ratios and cleaning;
    • Architecture design such as mixture-of-experts;
    • Post-training algorithms;
    • Inference optimization.

    Not whether it “ever pulled samples from some particular service.”

    1.3 Three Ceilings of Distillation

    Limitation Concrete manifestation Engineering impact
    Capability ceiling The student’s ceiling is usually constrained by the teacher and the training method Hard to surpass the teacher’s capability tier through distillation alone
    Memory, not understanding The student may memorize templates and phrasing rather than genuinely learn to reason Performance drops noticeably on new problems, new languages, and multi-step reasoning
    Distribution mismatch The style, domain, and refusal boundaries of distillation data come from the teacher Product tone converges, and hallucinations and biases are inherited along with it

    For this reason, no serious model team treats “collecting distillation data” as the core of its technical roadmap. At most, it is a supplementary path for improving data diversity and coverage. Put differently, “cutting off distillation” can weaken the latter—not the former.


    2. What API Protection Does and Doesn’t Prevent

    2.1 Why “Reasoning Traces” Are Worth Protecting

    After reasoning models became widespread, many services return step-by-step thinking content before the final answer. This content may be the native output or a summarized version. Reasoning traces are worth protecting more than final answers, for three reasons:

    1. High information density: They contain how the problem is decomposed, intermediate conclusions, and self-check steps. Training data that gives only the answer does little for reasoning ability; samples with complete steps teach far more effectively. Clearly, the latter is the “high-value training data.”
    2. Easy to generate at scale: Automated scripts can produce large numbers of variant samples for the same problem in a very short time.
    3. Extremely low marginal cost: Before protections were in place, the cost of bulk extraction through calls was far below the cost of organizing labeled data.

    Precisely because of this, when a provider focuses its protections on “trace outputs,” you can see effects like “extractable reasoning content dropping by more than 90%.” Technically, this is a reasonable defense aimed at the economics of data extraction.

    2.2 Protection Methods and the “Cost of Being Copied”

    Mainstream protections mostly focus on “raising the marginal cost of extraction”:

    • Output side: Return only the final answer, or provide reasoning traces only as summaries.
    • API parameters: Lock down parameters such as temperature and sampling, and limit the number of candidates returned per call.
    • Account and call risk control: Detect bulk registrations, proxy pools, and abnormal call rhythms and fingerprints, combined with rate limiting.
    • Content protection: Refuse requests that ask the model to reproduce system prompts, training-data fragments, or large volumes of repeated phrasings.

    The combined effect is to push the acquisition cost per effective sample up several-fold or more, so that low-cost “large-scale copying” largely stops working. But the technical boundary is equally clear:

    • Answers are always obtainable: The provider must output answers, and the problem set can be probed repeatedly. Problems from public leaderboards and public evaluation sets are, from the start, targets that everyone optimizes against together.
    • Many external alternatives exist: Open-weight models can be deployed locally and used to generate training data of any scale. Traditional closed-source models, or other open endpoints, can also serve as teachers.
    • Protection has a lag window: In the early days after a new model launches, the old protections often don’t yet cover its new output formats.

    So the significance of API protection lies in “raising prices,” not in “cutting supply.” Going straight from “protection is in effect” to “the competition has stalled” skips the most necessary link: a comparable curve of measured outcomes.

    2.3 Distillation and Model Alignment

    Distillation also serves legitimate purposes:

    • Transferring a flagship model’s instruction-following, format constraints, and value alignment to on-device or low-cost small models;
    • Compressing domain knowledge into specialized models through an in-house teacher, significantly lowering per-inference cost.

    In the widely accepted industry definition, distillation can be either the starting point of innovation or a tool that is misused—what matters is data licensing and how it is used.


    3. A Narrative Timeline Is Not Evidentiary Proof

    The original argument is structured as: A (cut off distillation) → B (models stall), with a “timeline” as support. From a methodological standpoint, there are at least four holes.

    Hole 1: Substituting temporal order for causation. Strictly speaking, correlation is a necessary but not a sufficient condition for causation. For the argument to hold, “Chinese models collectively stalled” must first become a fact that can be quantified and observed.

    Hole 2: Substituting “version cadence” for “capability gains.” Release cadence is determined jointly by training windows, compute scheduling, safety evaluations, and product strategy. Three months without a flagship update could mean:

    • The capability isn’t yet strong enough to support a convincing upgrade;
    • Polishing hasn’t met the bar;
    • Resources were invested in other directions such as inference-cost optimization, long context, or tool calling.

    None of that is equivalent to “capability hasn’t improved.”

    Hole 3: Substituting a single observer’s “word of mouth” for benchmarks. Feedback like “somewhat flat” is influenced by expectation management and communication style. And the fact that the same observer rated the same model “the best open model in the world” and “not a big breakthrough” within two months shows exactly that their standard drifts with whatever the current frame of reference is.

    Hole 4: Single-cause attribution. The post-training paradigm is shifting from “piling on labeled data” to “large-scale reinforcement learning,” making the relationship between capability and training compute more complex. Compute supply, data supply, architecture changes, and test-time compute strategies are all variables that cannot be ignored. Attributing all change to a single external event is a textbook single-cause fallacy.

    3.1 One Technical Detail Worth Noting

    Even granting that “API protection compressed extractable content,” tying that to “the competition stalled” still requires answering three questions:

    1. Is extraction volume at an inflection point? 16 million calls across about 24,000 accounts is roughly 670 calls per account. Against pre-training and post-training corpora that run into the billions or tens of billions of tokens, how large an increment is that? Has it reached the point where “extract or not decides how strong the next generation is”? There is currently no public data supporting that judgment.
    2. Do the time windows line up? The events happened in February, but model training usually starts earlier. If the bulk of the post-training data was already assembled before the events, then this generation of models was already set in stone by the time the protections took effect.
    3. Is the stalling widespread? “Collective stalling” would need to cover comparable metrics across the major teams; it cannot be generalized from the word of mouth about a single version.

    4. Judging “Standing Still”: Metrics and an Evaluation Process

    4.1 Two Frames: Ranking vs. Generational Gains

    “Best in the world” and “not a big breakthrough” can both be true for the same version:

    • “Best” is a horizontal ranking: Among the candidates at the time, it beat every competitor.
    • “Breakthrough” is a vertical increment: How much it improved relative to the previous generation.

    If a model stays at #1 across several versions, the increment in later versions naturally narrows—this is consistent with the law of diminishing returns in capability growth. The two frames use different measuring sticks, yet both get treated as standards for measuring the same thing. That is exactly the root of why word of mouth tends to “flip.”

    4.2 Four Layers of Evaluation Metrics

    Layer What to look at Notes
    Capability Task success rate and error rate per dimension Use self-built eval sets aligned to your business scenarios; avoid merely “fitting” to public leaderboards
    Efficiency Cost per task, P95 latency, throughput Flagship capability upgrades often come at the cost of “reasoning on by default,” driving up token consumption and latency noticeably
    Stability Variance across repeated calls on the same input, drift over long sessions Tied to predictability in production
    Business Conversion rate, resolution rate, human-takeover rate, rework rate The ultimate “real money”

    A single evaluation only captures a cross-section at one point in time; continuous evaluation is what draws a “trend line.” The recommendation is to run scoring continuously under a consistent methodology and fixed cadence, turning “feels like no progress” into “the metric curve has flattened / is still climbing.”

    4.3 Self-Built Eval Sets: Turning “Feelings” into Reproducible Numbers

    1. import json
    2. import math
    3. import random
    4. # 1) Eval set: at least 50 real samples per task category, covering typical distributions and edge cases
    5. # 2) Weights: assigned by business value; e.g. code-generation scenarios weigh "runnable/mergeable" more heavily
    6. weights = {"correctness": 0.5, "quality": 0.2, "safety": 0.2, "efficiency": 0.1}
    7. def evaluate(model_output, ref_answer):
    8. # Illustrative: correctness, format compliance, safety-policy hits, token consumption
    9. return {
    10. "correctness": 1.0 if model_output.strip() == ref_answer.strip() else 0.0,
    11. "quality": 1.0,
    12. "safety": 1.0,
    13. "efficiency": 1.0,
    14. }
    15. def weighted_score(scores: dict) -> float:
    16. return sum(weights[k] * scores.get(k, 0.0) for k in weights)
    17. def bootstrap_ci(vals, n=1000, alpha=0.05):
    18. """Bootstrap resampling to estimate the 95% confidence interval and check whether version differences are significant"""
    19. means = []
    20. for _ in range(n):
    21. sample = random.choices(vals, k=len(vals))
    22. means.append(sum(sample) / len(sample))
    23. means.sort()
    24. lo = means[int(n * alpha / 2)]
    25. hi = means[int(n * (1 - alpha / 2)) - 1]
    26. return lo, hi
    27. def compare(old, new):
    28. s_old = [weighted_score(evaluate(o, r)) for o, r in old]
    29. s_new = [weighted_score(evaluate(o, r)) for o, r in new]
    30. lo, hi = bootstrap_ci([b - a for a, b in zip(s_old, s_new)])
    31. mean_delta = sum(s_new) / len(s_new) - sum(s_old) / len(s_old)
    32. verdict = "significant improvement" if lo > 0 else ("significant drop" if hi < 0 else "no significant difference")
    33. print(f"Mean score: old={sum(s_old)/len(s_old):.3f} new={sum(s_new)/len(s_new):.3f}")
    34. print(f"Δ = {mean_delta:+.3f}, 95% CI = [{lo:+.3f}, {hi:+.3f}] -> {verdict}")

    4.4 LLM-as-a-Judge: Efficient, but Must Be Calibrated

    Using a strong model for automatic scoring lowers cost but introduces several known biases:

    • Position bias: Scores can change when the order of the same set of answers is swapped.
    • Length preference: A tendency to give longer answers higher scores.
    • Self-preference: More leniency toward outputs similar in style to its own.

    Calibration methods:

    1. Fix the presentation order, or average over both orders after swapping;
    2. Align against human annotations and compute the agreement between the judge and human raters;
    3. Keep human spot-checks on key dimensions.

    Illustrative prompt:

    1. You are a rigorous technical reviewer. Evaluate the model's answer against the following rubric:
    2. [correctness] Did it complete the task correctly and give a runnable/verifiable result?
    3. [quality] Is the structure clear, and does it address all constraints of the question?
    4. [safety] Does it contain any unsafe or non-compliant content?
    5. [efficiency] Is it redundant, and does it complete the task within a reasonable length?
    6. Requirement: give a 1-5 score per dimension with a brief justification; give a conservative score when unsure.

    4.5 Quantifying ROI: Was the Upgrade Worth It?

    1. Cost per task = (input tokens × input unit price + output tokens × output unit price) + retry/human-takeover cost
    2. Net benefit = incremental successful tasks × per-success business value − (cost difference per task × task volume)
    3. ROI = net benefit / (cost difference per task × task volume)

    A simplified example:

    • The new version lifts task success rate from 70% to 76%;
    • The business value of a single success is 50 yuan;
    • Because the new version enables reasoning by default, output tokens and latency rise, pushing cost per task from 0.02 yuan to 0.05 yuan.

    Net benefit per 10,000 calls: 10000 × (6% × 50 − 0.03) = 29,700 yuan, an ROI of roughly 990×. As you can see, a modest capability gain can often cover the cost of being “more expensive and slower.” Without looking at ROI, it’s hard to tell whether an upgrade is a genuine “breakthrough” or just a “price hike.”

    4.6 Gray Release and Rollback: Don’t Let Upgrades Become a “Black-Box Gamble”

    Illustrative config:

    1. {
    2. "model_route": [
    3. {"model": "legacy-chat-v1", "weight": 0.9},
    4. {"model": "chat-v2-preview", "weight": 0.1, "guardrails": {
    5. "max_output_tokens": 2048,
    6. "timeout_ms": 15000,
    7. "retry": 2,
    8. "fallback": "legacy-chat-v1"
    9. }}
    10. ],
    11. "rollout_rule": {
    12. "window": "24h",
    13. "advance_if": {"success_rate_delta": "> +2%", "p95_latency_delta": "< +15%"},
    14. "rollback_if": {"success_rate_delta": "< -2%"}
    15. }
    16. }
    • Split traffic by weight so the new version first takes a small share;
    • Set guardrails: cap output length and timeouts, and automatically fall back to the old version on failure;
    • Auto-promote / auto-rollback: use online metrics such as success rate and P95 latency as the conditions for promotion and rollback.

    5. Engineering Response: What to Change After API Protections Are Upgraded

    Change Potential impact on the business Response
    Reasoning traces no longer returned, or summaries only Missing intermediate steps makes it hard to diagnose where errors come from Keep full input/output logs; require structured output (e.g., JSON fields) on critical paths and validate at the field level
    Parameters locked, candidate count limited Sampling and ensemble strategies break Constrain diversity at the prompt level; add resampling and self-check retries for generation tasks
    Stricter risk control, more frequent throttling Batch jobs hit queues and timeouts Add queues and backoff retries; split peaks; cache results in object storage and use message queues to smooth bursts
    Higher cost per token Per-unit business cost rises Tiered routing: simple tasks to lightweight models, complex tasks to flagship; enable prompt and semantic caching

    Typical backoff-retry code:

    1. import random
    2. import time
    3. def call_with_backoff(request, max_retries=3, base=0.5, cap=8.0):
    4. for attempt in range(max_retries + 1):
    5. resp = request() # Illustrative: replace with your actual model-call logic
    6. if resp.status_code not in (429, 500, 502, 503, 504):
    7. return resp
    8. if attempt == max_retries:
    9. break
    10. # Exponential backoff + jitter: avoid many clients retrying at the same instant
    11. delay = min(cap, base * (2 ** attempt)) * (0.5 + random.random())
    12. time.sleep(delay)
    13. raise RuntimeError("Model service is temporarily unavailable, please try again later")

    6. Architecture and Governance: Turning “Model Selection” from a Guess into a Reproducible Engineering Decision

    1. Isolate differences behind an abstraction layer: Introduce a unified gateway layer between business code and model services, switching model versions, parameters, and guardrails via configuration to avoid tight coupling to any single provider.
    2. Version pinning and observability: Log the model version, key parameters, latency, and token consumption for every call. For content produced from model outputs or interactions, retain and support retroactive retrieval so problems are easy to locate and audits are straightforward.
    3. Build a continuous evaluation pipeline: Chain self-built eval sets, automatic scoring, human spot-checks, and online gray release into one pipeline, with every model upgrade going through the same gate so decisions rest on data rather than hearsay.
    4. Maintain multi-vendor fallbacks: Keep the interface layer abstract, keep at least one backup channel ready, and periodically run baselines on the same eval set to keep switching costs controllable and risks predictable.
    5. Data and compliance governance: Desensitize input data, define data-retention and cross-border-transfer policies, apply content-safety filtering to model outputs, and keep human review in the loop for auto-generated content.
    6. Leverage general cloud capabilities: Cloud providers such as Baidu AI Cloud typically offer general capabilities—model-service onboarding and management, logging, monitoring and alerting, message queues, and object storage—which can support version tracking, gray-release traffic splitting, cost monitoring, and fault alerting, closing the loop of evaluate → gray release → observe → rollback.

    Conclusion: Turn the Argument into a Verifiable Engineering Problem

    The narrative “cutting off distillation caused collective stalling” sounds internally consistent, but every step in it can be tested against technical facts:

    • Distillation is a general-purpose technique with limited capability boundaries;
    • API protection raises the cost of data extraction rather than closing the channel completely;
    • Neither release cadence nor perceived word of mouth can replace quantifiable evaluation and ROI;
    • And the narrator’s own contradictory statements are a reminder to treat “timeline-style” arguments with caution.

    For developers and enterprise users, the more valuable questions are not “who copied whom,” but:

    • Has my task success rate improved?
    • Has my cost per unit gone down?
    • Can the next upgrade be proven with gray-release data?

    When all three of these can be answered with data, the mudslinging over “the model wars” becomes the basis for engineering decisions.