After the Distillation Door Closes: Why Model Iteration Has Slowed

    A popular claim says the closure of external distillation channels has caused leading Chinese models to slow down. This article separates the claim from engineering reality: it explains what knowledge distillation can and cannot transfer, why the industry is shifting toward test-time compute and verifiable reinforcement learning, why slower releases do not necessarily mean stagnation, and how teams should measure model progress with versioned internal benchmarks and fixed evaluation protocols.

    On an overseas social platform, a technology blogger recently advanced a causal chain. According to the post, a frontier lab in February publicly accused several teams of using more than 24,000 unauthorized accounts to make over 16 million calls, in an effort to extract reasoning and coding capabilities. Then, in October, the same lab announced that its new protective mechanisms had reduced “extractable reasoning content” by more than 90%.

    From this, the blogger inferred that the external “standard answer” channel had been closed. As a result, leading Chinese model teams were said to be collectively slowing down, and turning to other closed-source flagship models had not solved the problem either.

    The post also listed several pieces of supposed evidence:

    • A team focused on long-context models released only a minor upgrade about 100 days after its previous flagship model.
    • A team focused on open-source development had not yet delivered a Pro-level flagship.
    • An enterprise-oriented team used to update in less than two months, but had now gone three months.
    • A flagship preview from a cloud platform received relatively muted feedback.

    It is worth noting that, two months earlier, the same blogger had publicly opposed the idea that Chinese model progress should be attributed to distilling foreign models. At that time, one generation of open-source models was even described as “the best open models in the world.” A shift in position within a single month is, by itself, more informative than the conclusion: under traffic-driven incentives, cautious and conditional statements rarely spread, while absolute claims often attract hundreds of thousands of reads.

    As technical readers, however, our real questions are not whether the blogger is credible. We need to answer two harder questions:

    1. How much of model capability actually comes from knowledge distillation?
    2. Even if distillation channels were completely closed, would that alone explain the pace change across the entire industry?

    2. First Align the Concept: What Knowledge Distillation Can and Cannot Transfer

    Knowledge distillation is a completely standard technique in the industry, and it carries no inherent negative meaning. Its typical form is to train a student model to fit the probability distribution produced by a teacher model, rather than only fitting one-hot labels. Soft labels carry the relative relationships between classes, often called “dark knowledge,” so they contain more information than hard labels.

    1. import torch
    2. import torch.nn.functional as F
    3. def distillation_loss(student_logits, teacher_logits, labels,
    4. temperature=2.0, alpha=0.7):
    5. """
    6. Classic soft-label distillation:
    7. L = alpha * KL(student || teacher) + (1 - alpha) * CE(student, hard)
    8. A higher temperature smooths the distribution, making weak signals
    9. easier for the student model to capture.
    10. T**2 compensates for the gradient magnitude reduction after softening.
    11. """
    12. log_p = F.log_softmax(student_logits / temperature, dim=-1)
    13. q = F.softmax(teacher_logits / temperature, dim=-1)
    14. kd = F.kl_div(log_p, q, reduction="batchmean") * (temperature ** 2)
    15. ce = F.cross_entropy(student_logits, labels)
    16. return alpha * kd + (1 - alpha) * ce

    This code explains how distillation works, but it also exposes its ceiling. No term in the optimization objective is directly related to “how the world works.” What it fits is a conditional distribution, p(y|x). Therefore, what distillation can transfer efficiently is:

    • output style;
    • formatting conventions;
    • the shape of common reasoning paths;
    • instruction-following behavior.

    These are the surface layer of a task.

    What distillation cannot conjure from nothing is:

    • the knowledge capacity supported by parameter scale;
    • stability of intermediate states in long chains of reasoning;
    • broad distribution coverage accumulated during training.

    This is similar to copying the solution steps of an experienced engineer. You may produce cleaner-looking solutions and pass most routine problems, but when the task requires looking up information on the spot, sustaining dozens of steps without error, or building your own verification environment to confirm correctness, the copied steps will fail. More importantly, distillation inherits the teacher model’s mistakes along with its strengths. Where the teacher confidently makes an error, the student is likely to make the same error just as confidently—only smaller and harder to detect.

    So the statement “distillation can only imitate surface reasoning paths; foundational capability has to be built through compute and reinforcement environments” is not rhetorical. It is a mathematical consequence of the training objective.

    3. The Real Ceiling: The Paradigm Shift from Pretraining to Test-Time Compute

    Attributing slower releases simply to external blocking misses a more fundamental fact: in the second half of 2026, frontier labs around the world ran into the same wall. The marginal benefit of simply scaling up pretraining data has clearly declined.

    The growth rate of high-quality text data is not keeping up with the growth rate of compute budgets. Repeated training is approaching zero marginal return. Synthetic data carries risks of self-contamination and diversity collapse.

    The industry’s response is not to keep pushing harder in the same dimension. It has shifted toward two other areas.

    Test-time compute

    With the same set of parameters, allowing a model to generate more candidates during inference, perform multiple rounds of self-checking, or use a verifier to select the best answer can yield larger gains than another training round. This creates a direct engineering consequence: the latency and cost structure of a single request changes fundamentally. Vendors must first solve inference cost control, batch scheduling, and cache reuse before they can confidently expose “multi-sampling” as a product capability.

    This kind of work is rarely reflected in public leaderboards, but it can significantly lengthen release cycles.

    Verifiable reinforcement learning in real environments

    This is the hardest wall right now, and it is also the part that distillation cannot replace. The reward signal must come from an executable and verifiable environment, not from reading someone else’s output distribution.

    1. def verifiable_reward(response, sandbox, tests):
    2. """
    3. Sparse reward based on verifiable outcomes: whether code runs and tests pass.
    4. This signal can only be produced by an execution environment, not by distillation.
    5. """
    6. code = extract_code_block(response)
    7. if code is None:
    8. return -1.0 # Format violation: strong penalty
    9. run = sandbox.run(code, timeout=10, limit="1cpu-2g")
    10. if run.error:
    11. return 0.0 # Executable but incorrect result
    12. passed = sum(1 for t in tests if sandbox.run_test(code, t).ok)
    13. return passed / max(len(tests), 1) # Pass rate as a continuous reward

    The cost of this process is extremely high. It requires:

    • thousands of isolated sandboxes;
    • automated task construction pipelines;
    • reliability checks for generated tests;
    • continuous defense against reward hacking.

    Building an environment cluster that can reliably produce high-quality signals is a months-long engineering effort. It is not something that can be solved by one version release.

    Put these two shifts together, and the conclusion becomes clear. Moving from a fast-food-style iteration cycle of “refreshing leaderboards every forty to fifty days” to a long-cycle iteration model based on test-time compute and environment-based reinforcement learning is a natural result of a paradigm shift. It is not evidence of capability regression.

    This pace change is also happening overseas. Flagship release intervals at multiple institutions have lengthened this year. Hangzhou and Beijing are not exceptions; they are points on the same curve.

    4. Slow Is Not the Same as Stagnation

    “No new flagship model released” does not equal “no research progress.” Based on public information and industry practice, the teams mentioned in the original claim are likely working on several types of unglamorous work that does not directly produce leaderboard entries:

    • Inference cost engineering: quantization, speculative decoding, KV cache compression, and batch scheduling optimization. The goal is to reduce unit token cost by an order of magnitude while maintaining similar quality.
    • Agentic capabilities: moving models from “answering questions” to “completing tasks.” This focuses on tool-calling protocols, multi-turn state management, failure recovery, and long-horizon task decomposition.
    • Enterprise workflow integration: permissions, auditability, private knowledge ingestion, observability, and gradual rollout. These capabilities may not show up on general-purpose benchmarks, but they determine whether a system can be deployed in real business environments.
    • Ultra-long context and task orchestration: as context length increases, retrieval strategy, attention cost, and the preservation of intermediate information become harder. This is a typical area with high investment, slow feedback, and low external visibility.

    A simple analogy: a restaurant used to release new dishes every week. Now, the kitchen is quietly brewing stock and rebuilding its supply chain. People standing at the door only count the number of dishes on the menu and conclude that the chef has run out of ideas.

    5. How to Observe Capability Curves Correctly

    If release cadence is not a reliable proxy for capability, what should be used instead? Teams should establish their own observation protocol:

    1. Maintain a versioned internal task set

    Turn real business samples into a held-out evaluation set. It must be strictly excluded from any training or prompt-optimization process to avoid contamination.

    2. Use a fixed evaluation protocol

    Specify whether the metric is pass@1 or pass@k, the sampling temperature, maximum output length, and any voting or verification strategy. The same model can differ by more than ten points under different protocols.

    3. Score capability dimensions separately

    Measure:

    • complex code execution;
    • stability in long-chain reasoning;
    • long-context recall;
    • instruction following and format compliance;
    • cost and latency.

    A single aggregate score is misleading. Dimension-level breakdowns are what provide decision value.

    1. EVAL_SET = "internal_tasks_v3.jsonl" # Versioned, strictly held out, never used for tuning
    2. def evaluate(client, cases, k=5, temperature=0.6):
    3. """
    4. pass@k protocol: sample k times per question; if any sample passes, score 1.
    5. Do not mix with pass@1; always state the protocol in reports.
    6. """
    7. scores = []
    8. for case in cases:
    9. samples = [client.generate(case.prompt, temperature=temperature)
    10. for _ in range(k)]
    11. scores.append(max(scorer(case, s) for s in samples))
    12. return sum(scores) / len(scores)

    4. Prioritize regression over chasing novelty

    Treat every model upgrade as a gradual rollout. First run the internal evaluation set on a small traffic slice. Expand only after metrics show no regression. The benefit of chasing a newer version is usually smaller than the loss caused by a silent capability regression.

    6. The Correct Place of Distillation: A Compression Tool, Not a Source of Capability

    Even if external channels were completely closed, distillation still has an irreplaceable role in a company’s own technical stack. Its purpose has never been “obtaining capability.” Its purpose is to deploy the capability of an existing strong model to larger-scale scenarios at lower cost.

    • Cost tiering: difficult requests go to large models, while high-frequency simple requests go to distilled smaller models, with routing policies controlling overall cost.
    • Edge and private deployment: resource-constrained terminals, offline environments, or internal networks can only be served by compressed lightweight models.
    • Format and domain adaptation: highly deterministic requirements such as output structure, terminology systems, and audit fields are more stable when fixed through distillation or data construction than when repeatedly tuned through prompts.
    • Compliant self-distillation: when data ownership and model rights are clear, using one’s own model to generate synthetic data for further training is currently a relatively safe approach. Bulk harvesting of outputs from unclear sources carries service-terms risk and can also bake external model bias and errors into one’s own weights.

    From an infrastructure perspective, these engineering needs mostly fall into general cloud service categories. Object storage and version management can support training and evaluation datasets. Container platforms can support elastic scaling for sandboxes and inference services. Message queues can support asynchronous tasks and request smoothing. Logging, monitoring, and alerting can support quality regression checks and cost attribution.

    Cloud platforms such as Baidu AI Cloud may provide combinations of these generic building blocks, but any specific service should be verified against the official documentation and, more importantly, against the evaluation protocol described in Section 5. The final decision should be based on reproducible internal testing, not on any single public leaderboard.

    7. Conclusion

    Understanding model evolution as “copying homework from the top student, and failing the exam if copying is not allowed” seriously underestimates modern AI engineering.

    Distillation can transfer solution format. It cannot transfer compute, environment scale, or the closed loop of verification. Release cadence reflects where investment is concentrated during a paradigm shift; it is not evidence of research stagnation.

    As for the loud public emphasis on blocking reasoning chains and anti-distillation rhetoric, the commercial motivation is not hard to understand. It can help protect the moat in enterprise coding scenarios and prepare a sufficiently tense narrative for the next funding round.

    For engineering teams, the useful takeaway is simple: use your own task set, use a fixed evaluation protocol, and evaluate reproducibly. Leave the discussion of release cadence to spectators.