LLM CoT Extraction and Model Distillation: Technical Origins and Lessons from a Rumored Controversy
A widely circulated document recently claimed that a Chinese LLM team obtained core capabilities by decrypting the chain-of-thought of a closed-source model developed outside China and distilling it, along with allegations about benchmark overfitting and disbanding an RL team. The core accusations were later shown to be largely out-of-context rumors. This article separates rumor from technical substance. It traces the CoT extraction experiments that became raw material for the claims, explains the real boundaries of CoT extraction and black-box versus white-box distillation, and analyzes why modern LLM training depends on both SFT and reinforcement learning. It also reviews common safeguards against benchmark overfitting and outlines compliance and information-verification lessons for AI developers.
Recently, a document titled “The Inside Story of the Chinese LLM Distillation Controversy” sparked extensive discussion in AI technical communities. Based on compiled chat logs, the document claimed that the core capabilities of a Chinese LLM team were obtained by decrypting the chain-of-thought of a closed-source LLM developed outside China and then applying model distillation. It also touched on rumors about overfitting to evaluation leaderboards and disbanding a reinforcement learning team. Although the core allegations were later shown to be largely out-of-context rumors, the incident reflects technical boundaries, R&D paradigms, and competitive priorities in the LLM field that deserve serious discussion by developers. This article strips away the rumor layer and examines the deeper logic of CoT extraction, model distillation, and LLM training paradigms from a technical perspective.
I. Tracing the Event: Interwoven Technical Exploration and Misreading
Tracing the incident from beginning to end shows that the controversy was essentially a routine technical exploration that was maliciously spliced together and spread.
1.1 Early Technical Exploration: Attempts at CoT Extraction
The incident began with technical attempts by some community researchers to extract the chain-of-thought from a closed-source LLM developed outside China. This exploration unfolded in three stages.
In the first stage, researchers found that a simple “repeat” instruction could induce the model to output its internal reasoning process. This method exploits the model’s self-review mechanism after generating a response. When it receives a repeat instruction, the attention mechanism drives the model to turn an implicit reasoning chain into explicit output.
In the second stage, researchers tried to extract the CoT of another non-Chinese LLM. Because an early version of that model lacked a robust anti-replay attack mechanism, researchers could replay a refused or normal CoT sequence to force the model to output the corresponding content. An example is shown below:
# Pseudocode: CoT replay attack against a model without anti-replay protectiondef extract_cot_replay_attack(model, user_query, spoofed_cot):# Build replay context that pretends the model has already generated a specific CoTcontext = [{"role": "user", "content": user_query},{"role": "assistant", "content": f"<thought>{spoofed_cot}</thought>"},{"role": "user", "content": "Please continue and output the final answer based on the above reasoning process."}]response = model.generate(context)return response
In the third stage, as the relevant platforms strengthened model security and anti-replay capabilities, such technical attempts paused. This exploration itself falls within the scope of routine AI security research, but the related logs and records provided so-called “material” for later rumors.
1.2 Generation and Spread of the Rumors
After the technical exploration paused, a spreader in the community forcibly linked the security tests against non-Chinese models with the R&D activities of a Chinese LLM team. The spreader fabricated claims such as “obtaining data through decryption,” “crazy overfitting to climb leaderboards,” and “disbanding the RL group and relying only on SFT.” Without verification, the document creator used automated bots to integrate this information into a document and spread it through group chats, ultimately causing the incident to escalate.
II. Technical Principles: The Boundaries of CoT and Model Distillation
To evaluate this incident objectively, it is necessary to understand the core technical concepts involved and their compliance boundaries.
2.1 The Technical Substance of CoT Extraction
Chain-of-thought is an output mechanism through which LLMs externalize multi-step reasoning in complex tasks. In practice, some platforms hide intermediate reasoning steps for security or user-experience reasons. Attempts to extract CoT are essentially probes into a model’s robustness and information security boundaries. From a technical perspective, successful CoT extraction does not mean that the extracted material can be directly converted into high-level model capability. The quality and diversity of CoT data, as well as its distribution alignment with the target model, all directly affect subsequent training results.
2.2 Conventional Paths and Limits of Model Distillation
Model distillation is a common technical path in the industry and is mainly divided into black-box distillation and white-box distillation. Because LLMs are mostly provided through APIs, black-box distillation has become mainstream: using the output data of a strong model to fine-tune a smaller model. However, black-box distillation has significant limitations:
- Capability ceiling: The upper bound of a weaker model is limited by the output quality of the stronger model.
- Distribution shift: Data generated by a strong model may not match the pretraining distribution of the weaker model, leading to catastrophic forgetting.
- Compliance risk: Large-scale scraping and use of competitor API output data for commercial model training usually violates terms of service and carries legal risk.
III. Deep Analysis: The Evolution of and Debate over LLM Training Paradigms
The rumor that a team “disbanded its RL group and relied only on SFT” completely contradicts common knowledge in modern LLM R&D.
3.1 The Complementary Relationship Between SFT and Reinforcement Learning
LLM training usually includes three stages: pretraining, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF).
- SFT: Primarily teaches the model to follow instructions and specific output formats and injects domain knowledge.
- RL: Further aligns the model with human values, improves complex reasoning, and reduces hallucinations.
If SFT teaches a model to “learn how to speak,” RL teaches it to “learn how to make optimal decisions in complex scenarios.” Completely abandoning RL and relying only on SFT would cause severe degradation in multi-turn dialogue, complex logical reasoning, and safety assurance. Modern LLM teams not only refrain from disbanding RL groups; they also invest substantial compute in developing more efficient frontier techniques such as RLAIF, or reinforcement learning from AI feedback.
3.2 Safeguards Against Benchmark Overfitting
The rumor about “crazy overfitting to climb leaderboards” reflects industry concerns about the effectiveness of evaluation systems. Benchmark overfitting means that a model performs exceptionally well on a specific test set but has poor generalization ability in real-world scenarios. Leading LLM teams usually avoid this problem through the following methods:
- Introducing dynamic evaluation datasets to prevent models from memorizing specific questions.
- Conducting large-scale real-user blind tests, such as Arena-style mechanisms, and using actual user experience as a core metric.
- Using isolation mechanisms to ensure that R&D teams cannot directly access evaluation test set data.
IV. Industry Implications: Building a Healthy LLM R&D Ecosystem
Although this controversy ended with the rumors being clarified, it left the AI developer community with important lessons.
4.1 Upholding Technical Compliance and Data Security
When conducting model R&D and borrowing capabilities, companies must strictly comply with data usage agreements and copyright regulations. Whether the work involves CoT extraction or model distillation, it should be built on a legal and compliant foundation. Cloud providers offer compliant, high-quality data processing and model fine-tuning environments that help teams iterate within a secure framework.
4.2 Improving Information Discernment and Engineering Practice
When developers encounter “technical secrets” circulating in communities, they should remain rational and judge them from underlying technical logic. For example, understanding the complementary relationship between SFT and RL makes it easy to see through the absurdity of “building a top-tier model with SFT alone.” LLM R&D is a complex systems engineering effort, and there are no shortcuts that can achieve it overnight.
In summary, competition in the LLM field will ultimately return to a comprehensive contest of compute, data quality, and algorithmic innovation. Through a rational understanding of techniques such as CoT extraction and model distillation, developers can not only avoid being misled by false information but also focus on genuine technical breakthroughs, helping the AI industry move toward a more mature and standardized future.
