Knowledge Distillation in Practice: How a 9B Model Took the Trending Lead Through a Two-Stage Rebuild
A ~9B open-source model reached the top of a trending models list through two distinct engineering stages. Stage 1 is knowledge distillation: a trillion-scale model generated roughly 250,000 math and science samples containing full reasoning traces, which were used to fine-tune a 9B base model and then align it with lightweight reinforcement learning. The result was more correct answers and about 36.1% fewer reasoning tokens per correct answer. Stage 2 adjusts refusal behavior directly in the weights using an internal-activation-analysis method. This walkthrough covers the data generation flow, the SFT and RL steps, the activation-steering procedure, and a checklist for customizing small models. It also explains how to align evaluation scope across stages and shows which metrics the second version still needs to re-test, along with the infrastructure such a pipeline typically requires.
1. The phenomenon: a 9B model tops a trending list
Recently, an open-source model with around 9 billion parameters topped the trending models list on an open-source model community. It sits somewhere between a general-purpose large model and a small base model. It wasn’t trained from scratch, though. It gradually took shape through two rounds of modification:
- Community developer A distilled knowledge from a model with total parameters in the trillions into this small model.
- Developer B then further adjusted the model’s refusal behavior on top of that.
The case is interesting not for its ranking, but for what it shows. It breaks “how to compress a general-purpose large model’s reasoning ability into a small model at a controllable cost” into two engineering stages with clear boundaries. Each stage has its own results that can be measured independently. Below is a walkthrough organized around that technical thread.
2. The big picture: two rounds of changes, two separate scorecards
To understand this model, the key is to recognize that it went through two rounds of changes, and each produced its own independent evaluation results:
- Stage 1 (distillation + lightweight alignment), by developer A: produced a science-focused small model capable of solving problems. Two metrics were already measured: problem-solving accuracy and reasoning cost per problem.
- Stage 2 (refusal behavior adjustment), by developer B: built on A’s weights and used a method based on internal activation analysis to change the model’s refusal tendency. What was measured was the change in how often the model refused.
Why stress “two separate scorecards”? Because reproducers often mix them up. The version from Stage 1 was evaluated on problem-solving ability. The derivative that topped the list was evaluated on refusal behavior. Science accuracy and reasoning cost still need to be re-measured on the second version using the same problem set. They shouldn’t just reuse the first version’s numbers. We’ll return to this in the evaluation methodology section.
3. Stage 1: knowledge distillation from a large model to a small one
3.1 Generating problem-solving samples with a trillion-scale model
The first step of distillation is data generation. A model with total parameters in the trillions, placed in a high-compute, high-quality generation mode (similar to the maximum-quality modes many providers offer), is used to batch-generate training data.
In this case, roughly 250,000 math and science samples were generated. These samples contain far more than a question and a final answer. They also record the complete reasoning trace: where to start, how to break the problem down, how each step is derived, and what conclusion is reached. It’s this process-visible data that lets the small model learn the large model’s reasoning paths instead of just memorizing answers.
A few engineering points are worth noting:
- Keep the data type narrow and focused. Both stages in this case focused only on math and science reasoning. No coding data was used at all. A narrow vertical focus keeps the limited capacity of a small model concentrated on the target ability.
- Generation quality sets the ceiling. Using a costly, maximum-quality generation setup essentially trades compute for data quality. That investment is worth making on the key data batches.
The data generation flow can be sketched out in pseudocode:
# Pseudocode: batch-generate samples with reasoning traces from a large modelsamples = []for problem in math_science_problems:response = large_model.generate(prompt=problem,mode="max_effect", # high-quality generation configurationrequire_reasoning=True, # keep the full derivation process)samples.append({"question": problem, "reasoning": response})dataset = build_sft_dataset(samples) # ~250k samples
3.2 Supervised fine-tuning: teaching the small model how to solve problems
With this data in hand, developer A started from an open-source 9B base model and ran supervised fine-tuning (SFT). SFT maximizes the likelihood that the model’s output matches the target reasoning traces, teaching the small model to imitate how the large model solves problems.
At its core, this step is capability transfer plus parameter compression. The large model’s knowledge travels through data as the intermediate medium and gets “compressed” into a model with roughly two orders of magnitude fewer parameters. Compared with pretraining from scratch, distillation significantly cuts the compute and data needed to reach the same level of science reasoning ability.
3.3 Lightweight reinforcement learning: an alignment pass after SFT
After SFT, developer A added a round of lightweight reinforcement learning. This stage uses a family of sequence-level, within-group policy optimization algorithms. This type of lightweight RL alignment method is common in the industry and works roughly like this: sample multiple responses to the same prompt, then update the policy using each group’s relative advantages. Compared with traditional RL, it uses less GPU memory and is easier to reproduce in low-resource environments. That’s why it’s called “lightweight.”
One thing should be made explicit: like SFT, this RL round also focused only on math and science reasoning, with no coding data introduced. The core gains reported in the final model card are:
On science tests, the distilled small model solved more problems correctly, while the reasoning tokens consumed per correct answer dropped by about 36.1%.
This “higher accuracy, lower cost per correct answer” double win is the most persuasive argument for distillation-based approaches. It doesn’t just make the small model smarter; it also makes it more efficient when it answers correctly.
4. Stage 2: deliberately adjusting the model’s refusal behavior
Developer B didn’t redo the distillation. Instead, they reused the distilled weights that A had published. They then used a class of behavior adjustment methods based on internal activation analysis (commonly referred to in the industry as activation steering or representation steering) to change the model’s refusal tendency.
The approach works like this:
- Compare activations. Have the model process prompts of different types, then compare the activation distributions across its internal layers to find the directions most strongly associated with refusal.
- Locate the relevant layers. Identify which layers have weights strongly correlated with this refusal signal.
- Modify those layer weights. Make targeted changes to those layers that weaken the refusal signal’s influence on the final output, reducing how often the model refuses unnecessarily.
The key point is that this method doesn’t “teach the model not to refuse” by retraining it on new data. Instead, it directly intervenes in the model’s internal representation space, which makes it a low-cost, weight-level editing technique that needs no large labeled datasets. The trade-off is that such changes usually need supporting evaluation to confirm two things: refusals actually decreased, and general ability wasn’t noticeably hurt.
5. Evaluation methodology: why you should re-test with the same problem set
This is the most reusable part of the case for developers. The two rounds each measured part of the metrics. When reproducing the work or choosing a model, follow two rules:
- Align the evaluation scope. Science accuracy and reasoning token cost are Stage 1 results. The change in refusal count is a Stage 2 result. Before citing numbers across stages, re-run them with the same problem set and the same scoring scripts. Otherwise, the numbers aren’t comparable.
- Evaluate both capability and behavior. Any adjustment aimed at refusal should track two things at once: gains on the target capability and regressions in non-target capabilities. That way, fewer refusals don’t come at the cost of answer quality.
| Evaluation dimension | Stage | Metric | Re-test needed? |
|---|---|---|---|
| Problem-solving ability | Stage 1 (distillation + RL) | Science accuracy | Yes, for v2 |
| Reasoning cost | Stage 1 (distillation + RL) | Tokens consumed per correct answer | Yes, for v2 |
| Refusal behavior | Stage 2 (activation-based adjustment) | Change in refusal count | No; a comparison exists |
6. Reusable engineering practices
Generalizing this case yields a practical checklist for customizing and aligning small models:
- Go vertical first, then go broad. Use high-quality distillation data from a single domain, such as math and science reasoning, to get the small model into usable shape. Only then consider expanding its capabilities. This avoids diluting its capacity.
- Process matters more than results. Distillation data should keep the full reasoning traces so the model learns how to think, not just what to answer.
- Combine SFT with lightweight RL. Use SFT as the foundation and lightweight RL for preference alignment. It’s a cost-effective combination in low-resource environments.
- Try weight-level methods first for behavior adjustments. Methods like the activation-analysis approach above need no retraining, so they’re a good low-cost first choice for adjusting surface-level behavior such as refusal.
- Put evaluation first, with consistent scope. Every change needs a reproducible evaluation loop with measurements you can compare against a baseline. This matters most when citing metrics across stages.
Putting this kind of workflow into engineering practice typically takes supporting infrastructure:
- A model training/fine-tuning platform to host SFT and RL jobs.
- Object storage to hold large training datasets and weight versions.
- A logging service to record the inputs and outputs of each evaluation.
- Monitoring and alerting to track fluctuations in accuracy and reasoning cost.
Baidu AI Cloud and similar platforms offer general-purpose capability categories along these lines, including training platforms, object storage, logging services, and monitoring and alerting. Together, they can cover the full chain from data generation through training and alignment to evaluation tracking. That lets reproducers focus their energy on the methods and data themselves.
7. Conclusion
A 9B model can reach the top of a trending list not because of a single clever trick, but because of a clear two-stage pipeline: large-model distillation → supervised fine-tuning → lightweight reinforcement learning → targeted behavior adjustment.
The main takeaway is this: when compute and data are limited, efficiently compressing a large model’s capabilities into a small model comes down to three things:
- Constructing high-quality reasoning data.
- Training with a narrow vertical focus.
- Following a consistent, reproducible evaluation discipline.
Once you understand this chain, you can reproduce custom small models in your own scenarios at a controllable cost, with models that are stronger, more efficient, and more aligned.
