Distilling a 9B Model for Reasoning and Refusal Behavior Optimization
This article examines a representative workflow for enhancing a 9B open-source model: distilling math and science reasoning data from a much larger teacher model, applying supervised fine-tuning and lightweight reinforcement learning, and then selectively adjusting refusal behavior. It explains the reported gains, evaluation caveats, safety trade-offs, and why accuracy, inference cost, refusal rates, and safety boundaries must be measured under a unified test protocol.
1. From a Very Large Model to a 9B Model: A Reasoning Distillation Workflow
A recently released 9B-scale open-source model gained attention in the model community after going through multiple stages of modification. Its significance is not simply its relatively small parameter count, but its representative training path: a very large reasoning model first generated high-quality mathematics and science samples; supervised fine-tuning and lightweight reinforcement learning then transferred part of that problem-solving capability to a 9B model; finally, the model’s refusal behavior was selectively adjusted.
This type of work shows that model capability does not depend entirely on parameter count. With well-constructed data, carefully designed training objectives, and appropriate behavioral evaluation, smaller models can be substantially improved on targeted tasks. However, a distilled model should not be assumed to outperform its teacher across the board. It is better understood as a model optimized for a defined task scope.
2. Stage One: Building Distillation Data from Problem-Solving Processes
The core of distillation is not simply copying a teacher model’s final answers. It is preserving as much useful information as possible about how the teacher solves a problem.
In this case, the teacher model generated approximately 250,000 training samples for mathematics and science questions. The samples included not only final answers, but also information such as:
- How to identify the known conditions in a problem;
- How to select appropriate formulas, theorems, or reasoning paths;
- How to break down a complex problem;
- How to calculate and validate intermediate steps;
- How to check whether a result satisfies the problem requirements;
- How to present the final conclusion clearly.
Compared with simple question-answer pairs, data that includes process descriptions is more suitable for training reasoning models. The smaller model is not only learning isolated conclusions; it is learning reusable frameworks for solving problems.
However, process data also introduces quality risks. A teacher model may produce redundant derivations, incorrect intermediate steps, or proofs that appear plausible but are invalid. Before entering the training set, samples generally need format cleaning, answer verification, duplicate filtering, and difficulty stratification. For mathematics and science tasks, executable calculation checks, rule-based validation, or manual spot checks should be used whenever possible to prevent flawed reasoning chains from being amplified by the smaller model.
3. Stage Two: Supervised Fine-Tuning and Lightweight Reinforcement Learning
After reasoning samples are collected, they can be used to perform supervised fine-tuning on a 9B base model. The purpose of supervised fine-tuning is to teach the model the teacher examples’ response formats, problem decomposition methods, and derivation patterns.
This stage generally affects three aspects of model behavior:
Task Recognition
The model can more quickly identify whether a question is a calculation, proof, physics derivation, or multi-step analysis task.Process Organization
The model becomes more likely to state the conditions first and derive the solution step by step, rather than returning an unexplained conclusion.Answer Structure
The model can follow the style of the training samples and provide relatively complete solution processes along with final results.
After supervised fine-tuning, the training workflow applies lightweight reinforcement learning for further optimization. Compared with large-scale reinforcement learning, lightweight approaches usually focus on output quality within a limited task scope, such as whether an answer is correct, whether reasoning is complete, and whether the required format is followed.
This step can correct common limitations of supervised fine-tuning. A model may imitate a teacher’s writing style without actually improving answer accuracy. It may also generate very long derivations that contain ineffective or irrelevant steps. Outcome-based rewards or rule-based rewards can encourage the model to reduce unnecessary output and improve the accuracy and consistency of its conclusions.
It is important to note that if the training data is concentrated mainly on mathematics and science, without covering programming, knowledge-based question answering, long-form analysis, and other task types, the resulting gains will also be concentrated in scientific reasoning. Improved performance on a specialized benchmark should not be interpreted as evidence that general model capability has improved in every domain.
4. Evaluation Results: Accuracy Is Not Enough—Inference Cost Also Matters
According to publicly reported evaluation records, the distilled 9B model answered more questions correctly on science-oriented tests while reducing the number of reasoning tokens consumed per correctly answered question by approximately 36.1%.
These metrics correspond to two critical goals in model deployment:
- Accuracy determines whether the model can solve a problem.
- Inference cost affects deployment cost, response speed, and concurrency capacity.
If a model improves accuracy only by generating much longer outputs, its practical value may be limited. Longer reasoning traces consume more compute resources and may increase the risk of accumulated errors. For this reason, examining both the number of correctly answered questions and the number of tokens required for each correct answer is more informative than measuring accuracy alone.
That said, fewer reasoning tokens should not automatically be interpreted as an absolute efficiency improvement. Evaluation should verify that the following conditions are consistent:
- Whether the question set and difficulty level are identical;
- Whether the maximum output length is the same;
- Whether the same sampling configuration is used;
- Whether answers are judged under the same correctness standard;
- Whether timeout, truncation, and formatting-error samples are excluded;
- Whether statistics include all questions or only correctly answered questions.
In particular, the metric of average tokens per correct answer can be affected by sample selection. A more rigorous report should also provide overall average output length, accuracy, throughput, and performance across different difficulty levels.
5. Stage Three: Adjusting Refusal Behavior Rather Than Retraining All Capabilities
After reasoning distillation was completed, another developer further modified the existing model weights to adjust refusal behavior.
Refusal behavior refers to a model’s tendency to decline answering, avoid answering, or request more information when presented with certain inputs. Refusal is not inherently a defect. In situations involving safety, privacy, illegal-risk concerns, or insufficient information, appropriate refusal is necessary. However, frequent refusals to answer ordinary mathematics, physics, or technical questions reduce usability.
The derivative version used a behavior-adjustment method based on internal activation analysis. The general idea was to compare changes in intermediate-layer activations when the model processed different types of prompts, identify signals associated with refusal tendencies, and then selectively modify the weights of certain network layers to reduce unnecessary refusals.
Publicly available configuration information indicates that the adjustments were concentrated in several middle layers of the model rather than applied across all parameters. This can keep the impact relatively controlled and help preserve existing knowledge and reasoning capability. However, it also carries risks: signals associated with refusal may be related to safety judgments, uncertainty assessment, and task-boundary recognition. Excessive modification could shift the model from over-refusing to answering without appropriate safeguards.
Therefore, behavior calibration should not be evaluated only by counting how much refusal frequency has declined. It should also assess:
- Response rates for normal questions;
- Blocking rates for high-risk prompts;
- The ability to request clarification when information is insufficient;
- Caution when dealing with uncertain factual content;
- Resistance to maliciously rewritten prompts.
6. Comparative Tests Show a Major Reduction in Refusals
In a comparative test containing 100 prompts, the original distilled version refused all prompts, while the behavior-adjusted derivative version refused only three. This result indicates that targeted modification did change the model’s output tendency.
The model card also reported a KL divergence of approximately 0.0422. KL divergence measures the difference between two probability distributions. In the context of model modification, it can help describe the extent to which the output distribution has shifted before and after an adjustment. A smaller value generally suggests that the overall output distribution has not changed dramatically. However, it does not prove that model behavior is better, nor can it independently demonstrate that safety or accuracy has been preserved.
More importantly, the refusal test and the science-capability test did not necessarily come from the same experimental run. Science accuracy, reasoning efficiency, and format-following results were primarily reported for the distilled version, while refusal counts came from the later derivative version. Therefore, these two sets of results should not be combined into a single claim that the final version improved across every metric.
The correct approach is to run a unified reevaluation under consistent conditions:
- Use the same question and prompt sets;
- Fix model inference parameters;
- Record accuracy, refusal rate, and token consumption;
- Report separate results for safety-related, normal-task, and boundary-case prompts;
- Test both pre- and post-modification models repeatedly;
- Report averages, variation ranges, and failure cases.
7. What This Distillation Approach Suggests for Model Development
First, high-quality process data matters more than simply increasing dataset size. When training examples show problem analysis, step decomposition, and result validation, smaller models are more likely to learn transferable methods.
Second, the boundaries of specialized optimization should be explicit. Mathematics and science data can substantially improve related tasks, but this does not mean the model becomes stronger in every scenario. Evaluation reports should clearly state the coverage of the training data and avoid overinterpreting specialized benchmark results.
Third, reasoning efficiency must be treated as a core metric. In real deployments, accuracy, latency, output length, and compute cost must be balanced together. Pursuing longer reasoning processes alone can increase cost and slow responses.
Fourth, refusal-behavior adjustments require safety regression testing. Reducing refusals does not automatically improve usability. Without evaluation on risk-related prompts, a model may lose necessary safety boundaries.
Fifth, experimental results from different derivative stages must remain separate. Distillation, reinforcement learning, and behavior editing are different operations with different potential effects. Reliable conclusions require evaluations conducted on the same model version, dataset, and testing protocol.
8. Conclusion
This 9B-model modification case demonstrates a typical path for targeted small-model enhancement: use a much larger model to generate high-quality reasoning samples, transfer science-oriented capability through supervised fine-tuning and lightweight reinforcement learning, and then apply localized calibration to refusal behavior.
Its main value is not a particular leaderboard position. Instead, it offers a reusable engineering approach: use data distillation to reduce the cost of capability transfer, use task-specific training to improve performance in limited scenarios, and use behavioral analysis to control how the model interacts with users.
For developers, the next priority should not be parameter compression alone. It should be building a more complete evaluation framework that separately measures accuracy, reasoning cost, refusal behavior, safety boundaries, and generalization. Only by validating these metrics under a shared experimental protocol can teams determine whether a derivative model is genuinely stronger or merely changed on a narrow local metric.
