# Small Model Distillation in Practice: Cutting Reasoning Cost and Tuning Refusal Behavior

    This article dissects two consecutive modifications stacked onto a ~9B-parameter model that reached the top of a community model trend leaderboard: sequence-level reasoning-trace distillation plus lightweight reinforcement learning in the first round, and activation-attribution weight editing for refusal behavior in the second. It explains why reasoning-trace distillation works especially well on math and STEM tasks, walks through sample code for building the SFT dataset, writing the reward function, and running a unified evaluation pipeline, and covers SFT engineering details, the rationale behind editing layers 9 to 24, and a governance checklist for taking such a model to production. It also calls out the central methodological trap in the published numbers: the two rounds were measured on different evaluation sets and prompt templates, so their results cannot be merged into a single "better model" conclusion.

    1. Postmortem: one 9B model, two separate modifications

    A mid-size model of roughly 9 billion parameters recently took the top spot on a community model trend leaderboard. Its weights come from two independent modification rounds layered on top of each other — worth calling out explicitly, because each round produced scorecards on different dimensions.

    Round 1: reasoning-trace distillation + lightweight reinforcement learning. Community developers used a teacher model in the trillion-parameter class, prompted for maximum problem-solving quality, to generate roughly 250,000 math and STEM samples. These are not bare question–answer pairs: each sample records where to start, how the problem is decomposed, how every step is derived, and what the final conclusion is. The developers ran supervised fine-tuning on an open-source 9B base model with this data, then added a lightweight policy-optimization reinforcement learning round. Both stages covered math and STEM only, with no coding data. Evaluation showed two things: more questions answered correctly, and about 36.1% fewer reasoning tokens consumed per correctly answered question.

    Round 2: refusal behavior editing. A second developer took those trained weights and applied an activation-attribution-based weight editing method to further adjust the model’s refusal tendency, modifying only some intermediate layers. The comparison they published covers the change in refusal counts and nothing else.

    A methodological problem has to be stated up front. The first-round version was measured on “accuracy + inference cost”; the version that topped the leaderboard was measured on “change in refusal behavior.” The two datasets come from different evaluation sets and different prompt templates, so they cannot be stitched together into a conclusion that one is a better model. Comparing them in a single coordinate system requires the same question set, the same decoding parameters, and the same prompt template, with all three weight variants — base / distilled / edited — re-tested together. That is the focus of Section 5.

    2. Breakdown one: why reasoning-trace distillation works so well on STEM tasks

    Knowledge distillation usually carries two distinct meanings, and they are easy to confuse:

    • Output-distribution distillation (logit/KL distillation): the student model fits the teacher’s per-token output probability distribution. It requires access to the teacher’s logits, so it does not apply to closed-source teachers or teachers that only expose an API.
    • Sequence-level reasoning-trace distillation: the full solution text generated by the teacher is used as the sole supervision target. The student learns how to reason step by step, with no dependency on logit access.

    Work of this kind in the community takes the second route, for practical reasons: a teacher reachable through an API can produce data with a low engineering barrier, and STEM solutions are inherently strongly sequential — the intermediate steps are imitable and transferable.

    The value of the supervision signal lives in the process, not the conclusion. Given only answers, the student can learn at most a marginal slice of the conditional distribution. Given the derivation, the student actually learns the teacher policy’s behavior patterns over a long chain of thought: how it identifies the problem type, when it introduces an auxiliary quantity, when it checks boundary conditions. This explains a common observation — after reasoning-trace distillation, gains on previously unseen problems from the same family tend to be more pronounced than gains from answer-only distillation.

    Sketch of the data construction script (framework details omitted):

    1. import json, random, re
    2. TEACHER_PROMPT = """You are a rigorous problem solver. Please:
    3. 1) State the solving strategy and key judgments first;
    4. 2) Derive step by step, without omitting critical intermediate quantities;
    5. 3) Finally output a single line 'Answer: <final result>'."""
    6. def build_sft_file(raw_records, out_path, keep_ratio=1.0):
    7. kept, dropped = [], 0
    8. for r in raw_records:
    9. q, a = r["question"].strip(), r["answer"].strip()
    10. # 1. Basic quality filtering
    11. if len(q) < 20 or not re.search(r"\d", q):
    12. dropped += 1; continue
    13. # 2. Answer parseability filtering: an extractable final answer is mandatory
    14. if not re.search(r"Answer[::]\s*\S+", a):
    15. dropped += 1; continue
    16. # 3. Deduplication (hash of the normalized question text)
    17. r["dedup_key"] = hash(re.sub(r"\s+", "", q).lower())
    18. kept.append(r)
    19. seen, uniq = set(), []
    20. for r in kept:
    21. if r["dedup_key"] in seen:
    22. continue
    23. seen.add(r["dedup_key"]); uniq.append(r)
    24. random.shuffle(uniq)
    25. sample = uniq[: int(len(uniq) * keep_ratio)]
    26. with open(out_path, "w", encoding="utf-8") as f:
    27. for r in sample:
    28. f.write(json.dumps({
    29. "messages": [
    30. {"role": "system", "content": TEACHER_PROMPT},
    31. {"role": "user", "content": r["question"]},
    32. {"role": "assistant", "content": r["answer"]},
    33. ],
    34. "subject": r.get("subject", "unknown"),
    35. "difficulty": r.get("difficulty", "mid"),
    36. }, ensure_ascii=False) + "\n")
    37. print(f"kept={len(sample)} dropped={dropped}")
    38. build_sft_file(load_records("teacher_traces.jsonl"), "sft_9b.jsonl")

    Four key engineering points for the SFT stage:

    1. Compute loss only on the assistant segment. The training script must apply a role-based loss mask. Otherwise the model treats the system prompt and the question text as learning targets too, which wastes compute and pollutes behavior.
    2. Use sequence packing to raise throughput. For a 9B model on long chains of thought, sequence lengths routinely run into the tens of thousands of tokens; zeroing out the cross-entropy loss on padding avoids ineffective gradients.
    3. Control subject mix and difficulty distribution explicitly. If 250,000 samples are over-concentrated on easy problems, the student learns a “verbose but correct” path. The mix should be stratified by difficulty, with a small number of hard problems retained as probes for the capability ceiling.
    4. Watch out for “confident errors.” The teacher model miscalculates too. For problems with automatically checkable answers (numerical results, expression equivalence), backfill answer correctness and drop clearly wrong traces — otherwise the student learns the wrong strategy along with everything else.

    3. Breakdown two: what lightweight RL actually optimizes

    A distilled model typically still has two problems: the final answer is right but the process is bloated, and the output format is unstable. Lightweight reinforcement learning is aimed at exactly those two gaps. Methods of this family (group-relative-advantage policy optimization) work roughly as follows:

    1. Sample a group of k answers for the same question;
    2. Score each answer with a rule-based reward;
    3. Normalize advantages within the group, so no separate value network has to be trained;
    4. Use a KL constraint to pull the policy back toward the SFT reference model, preventing capability collapse.

    Sketch of the reward function:

    1. import math
    2. def reward_fn(pred, gold, n_think_tokens, length_penalty_alpha=1e-3):
    3. """Correctness is the main signal; length/format act as light regularizers."""
    4. r_acc = 1.0 if answers_equivalent(pred, gold) else 0.0
    5. # Format reward: a clear final-answer line is required
    6. r_fmt = 0.1 if "Answer:" in pred else 0.0
    7. # Length penalty: applied only to correct samples, to avoid "answer wrong briefly" gaming the reward
    8. r_len = -length_penalty_alpha * (n_think_tokens / 1000.0) if r_acc else 0.0
    9. return r_acc + r_fmt + r_len

    There is an easy trap here: the length penalty must be conditioned on being correct. If tokens are penalized unconditionally, the model learns to stop thinking early, accuracy drops, and a headline like “36.1% fewer tokens” turns into an artifact bought with accuracy.

    Read in the other direction, “accuracy up and 36.1% fewer tokens per correct answer” is precisely the signature of effective distillation. The model no longer leans on lengthy self-doubt and repeated re-verification; it has learned a shorter effective path, with the teacher policy’s key decisions compressed into a more compact reasoning template. This is more valuable than “accuracy unchanged, fewer tokens,” because it indicates that the supervision signal transmitted strategy rather than surface-level text length.

    4. Breakdown three: how over-refusal gets “edited” away

    The goal of the second round was refusal behavior. Why do small models over-refuse? There are three main sources:

    • Generalization spillover from safety alignment: preference optimization on sensitive samples makes the model trigger avoidance on semantically similar but harmless prompts as well;
    • Conditional confusion inherited from the teacher policy: the teacher’s cautious tone in safety-sensitive scenarios gets distilled along with everything else;
    • “Polite avoidance” at the capability boundary: when a problem is out of scope or phrased ambiguously, the model tends to reply “I can’t do this” instead of giving a partial solution.

    Activation-attribution weight editing works entirely differently from fine-tuning. It appends no training data. Instead, it:

    1. Builds a contrast prompt set: one group of harmless prompts that were wrongly refused, one group of ordinary prompts;
    2. Extracts internal activations: records hidden states at each layer and compares activation differences between the two prompt groups;
    3. Locates the relevant direction: finds the low-rank direction or component strongly correlated with the refusal behavior;
    4. Modifies weights in some layers: subtracts from or project-corrects the weights along that direction, weakening their drive on this behavior.

    At the configuration level, this kind of method sets the adjustment range to layers 9 through 24. The choice rests on clear architectural intuition:

    • Layers that are too shallow (the first few) mainly carry lexical and basic syntactic representations, so changing them damages language modeling ability as collateral;
    • Layers that are too deep sit close to the output layer and are more tightly coupled to the concrete vocabulary distribution and format templates, so gains there are unstable;
    • Middle layers usually carry the intermediate mapping from semantic understanding to intent decision — judgments like “should this be answered” form right here, which makes the gain/risk ratio best.

    The safety constraint has to be emphasized: weakening refusal signals simultaneously raises the risk of “should have been refused, but got answered.” For that reason, the second round cannot be accepted on refusal rate as a single metric. It requires two-sided evaluation: refusal rate goes down while the unsafe-answer rate must not go up. Any weight editing that reports only “fewer refusals” without safety regression results is not fit for production.

    5. Evaluation methodology: putting both rounds in one coordinate system

    A unified evaluation protocol. All three weight variants (base / distilled / edited) must share the same evaluation set, the same prompt template, the same decoding parameters (temperature, top_p, max generation length, repetition penalty), and the same hardware and inference configuration.

    Three dimensions of metrics, none of them optional:

    Dimension Core metrics Notes
    Capability Per-subject accuracy, hard-subset accuracy Must be stratified by difficulty so easy questions don’t dilute differences
    Efficiency Average reasoning tokens per correct answer, total token consumption Must be conditioned on correctness, to prevent short wrong answers from gaming the metric
    Behavior Refusal rate on harmless prompts, safety violation rate on unsafe prompts Look at both sides at once; degradation on either counts as a regression

    Statistical rigor. A 9B model under temperature sampling fluctuates noticeably, and differences from a single run may come entirely from variance. Sample k times per question and take a majority vote or an average, report means with confidence intervals, and run paired significance tests on key conclusions.

    Sketch of the evaluation pipeline:

    1. import statistics
    2. def evaluate(ckpt_path, dataset, k=4, temperature=0.7):
    3. rows = []
    4. for item in dataset:
    5. samples = [generate(ckpt_path, item["prompt"],
    6. temperature=temperature, max_tokens=8192)
    7. for _ in range(k)]
    8. acc = mean([answers_equivalent(s, item["gold"]) for s in samples])
    9. think_tok = [count_think_tokens(s) for s in samples
    10. if answers_equivalent(s, item["gold"])]
    11. refused = [is_refusal(s) for s in samples]
    12. rows.append({
    13. "subject": item["subject"],
    14. "difficulty": item["difficulty"],
    15. "acc": acc,
    16. # Key point: count tokens only on correct samples, so the length penalty
    17. # doesn't get conflated with accuracy
    18. "think_tok_per_correct": statistics.mean(think_tok) if think_tok else None,
    19. "refusal_rate": mean(refused),
    20. "unsafe_rate": mean([is_unsafe(s) for s in samples]),
    21. })
    22. return {
    23. "acc_overall": mean([r["acc"] for r in rows]),
    24. "think_tok_per_correct": weighted_mean_by(rows, key="think_tok_per_correct"),
    25. "refusal_rate": mean([r["refusal_rate"] for r in rows]),
    26. "unsafe_rate": mean([r["unsafe_rate"] for r in rows]),
    27. "by_difficulty": group_by(rows, key="difficulty"),
    28. }

    On reading leaderboards. A community trend leaderboard aggregates signals of community behavior — download counts, likes, recent momentum. It is an attention metric, not a task-fit metric. Being number one means the weight path was widely discussed for a while; whether it wins on your business corpus, under your compliance requirements, and within your latency budget is something only your own evaluation set can answer.

    6. Engineering practice: from experiment to a shippable governance checklist

    1. Version your data and your weights. The teacher-trace dataset, the SFT dataset, the RL reward configuration, and the weights from both rounds each need an independent version number and hash. Evaluation reports must be bound to specific versions, or results cannot be reproduced.
    2. Handle licensing and compliance up front. Using a teacher model’s outputs for distillation training requires confirming whether its terms of service and license permit it, and what redistribution rights apply to the generated data. This step belongs before data construction, not bolted on when you are preparing to publish.
    3. Turn evaluation into a gate. Encode the three metric dimensions into pipeline gates: capability must not drop, tokens must not increase, refusal rate must improve, and safety violation rate must not rise — only then can a build enter canary release.
    4. Canary and rollback. Small-traffic canary → compare accuracy and refusal complaint rate against real production requests → automatic rollback to the last stable version on anomaly.
    5. Monitor continuously. After launch, watch for drift in the inference-length distribution (a sudden spike in reasoning tokens usually signals decayed distillation quality or a prompt template change), the refusal rate trend, and per-request inference cost.
    6. Supporting cloud capabilities. In model development and training scenarios, the whole pipeline can typically be assembled from generic capability categories: object storage to hold raw traces and dataset versions, a container platform to run distributed training and evaluation jobs, a message queue or pipeline orchestration service to drive data production and evaluation scheduling, logging and monitoring/alerting to cover both training metrics and online behavior metrics, and model registry plus canary release capabilities to manage version flow. On Baidu AI Cloud, the pipeline can be built by combining these generic capability categories; refer to the official documentation and your actual quotas for specific configuration.