Reasoning Trace Distillation and Synthetic Data: Practices for Training Small Models

I. Background: Why Small Models Need to Stand on the Shoulders of Large Models

Current large language model (LLM) technology exhibits a clear capability stratification. Flagship models excel at complex tasks such as mathematical reasoning, code generation, and multi-step planning, but they come with high training and inference costs and significant deployment barriers. Conversely, smaller parameter models offer flexible deployment, rapid response, and lower unit costs, but struggle to replicate the reasoning proficiency of large models under zero-shot conditions. Enabling small models to achieve large-model-like task performance at a controllable cost has become a focal point for the industry.

Currently, three widely validated technical approaches are commonly combined:

1. Reasoning Trace Distillation: Using the complete problem-solving thought process of a large model as a training signal, allowing the small model to “learn the process” rather than just “learn the answer.”
2. Synthetic Data Augmentation: Utilizing large models to batch-generate high-quality corpora, breaking through the scale bottleneck of high-quality human-annotated data.
3. Architecture Paradigm Adaptation: Referencing well-validated structural designs from the community to reduce architecture trial-and-error costs.

These three paths correspond to “what to train, how to train, and how to build,” collectively forming a complete methodology for the post-training phase of small models.

II. Reasoning Trace Distillation: From “Learning Answers” to “Learning Processes”

2.1 What is a Reasoning Trace?

A reasoning trace refers to the intermediate thought content generated by the model before providing the final answer, typically presented as a Chain of Thought (CoT). Compared to data containing only “question-answer” pairs, trace data includes rich behavioral signals such as problem decomposition, step-by-step calculation, self-verification, and backtracking correction. For small models with limited parameters, explicit process supervision is often more effective than mere answer supervision because it compresses the solution space and reduces the model’s tendency to “guess answers.”

2.2 Collection and Construction of Trace Data

The standard collection process involves four stages:

1. Task Design: Covering scenarios like math, code, logical reasoning, and multi-hop QA. Prompts instruct the model to “think first, answer later” and constrain the output format for easy parsing.
2. Multiple Sampling: Sampling multiple candidate traces for the same question using different temperature parameters to increase process diversity.
3. Result Verification: Executing programmatic comparisons for tasks with ground-truth answers, retaining only the traces that yield correct answers (i.e., rejection sampling).
4. Trace Cleaning: Removing samples with circular redundancy, formatting errors, or severe step jumps, and unifying them into a structured format.

A representative data format is as follows:

{
  "id": "math-000123",
  "question": "Solve the equation 3x + 7 = 22",
  "reasoning_trace": [
    "Step 1: Subtract 7 from both sides to get 3x = 15",
    "Step 2: Divide both sides by 3 to get x = 5",
    "Verification: 3 * 5 + 7 = 22, which matches the original equation"
  ],
  "final_answer": "x = 5"
}

The core logic of rejection sampling can be expressed in pseudocode:

def collect_traces(prompts, teacher, verifier, k=8):
    dataset = []
    for p in prompts:
        for cand in teacher.sample(p, n=k, temperature=0.7):
            if verifier.check(p, cand.answer):  # Keep only verifiable correct traces
                dataset.append(format_trace(p, cand))
                break
    return dataset

2.3 Training Strategy: SFT Foundation, RL Enhancement

• Supervised Fine-Tuning (SFT): Training using traces as target sequences to help the small model acquire standard thinking formats and reasoning patterns. During this phase, attention must be paid to loss masking settings, typically computing loss only on the response portion.
• Reinforcement Learning (RL) Continuation: After SFT establishes reasoning habits, RL is applied using verifiable rewards (e.g., answer correctness, test case pass rates) to guide the model in further optimizing path selection within the correct format.
• Curriculum-based Data Orchestration: Training in stages from easy to difficult, preventing early training instability caused by overly difficult samples dominating the learning process.

III. Synthetic Data: Balancing Scale and Quality

3.1 The “Generation-Filtering-Verification” Closed Loop

The upper limit of synthetic data depends on the quality control system. A mature approach involves building a three-stage closed loop:

1. Generation: Using seed tasks, knowledge graphs, or outline-driven prompts to control topic coverage and difficulty distribution.
2. Filtering: A dual-layer check comprising a rule-based layer (language fluency, format validity, length distribution) and a model-based discriminative layer (quality scorer).
3. Verification: Executing programmatic checks on verifiable tasks, such as cross-checking math answers, executing code samples, and tracing fact-based assertions.

3.2 Diversity and Decontamination

• Combining exact n-gram deduplication with semantic vector nearest-neighbor deduplication to prevent the model from overfitting to a few expression templates.
• Actively controlling the ratio of difficulty levels and question types to avoid capability bias caused by an overscreening of simple questions.
• Conducting overlap detection (decontamination) against commonly used public evaluation benchmarks to ensure subsequent evaluation results are authentic and credible.

3.3 Mixing with Real Data

Synthetic data should not be used in isolation. In practice, it is typically mixed with real corpora at a specific ratio and dynamically adjusted across different training stages: increasing the proportion of high-quality synthetic reasoning data in the mid-to-late stages to strengthen target capabilities, while retaining sufficient real corpora to maintain language understanding and generalization capabilities and prevent distribution drift.

IV. Architecture Adaptation: Referencing Paradigms Rather Than Copying Configurations

The open-source community has accumulated numerous validated structural design paradigms, such as Mixture of Experts (MoE), Grouped-Query Attention (GQA), Rotary Position Embedding (RoPE), and long-context extension technologies. When adapting these findings, three principles should be kept in mind:

1. Capacity Matching: Small models have limited parameters. The number of experts, attention heads, and hidden layer dimensions should be reconfigured based on scaling experience. Directly copying large model configurations often leads to low parameter utilization.
2. Stability First: Structurally unstable training is difficult to troubleshoot in small-scale experiments. Priority should be given to component combinations thoroughly validated by the community.
3. Inference Friendliness: Small models are frequently deployed in resource-constrained environments. Structural selection must balance memory footprint, throughput, and quantization compatibility.

In a nutshell: what should be borrowed are “design philosophies and validated components,” not “hyperparameter tables.”

V. Engineering Implementation: A Complete Training Pipeline

Stage	Objective	Key Data
Mid-training	Build foundational linguistic intuition for reasoning and code	Large-scale mixed synthetic and real corpora
Supervised Fine-Tuning	Establish standard thinking and answering formats	High-quality reasoning trace data
Reinforcement Learning	Align with verifiable objectives and stabilize output	Task sets equipped with verifiers

Around this pipeline, there are four engineering support pillars:

1. Data Engineering: Multi-source collection, cleaning and deduplication, ratio orchestration, and version management to ensure every training run is reproducible and traceable.
2. Training Management: Distributed parallelism, checkpoint resumption, mixed precision, and gradient checkpointing to balance efficiency and stability.
3. Evaluation System: A three-tier combination of general capability benchmarks, reasoning-specific evaluations, and manual spot-checks, remaining vigilant against overfitting signals where the model “only improves on the synthetic distribution.”
4. Infrastructure: Can be built using general cloud capabilities. For instance, using object storage for corpus assets, container platforms for orchestrating training tasks, GPU resource pools for elastic computing power, and logging/monitoring services to track loss curves and throughput metrics. As an example, Baidu AI Cloud’s general product categories like object storage and container services can cover the foundational needs of the scenarios above.

VI. Risks and Compliance: Non-Negotiable Boundaries

1. Upstream Terms of Service: When using outputs from other models as training data, the permitted scope of their respective service terms and licenses must be verified to mitigate compliance risks.
2. Data Compliance: Synthetic data may reproduce private or copyrighted content present in upstream training corpora, necessitating the establishment of screening mechanisms.
3. Capability Inheritance Bias: Small models may inherit flawed reasoning patterns from the teacher model. This requires continuous monitoring via an independent validation set and rolling back data versions if necessary.

VII. Conclusion

The rapid growth of small models leveraging large model reasoning traces and synthetic data fundamentally shifts the competition from “parameter scale” to “data quality and training strategy.” Reasoning trace distillation solves “how to think,” the synthetic data closed loop solves “what to train,” and architecture paradigm adaptation solves “how to build.” The synergy of these three, coupled with strict quality control, evaluation, and compliance boundaries, enables developers to train small models that excel in target scenarios at a controllable cost. This approach is particularly friendly for resource-constrained teams and represents a mainstream practice in the current engineering implementation of large models.