How to Train Small Models with Large Models: Distillation, Synthetic Data, and Architecture Practices

1. From Stacking Parameters to Borrowing Capability

For years, gains in language-model capability scaled almost in proportion to parameter count and raw data volume. Today, an increasingly mainstream engineering practice looks quite different: during post-training, lightweight models consume large volumes of reasoning traces produced by much larger models as supervision signals; earlier versions of this approach relied directly on synthetic data generated by large models for alignment training; and their network designs borrow from the mature open architectures of the day.

This “large models feeding small models” technical route has become popular because of three real-world constraints:

1. High-quality human annotation is expensive and scarce. An expert-verified mathematical derivation or multi-turn dialogue sample costs far more per unit than ordinary web text, and delivery cycles are measured in weeks. When a small model needs supervised fine-tuning across millions of samples, a purely human pipeline is almost never affordable.
2. Large-model outputs are themselves high-density knowledge carriers. The complete chain of thought a large model leaves behind when solving a problem contains both domain knowledge and process-level skills: decomposing the problem, backtracking from dead ends, self-verifying. Such signals are nearly impossible for humans to write from scratch, yet they can be harvested at scale.
3. Inference cost and deployment constraints. Enterprise scenarios often require models that run with limited GPU memory, low latency, and private deployment; small models are a hard requirement, and the capability gap is filled through distillation and synthetic data.

Once you understand these three constraints, it becomes clear why the teacher–student division of labor is evolving from scattered experiments into a systematic data engineering discipline.

2. Reasoning-Trace Distillation: Teaching Small Models How to Think

2.1 How Reasoning Traces Differ from Conventional Fine-Tuning Data

Conventional supervised fine-tuning data is typically an input–output pair: one question, one reference answer. A reasoning trace instead preserves the model’s full intermediate thinking process—how the problem was decomposed, which paths were attempted, at which step a dead end was discovered, how the reasoning was corrected—before finally arriving at the answer.

For a student model, the two types of signal carry entirely different value: the former teaches what the answer is; the latter teaches how to think. And “how to think” is precisely the decisive variable in reasoning-heavy tasks such as mathematics, coding, and complex planning.

2.2 Collection Strategy: Diversity Matters More Than Quantity

A few practices have been validated repeatedly in the field:

• Multi-temperature sampling: generate multiple solutions to the same problem at different sampling temperatures (for example, 0.3 / 0.7 / 1.0) to capture both conservative and exploratory approaches;
• Multiple solutions per problem: keep several distinct reasoning paths for the same question so the model sees that many roads lead to a correct answer, rather than overfitting to a single path as the only valid pattern;
• Handling failed samples: purely incorrect traces are usually discarded, but traces that go wrong first and then self-correct have proven valuable in some practices as demonstrations of process-level skills.

2.3 Two Post-Training Stages

Collected traces are typically consumed in two stages:

Stage one: cold-start supervised fine-tuning. Feed the filtered traces directly into SFT so the student quickly learns the reasoning format and basic strategies, providing a usable initial policy distribution for subsequent reinforcement learning.

Stage two: reinforcement learning with verifiable rewards. On tasks with objective correctness (numerical computation, code that must pass test cases), let the student model sample on its own and reward only outputs that are correct in answer and reasonable in process. This pushes capability beyond the level of the teacher data itself—the key mechanism behind several lightweight models outperforming their teacher source on certain benchmarks.

2.4 Quality Filtering: The Unglamorous Work That Sets the Distillation Ceiling

Teacher models are not always right—garbage in, garbage out. Common filtering measures include:

• Correctness verification: anything that can be verified programmatically (running code against tests, checking numerical answers) is verified automatically; for everything else, multi-sample voting consistency serves as the fallback;
• Format and length rules: discard truncated traces, circular reasoning, and overlong responses that spin without progress;
• Near-duplicate removal: deduplicate via text fingerprints or embedding similarity to keep massive homogeneous samples from teaching the model to memorize rather than reason.

An illustrative pipeline (pseudocode):

# Reasoning-trace collection and filtering pipeline (illustrative)
def build_trace_dataset(teacher, tasks, cfg):
    traces = []
    for task in tasks:
        # Multi-temperature sampling increases path diversity
        for temp in cfg.temperatures:
            resp = teacher.generate(
                task.prompt, temperature=temp,
                top_p=0.95, max_tokens=cfg.max_tokens,
            )
            # Verifiable tasks: check the answer programmatically
            if task.verifiable and not check(task, resp.answer):
                continue
            # Rule-based filtering: truncation / loops / runaway length
            if not passes_rules(resp, cfg):
                continue
            traces.append(to_sample(task, resp, meta={"temp": temp}))
    # Embedding-based near-duplicate removal
    return dedup_by_embedding(traces, threshold=0.92)

3. Synthetic Data: A Custom Curriculum for Small Models

If reasoning traces answer “can it reason,” synthetic data answers “coverage”—covering business scenarios, long-tail instructions, and diverse formats.

3.1 Four Common Forms

1. Instruction–response pairs: start from seed instructions and expand into new ones that vary in phrasing, difficulty, and domain;
2. Multi-turn dialogue: constructed by a large model playing multiple roles, with deliberate emphasis on clarification, follow-up questions, and error-correction behaviors;
3. Code and tool calls: automatically generate functions, call sequences, and exception-handling examples, which can be verified programmatically;
4. Knowledge injection: rewrite private documents into question-and-answer pairs, summaries, tables, and other formats to complete domain adaptation.

3.2 Generation Strategy: Seed-Driven, Not Aimless

Mature practices generally follow a seed–expand–refilter loop. First, seed samples are drawn from real business logs or high-quality public corpora to anchor the data distribution. Then a large model performs controlled mutation—rewriting style, raising difficulty, switching domains—while keeping the task structure unchanged. Finally, an independent evaluator or a rule set scores each synthetic sample, and only high-scoring samples enter the training set.

3.3 Preventing Model Collapse

When synthetic data accounts for too large a share and comes entirely from a single teacher, output distributions tend to narrow and diversity decays continuously—a risk the industry generally calls model collapse. Mitigations include:

• Mixing real and synthetic data at a fixed ratio; a common practice is to keep at least 20–30% real corpus;
• Introducing multiple teacher sources, or applying differentiated sampling parameters to the same teacher;
• Periodically monitoring output diversity and novelty on a holdout set (for example, the effective radius in embedding space).

4. Architecture Borrowing: Trimming Mainstream Designs

Lightweight models borrowing the mature open architectures of the moment is simply industry practice—published research at the architecture level is a shared asset for the entire field. The real engineering threshold lies in what you trim after borrowing:

• Depth-to-width ratio: under GPU-memory constraints, small models can adopt narrower-deeper or wider-shallower layouts; decide only after measuring throughput and latency on the actual deployment hardware;
• Attention and positional-encoding variants: adopting mainstream choices directly reduces the rework costs that come with training instability;
• Sparse expert structures: some mid-sized models adopt designs with a large total parameter count but a small activated parameter count, trading per-inference cost for capability gains—but these place higher demands on data volume and routing stability, so smaller teams should evaluate them carefully;
• Vocabulary and tokenizer: keeping them consistent with the teacher model aligns token distributions during distillation and reduces transfer loss.

A pragmatic rule of thumb: architecture decisions should serve deployment constraints and data scale, not chase whatever is fashionable in the latest papers.

5. Engineering the Pipeline

5.1 Data and Training Infrastructure

This kind of training is fundamentally a data-intensive engineering effort. Recommended infrastructure choices:

• Use object storage as the single home for raw corpora, intermediate artifacts, and final datasets, with versioning so that every training run can be traced back to an exact data snapshot;
• Run collection and filtering jobs on a container platform so they scale horizontally and failed jobs can be replayed;
• Use logging services and monitoring/alerting to record per-batch data distributions, discard rates, and training-metric drift for post-hoc root-cause analysis.

On public clouds such as Baidu AI Cloud, standard service categories—object storage, container platforms, logging services, and monitoring/alerting—cover the vast majority of this pipeline’s needs; teams running their own infrastructure can build on open-source components instead. Either way, the priorities remain the same three properties: traceable, replayable, observable.

5.2 The Evaluation Loop: Don’t Rely on Public Benchmarks Alone

Distillation and synthetic data efforts are most likely to stumble at the evaluation stage. Build a three-layer evaluation system:

1. Public benchmarks for horizontal positioning—but first check whether the benchmark may already be present in the teacher model’s training data; contamination can inflate scores significantly;
2. Business holdout sets for vertical regression, built from sampled real production data; every change to the data recipe must be run through them in full;
3. Manual spot checks as the final backstop, focused on style drift and hallucination rate.

6. Four Common Pitfalls

1. Teacher licensing and compliance: before training on any model’s outputs, confirm what its terms of service permit regarding the use of outputs to train downstream models. This is a legal question rather than a technical one, but it cannot be left unresolved until after launch.
2. Single-teacher homogenization: when all data comes from one model, the student tends to inherit its style and blind spots; mixing multiple sources is cheap insurance.
3. Benchmark contamination: problems from popular benchmarks very likely appear in the teacher’s data, producing inflated post-distillation scores that don’t hold up in real use.
4. Distillation without reinforcement: an SFT-only student is capped by the ceiling of its teacher data; only by adding reinforcement learning with verifiable rewards does it stand a chance of surpassing the teacher in narrow domains.

7. Conclusion

Small models drawing strength from large models has evolved from a training trick into a systematic methodology: reasoning traces transfer process-level capability, synthetic data broadens task coverage, and architecture borrowing keeps engineering risk low. It is the combination of the three that lets models with only a few billion parameters compete with far larger models in specific scenarios. For engineering teams, the real moat is not which vendor’s teacher model you used—it is the filtering quality of your data pipeline, the rigor of your evaluation loop, and a clear-eyed understanding of your deployment constraints. Get these fundamentals right, and model iteration turns from luck into reproducible engineering.