Data Reuse and Distillation in LLM Training: Principles, Pathways, and Engineering Practices

1. Background: Where Does Post-Training Data Come From?

In the LLM research and development lifecycle, training is generally divided into two major phases: pre-training and post-training. The pre-training phase relies heavily on massive public corpora to equip the model with foundational language understanding and generation capabilities. The post-training phase, however, focuses more on alignment, reasoning enhancement, and task adaptation. The quality and distribution of data used during post-training often have a more direct impact on the final model performance than sheer data scale.

However, the manual acquisition cost of high-quality annotated data remains prohibitive. Especially in scenarios involving complex reasoning, multi-step logic chains, and code generation, manually constructing the complete trajectory of “question - intermediate reasoning - final answer” is both expensive and difficult to scale. Consequently, the industry has gradually adopted a common technical approach: utilizing existing stronger models to generate high-quality trajectories or samples for learning, which are then used to train the target model. This approach is typically recognized as a practical form of data distillation or knowledge distillation.

This article will systematically break down this technical pathway across several dimensions: reasoning trajectory collection, synthetic data generation, architecture referencing, and engineering implementation.

2. Collection and Utilization of Reasoning Trajectories

2.1 What is a Reasoning Trajectory?

A reasoning trajectory refers to the complete intermediate process a model generates when solving a complex problem. This includes, but is not limited to:

• Problem decomposition steps
• Intermediate calculations or logical deductions
• Filtering and validation of candidate answers
• Generation of the final conclusion

Compared to standard supervised data that only provides a “question - standard answer” pair, reasoning trajectories allow the target model to learn the process of “how to think,” rather than just memorizing “what the result is.”

2.2 Basic Workflow for Trajectory Collection

A relatively complete reasoning trajectory collection workflow typically involves the following steps:

Step 1: Construct a question set. The question set can be derived from public datasets, anonymized real-world user queries, or automatically generated using rules and templates. The questions must possess a certain level of complexity to reflect the value of the reasoning process.

Step 2: Invoke the teacher model to generate trajectories. Input the questions into a highly capable model (hereafter referred to as the teacher model) and enable the mode that exposes the intermediate reasoning process. If the model supports outputting structured reasoning steps, they can be recorded directly; otherwise, post-processing parsing is required to segment the output text into distinct steps.

Step 3: Quality filtering and cleaning. Not all generated trajectories are suitable for training data. Common filtering strategies include answer correctness verification, reasoning step completeness checks, deduplication, and consistency evaluation with human preferences. This can be achieved using rules, automated scoring models, or a small amount of manual sampling.

Step 4: Format standardization. Standardize the trajectories into a data format that is easy for the target model to learn. A common practice is to store each reasoning step as an independent field or use fixed delimiter tokens within the text. For example:

{
  "question": "A city's subway fare is calculated by distance: 3 yuan for the first 6 km, and an additional 1 yuan for every extra 10 km. How much is the fare for a 32 km ride?",
  "steps": [
    "The distance exceeding the initial range is 32 - 6 = 26 km",
    "How many 10 km segments are in 26 km: ceil(26 / 10) = 3",
    "Total fare = 3 + 3 * 1 = 6 yuan"
  ],
  "final_answer": "6 yuan"
}

This structure is beneficial for subsequent loss function design, masking control, and evaluation during training.

2.3 Training Strategies Utilizing Trajectory Data

When utilizing reasoning trajectories to train the target model, the following strategies can be employed:

• Full trajectory supervision: Use the question as input and concatenate the step sequence with the final answer as the output target, calculating cross-entropy loss for each position.
• Staged training: First train the model to generate individual reasoning steps, then gradually scale up to the full trajectory to reduce learning difficulty.
• Weighting key steps: Assign a higher loss weight to the last reasoning step before the final answer, forcing the model to focus on the critical components that determine the conclusion.
• Adding format constraints: Uniformly use special tokens in the training samples to distinguish questions, steps, and answers, preventing the model from outputting verbose and irrelevant content during inference.

It is crucial to note that distilled trajectory data often contains inherent pattern biases. The teacher model might lean towards a fixed reasoning style or exhibit abnormally stable behavior on specific types of problems. If used directly for training without processing, the target model may inherit these biases. Therefore, increasing the diversity of data sources and applying regularization techniques during training are necessary.

3. Generation and Usage of Synthetic Data

3.1 Positioning of Synthetic Data

In the context of this article, synthetic data refers to data generated by models rather than directly annotated by humans. It does not replace manually annotated data; rather, it is complementary. Synthetic data excels at covering long-tail scenarios, rapidly scaling up volume, and reducing marginal costs. Conversely, manual annotation remains irreplaceable for critical boundaries, safety compliance, and domain-specific expertise.

3.2 Common Methods for Generating Synthetic Data

A common method for generating synthetic data starts with a set of “seed questions.” Through prompting, the teacher model is instructed to expand or rewrite these seeds to generate more variations. For example:

• Synonymous rewriting: Change the phrasing, scenario, or entity names while preserving the original semantics.
• Difficulty escalation: Add constraints or multi-step reasoning requirements to the original problem.
• Format conversion: Convert formats between multiple-choice, true/false, and question-and-answer.
• Counterfactual generation: Alter certain conditions to generate new questions, thereby enhancing the model’s robustness.

Programmatic methods can be used for batch control during this process. For instance, defining a structured prompt template for a generation task:

Given the following example question:
{seed_question}
Please generate 5 new questions with the following requirements:
1. The semantics should be related to the example question but not exactly the same;
2. Each question must involve at least 3 steps of reasoning;
3. The output format should be a JSON array, with fields including question, steps, and final_answer.

The teacher model’s output is then parsed into a usable dataset.

3.3 Data Quality Control Loop

The core risks of synthetic data lie in “error amplification” and “distribution shift.” If the generated data contains incorrect answers or flawed reasoning, the target model may learn them as correct knowledge. Therefore, constructing a control loop is essential:

1. Automated verification: Math problems can be verified via programmatic recalculation; code problems can be tested by executing test cases; knowledge QA can be compared against retrieval results.
2. Consistency checks: Sample the same question multiple times and observe answer stability. Low-consistency samples can be flagged as low-confidence and assigned lower training weights.
3. Distribution monitoring: Track distributional differences between synthetic data and real-world business data across dimensions such as length, category, and difficulty, adjusting generation strategies promptly.
4. Feedback correction: Feed the failure samples of the trained target model on the validation set back into the next round of data generation or cleaning, forming an iterative optimization loop.

The value of synthetic data can only be reliably unlocked when it is placed within a complete quality control loop.

4. Boundaries and Methods of Architecture Referencing

In model R&D, referencing the architectural designs of existing models is a common industry practice. “Architecture referencing” here does not mean directly copying a set of code or configurations, but rather learning from and validating design concepts such as the main model structure, attention mechanisms, normalization positions, layer configurations, and activation functions.

For developers, a more pragmatic strategy is:

• Treat structural designs described in public papers as candidate options, not default optimal solutions.
• Quickly conduct ablation experiments on small-scale models to verify whether a structural change brings positive gains.
• Pay attention to the compatibility between the architecture and the training infrastructure. For example, certain structures have specific requirements for memory usage, communication overhead, and inference efficiency, which need to be evaluated based on available resources.
• Maintain modular design. Decouple components like attention computation, feed-forward networks, and positional encoding to facilitate flexible replacement and comparison.

For instance, when deciding whether to adopt a specific normalization strategy, pseudo-code can be used for experimental control:

for norm_type in ["pre_norm", "post_norm", "sandwich_norm"]:
    model = build_model(norm_type=norm_type)
    metrics = train_and_evaluate(model, dataset)
    record(norm_type, metrics)

This evidence-based approach to architecture selection based on experimental data holds more engineering value than blindly following the structure of a specific model.

5. Key Considerations in Engineering Practices

5.1 Data Compliance and Licensing

When using existing models to generate training data, developers must verify whether the terms of use permit such actions. Different model open-source licenses may have varying restrictions on distillation, data generation, and commercial use. Even if it is technically feasible, a compliance review must be completed first to avoid future risks.

5.2 Avoiding Data Contamination in Evaluation

When synthetic data and the target test set originate from the same teacher model or highly similar distributions, evaluation results may be artificially inflated. Ensure that the test set comes from an independent source, or exclude samples highly similar to the training data during the evaluation phase.

5.3 Model Capability Boundaries and Transferability

A strong teacher model does not imply that all of its behaviors are worth learning. Certain capabilities might be byproducts of the teacher model’s unique structure or training strategy, and may not be transferable when the target model’s conditions differ. Focus should be placed on general reasoning capabilities, problem-solving steps, and language expression, rather than blindly pursuing exact output replication.

5.4 Training Stability

When training with distilled trajectories, the model may sometimes prematurely converge to a “shortcut,” such as outputting only the final answer while skipping the reasoning process. To mitigate this, increase the proportion of full trajectory samples during the early stages of training, or introduce reasoning step completeness as a metric in the validation process to help the model retain its intermediate reasoning capabilities.

6. Conclusion

From the perspective of data reuse and knowledge distillation, this article has outlined the complete pathway for utilizing existing model reasoning trajectories and synthetic data during the LLM post-training phase. From trajectory collection, format standardization, and training strategy selection, to synthetic data generation, quality control loops, and the boundaries and experimental methods of architecture referencing, each step requires engineered design and continuous iteration.

For development teams, the value of mastering this methodology lies in its ability to significantly reduce the acquisition cost of high-quality training data under compliance prerequisites, improve model performance on complex reasoning tasks, and avoid the potential risks of “blindly copying data” through systematic evaluation and iteration mechanisms. Future trends point towards more automated data quality evaluation, more standardized distillation workflows, and increasingly diversified training data sources.