Practical Guide to Data Reuse and Reasoning Trajectory Transfer in LLM Training

1. Background: Why Reuse Reasoning Trajectories and Synthetic Data?

As large language models (LLMs) continue to evolve, training data is no longer confined to human-annotated text. Outputs generated by the models themselves—such as reasoning processes, intermediate steps, final answers, and multiple candidate responses—are increasingly being recycled as high-value samples for subsequent training phases.

The core logic behind this approach is that outputs from advanced models on complex tasks often embed superior reasoning paths, more standardized expression structures, and more stable problem-solving strategies. By using these outputs to train smaller models or subsequent iterations within the same model series, developers can effectively transfer certain “capabilities” from the source model to the target model.

This paradigm generally falls into two technical categories:

1. Synthetic Data Generation: Using existing models to mass-produce training samples.
2. Trajectory Distillation / Reasoning Trajectory Reuse: Retaining the complete reasoning path leading up to the final result, rather than just using the final answer itself.

In practice, these two approaches are frequently combined to form a low-cost data expansion strategy. However, it is crucial to understand that data reuse is not merely copying output text. Effective transfer requires filtering, cleaning, alignment, and the reconstruction of evaluation metrics. Otherwise, it may introduce systematic biases or even cause the target model to inherit typical reasoning errors from the source model.

---

2. Core Value and Common Forms of Reasoning Trajectories

A reasoning trajectory can be understood as a record of the model’s internal process when answering a question: the intermediate steps taken, the candidate results generated, and the final output selected. Common forms include:

1. Single-Step Reasoning Chains
   The model directly outputs the complete derivation process from question to answer, such as step-by-step calculations in math or premise deduction in logic puzzles.

2. Multi-Turn Self-Correction Trajectories
   The model provides an initial answer, then refines it based on self-inspection or multi-turn internal evaluation, ultimately forming a more stable answer. These trajectories are highly valuable for training because they contain both “error patterns” and “correction paths.”

3. Search and Planning Paths
   In scenarios requiring task decomposition, the model might list sub-goals first and then solve them sequentially. This trajectory helps the target model learn task decomposition and long-term planning capabilities.

4. Candidate Ranking Trajectories
   After generating multiple candidate answers, the model selects the optimal one through scoring or comparison mechanisms. This process retains the model’s rationale for judging the quality of different answers.

In actual training, developers rarely use all trajectories verbatim for the next round. Instead, they selectively retain certain phases based on the task type and the target model’s specific weaknesses. For example, if the target model struggles with mathematical reasoning, it is more beneficial to retain the step-by-step derivation and error correction segments rather than just the final numerical answer.

---

3. Engineering Workflow for Data Reuse and Synthetic Data

A relatively complete data reuse pipeline typically involves the following stages:

3.1 Source Model Output Collection

Within legal data usage boundaries, batch inference is used to obtain the source model’s outputs for target tasks. During collection, it is necessary to clearly define which fields to retain, such as:

• Input question
• Complete source model output
• Reasoning step segmentation results
• Intermediate scores or confidence levels
• Final answer
• Metadata (task type, difficulty level, data source, etc.)

The more complete the fields, the greater the flexibility for subsequent cleaning and filtering. However, excessive fields will significantly increase storage and processing costs, so they are typically tailored to specific task requirements.

3.2 Trajectory Parsing and Structuring

Raw text must be converted into structured records. For example, a multi-step response can be split into an ordered list of steps:

Raw Output:
Step 1: Analyze known conditions...
Step 2: Define variables...
Step 3: Establish equations...
Step 4: Solve and verify...
Structured Result:
[
  {"step": 1, "type": "analysis", "content": "Analyze known conditions"},
  {"step": 2, "type": "definition", "content": "Define variables"},
  {"step": 3, "type": "equation", "content": "Establish equations"},
  {"step": 4, "type": "solve", "content": "Solve and verify"}
]

The difficulty here lies in the vast differences in reasoning steps across various task types; generic parsing rules often fail to cover all scenarios. In practice, a combination of rule-based logic and lightweight classification models is commonly used to achieve structuring.

3.3 Quality Filtering and Debiasing

Not all source model outputs are suitable for reuse. Quality filtering generally evaluates the following dimensions:

• Final Result Correctness: For tasks with deterministic answers, results can be directly verified.
• Reasoning Logic Consistency: Checking for obvious step-skipping, contradictions, or circular derivations in intermediate steps.
• Expression Standardization: Identifying excessive omissions, mixed symbols, or formats detrimental to target model learning.
• Diversity Control: Preventing a single distribution caused by the repetitive appearance of similar questions.

Debiasing is equally critical. If the source model exhibits systematic errors in a specific sub-domain, direct reuse will amplify that bias. Developers must conduct sampling evaluations to identify sample types where the source model is prone to errors and appropriately reduce the weight of such samples in the training set.

3.4 Training Sample Construction

Filtered trajectories must be converted into training formats usable by the target model. Common practices include:

• Using Only the Final Answer
  Suitable for simple classification, extraction, or generation tasks, but inadequate for scenarios requiring improved reasoning capabilities.

• Using the Complete Reasoning Chain
  Allows the target model to learn the full path from question to answer, suitable for math, logic, and code generation tasks.

• Using Incomplete Trajectory Completion
  Providing only the first few steps and requiring the model to complete the subsequent steps to enhance planning capabilities.

• Using Error-Correction Pairs
  Pairing the source model’s incorrect response with its corrected response to help the target model learn error identification and self-correction.

An illustrative training sample format is as follows:

{
  "instruction": "Calculate the final result based on the following conditions and present the complete reasoning process.",
  "input": "A project completes 1/4 of the total in the first three days, and 2/3 of the remaining part in the next four days, leaving 60 items remaining. Find the total.",
  "reasoning": [
    "Let the total be x.",
    "The first three days complete x/4, leaving 3x/4.",
    "The next four days complete 2/3 of the remainder, which is (3x/4)*(2/3)=x/2.",
    "The first three days and next four days complete a total of x/4+x/2=3x/4.",
    "The remaining is x-3x/4=x/4, and the problem states 60 items are left.",
    "Therefore, x/4=60, x=240."
  ],
  "output": "240"
}

---

4. Key Technical Considerations for the Training Phase

4.1 Mixed Training Distribution

If training relies entirely on source model outputs, the target model may easily overfit the source model’s expression style, even mimicking its reasoning error patterns. Therefore, reasoning trajectory data is typically mixed with human-annotated data, original high-quality text data, and general instruction data. There is no fixed standard for the mixing ratio; it is generally determined through small-scale ablation experiments.

A common baseline configuration is: 40%–60% synthetic trajectory data, 20%–30% human-annotated data, and 20%–30% general text data. This ratio serves only as a starting point and must be dynamically adjusted based on validation set performance.

4.2 Training Objectives and Loss Design

For reasoning trajectory reuse, the training objective cannot focus solely on whether the final answer matches. A more robust approach is to force the model to learn the intermediate steps simultaneously. Step-wise supervision can be adopted, calculating the loss for each reasoning step separately:

loss = α * loss_reasoning + β * loss_final_answer

Here, loss_reasoning measures the discrepancy between the model’s predictions on reasoning steps and the source trajectory, while loss_final_answer measures the discrepancy between the final output and the target answer. The weights α and β can be adjusted based on the task’s requirement for process versus outcome.

4.3 Preventing Catastrophic Forgetting

During continuous training or fine-tuning, if new data occupies too high a proportion, the model may forget its original capabilities. To mitigate this, a replay strategy is commonly employed, retaining a certain proportion of old task samples within the new training data. Concurrently, general capability evaluation metrics should be monitored to detect capability degradation promptly.

4.4 Training Stability Control

Reasoning trajectory samples are typically longer than standard text, resulting in a long-tail sequence length distribution. During training, attention should be paid to the following:

• Setting a reasonable maximum sequence length to avoid Out-Of-Memory (OOM) errors.
• Truncating or segmenting overly long samples.
• Configuring gradient accumulation for samples of varying lengths to maintain training stability.
• Observing the loss curve for periodic fluctuations; if necessary, decrease the learning rate or increase the warmup steps.

---

5. Evaluation and Iteration

To determine if a data reuse strategy is effective, one cannot solely look at final task scores. It is recommended to establish a multi-dimensional evaluation system:

1. Final Task Accuracy
   Compare the target model’s performance before and after incorporating trajectory data.

2. Reasoning Step Rationality
   Use manual spot-checks or automated evaluation models to determine if the reasoning chain is complete and free of logical breaks.

3. Cross-Task Generalization
   Confirm that the model’s capability improvement is not strictly limited to the task types covered in training.

4. Error Pattern Changes
   Analyze whether model errors have shifted from “random errors” to “more controllable systematic errors,” and assess the difficulty of correcting them.

5. Training Cost-Benefit Ratio
   Compare the additional time spent on data collection, cleaning, and training against the magnitude of performance improvement.

If evaluation results show limited benefits, common optimization directions include: improving source model output quality, refining trajectory filtering criteria, adjusting training mix ratios, increasing error-correction samples, and optimizing loss weights.

---

6. Engineering Best Practices

First, data sources must be compliant. Developers should ensure they have the right to use the source model’s output data and adhere to relevant terms of use and data management requirements.

Second, avoid directly copying proprietary formats or templates from source model outputs. Standardize them into the team’s internal format to reduce future maintenance burdens.

Third, different batches of trajectory data may have version discrepancies. It is advisable to log the source model version, inference parameters, and sample size used during each collection for traceability.

Fourth, trajectory data is not suitable for all tasks. For open-ended creative tasks, over-reliance on existing trajectories may limit the target model’s diversity. In such cases, reduce the proportion of trajectory samples or utilize only the structural planning segments.

Fifth, small, iterative steps are better than massive overhauls. Validate the complete pipeline on a small scale before gradually expanding to more complex training scenarios.

---

7. Conclusion

Reusing reasoning trajectories and synthetic data generated by existing models is a pragmatic and highly efficient method in current model iteration pipelines. Its core is not simple replication, but rather transferring the implicit reasoning capabilities of the source model to the target model through structuring, filtering, mixed training, and rigorous evaluation. When implementing this, developers should focus on solving four main challenges: trajectory parsing, quality debiasing, training stability, and capability degradation control. Furthermore, establishing multi-dimensional evaluation metrics is crucial to continuously validate the strategy’s effectiveness. Only through these rigorous practices can data reuse serve as a reliable pivot for enhancing model capabilities, rather than a hidden risk for introducing bias.