Fully Open-Source LLM Technical Deep Dive: Transparent Practices from Data to Weights
This article explores the paradigm shift from open weights to a fully open "model flow" in large language models (LLMs). It examines critical technical decisions in base model training, such as quality-aware upsampling, mid-training domain trade-offs, and data decontamination. Additionally, it details post-training recipes for reasoning models—including SFT, DPO, and RLVR—and offers practical engineering insights for teams aiming to reproduce or adopt these transparent methodologies.
1. The Paradigm Shift from “Open Weights” to “Open Model Flow”
Traditional open-source model releases typically include only the final weights and basic inference code. While this allows the community to perform fine-tuning or deploy inference, it fails to answer critical questions like “Why does the model perform well or poorly?” and “How does the data impact its capabilities?”
A new generation of fully open-source paradigms emphasizes the release of a complete “model flow,” covering the assets of the entire training lifecycle:
- Datasets: Pre-training mixed data, mid-training data, long-context data, and post-training data.
- Code: Data preprocessing, model construction, and reinforcement learning training frameworks.
- Intermediate Checkpoints: Snapshots from different pre-training stages, useful for studying training dynamics and the impact of learning rate scheduling.
- Training Logs: Complete loss curves and records of evaluation metrics tracked across steps.
The value of this transparency lies in reproducibility and researchability. Researchers can validate their improvement hypotheses on the same data flow or conduct annealing experiments and data ablation from public intermediate checkpoints, rather than pre-training from scratch.
2. Key Technical Decisions in Base Model Training
1. Quality-Aware Upsampling over Flat Filtering
In pre-training data processing, a common practice is to set a threshold based on quality scores to directly truncate data. However, experiments show that in data-constrained scenarios, flat filtering is not the optimal solution.
A more effective strategy is quality-aware upsampling: instead of discarding low-scoring data, an upsampling curve is constructed based on quality scores, allowing high-quality data to be sampled at higher frequencies and low-quality data to participate in training at lower frequencies.
The underlying logic is that pre-training requires broad knowledge coverage. Simple truncation loses long-tail domain information (e.g., low-resource languages, specialized literature). Upsampling retains diversity while directing gradient updates more toward high-quality samples, balancing coverage and data quality.
2. Domain Trade-offs in Mid-Training
Mid-training is a stage inserted at the end of pre-training and before post-training, typically focusing on intensive training for code, math, and QA tasks. Public training logs reveal an important domain trade-off phenomenon:
- Increasing the weight of math and code data leads to significant improvements in corresponding benchmark scores.
- However, general QA and multiple-choice QA capabilities simultaneously decline.
This reflects the trade-off between different capabilities. Therefore, the core engineering challenge in mid-training data proportioning is not “maximizing a certain metric” but finding a balance between general and specialized capabilities. This is also why data mixture research still relies heavily on extensive ablation experiments.
3. Benefits of Early Post-Training Data Introduction
A counterintuitive yet important finding: introducing instruction-following data and Chain-of-Thought (CoT) data early during the mid-training stage can comprehensively improve the base model’s performance across benchmarks, even without subsequent Supervised Fine-Tuning (SFT).
This indicates that “instruction formats” and “reasoning formats” can be learned as part of the pre-training distribution. This provides a better initialization for the subsequent post-training stages, reducing both the difficulty and data requirements for post-training.
4. Complexity of Data Decontamination
Benchmark leakage is a widespread issue in open-source datasets. Decontamination presents a dilemma:
- Without decontamination, evaluation scores are artificially inflated, losing their reference value.
- With strict decontamination, scores on certain metrics drop, making it look like a regression.
The honest approach is to disclose the decontamination strategy and accept the score changes, providing an evaluation closer to the model’s true capabilities. For users, when comparing different models, it is crucial to check whether a unified, clean decontamination benchmark is used.
3. Post-Training Recipes for Reasoning Models
Flagship reasoning models typically adopt a three-stage post-training workflow:
- SFT (Supervised Fine-Tuning): Establishes foundational behavioral patterns using high-quality instructions and long CoT data.
- DPO (Direct Preference Optimization): Aligns response style and preferences through pairwise preference data.
- RLVR (Reinforcement Learning with Verifiable Rewards): For tasks that can be automatically verified (like math and code), uses answer correctness as a reward signal for reinforcement learning.
Incremental Learning for Preference Data Construction
When building high-quality preference data, an effective approach is incremental construction: find samples that the base model got wrong but answered correctly after intervention, and use the pre- and post-intervention outputs as a preference pair. Data constructed this way is highly targeted with clear signals, avoiding the noise issues associated with manual preference annotation.
RLVR Algorithm and Infrastructure Optimizations
At the reinforcement learning implementation level, public engineering experiences include:
- Removing the KL Divergence Constraint: In traditional RLHF, the KL penalty is used to prevent policy drift. However, in verifiable reward scenarios, correctness itself acts as a strong constraint. Removing the KL term can unlock greater exploration space.
- Introducing Token-Level Loss: Allocating sequence-level reward signals to a token granularity mitigates the credit assignment problem, ensuring that every decision step in a long reasoning chain receives more fine-grained gradient signals.
These improvements indicate that RL designed for verifiable tasks is diverging from traditional human-preference RLHF in algorithm design.
4. Engineering Insights and Practical Recommendations
For teams looking to reproduce or leverage this methodology, the following key points can be extracted:
- Data Proportioning is the Primary Lever: With fixed computing power, the returns from data mixing strategies often outweigh those of model architecture fine-tuning.
- Build an Auditable Data Pipeline: The end-to-end data sources, cleaning rules, and decontamination strategies should be fully traceable.
- Evaluate Honestly: Decontamination and multi-benchmark cross-validation should be standard procedures.
- Progressively Open-Source Checkpoints: Even without releasing all data, publishing intermediate checkpoints provides immense research value to the community.
- Modularize Post-Training Pipelines: Decouple the SFT, DPO, and RLVR stages to facilitate independent ablation and iteration.
Conclusion
The fully open-source model flow paradigm transforms large model training from a “black-box result” into a “reproducible scientific experiment.” It brings not just weights, but a complete set of validated data ratios, training recipes, and engineering decision logs. For teams engaged in LLM R&D, carefully studying the ablation conclusions and trade-offs in such comprehensive training logs often holds more long-term value than chasing a single benchmark leaderboard.
