Full-Stack Open-Source LLMs: Transparent Training from Data to Weights
Releasing only final model weights makes large language models difficult to reproduce, audit, and study. This article examines a full-stack open-source approach that publishes the complete model flow: raw corpora, preprocessing code, pretraining and midtraining data mixtures, post-training data, implementation code, intermediate checkpoints, and training logs. It covers data engineering in pretraining, PDF and scientific document processing, midtraining, SFT/DPO/RLVR pipelines, layered data releases, and the implications for practitioners and enterprises.
Full-Stack Open-Source LLMs: Transparent Training from Data to Weights
1. Why Releasing Only Model Weights Is No Longer Enough
Over the past few years, the open-source large model ecosystem has developed rapidly. However, for most projects, “open source” has meant releasing only the final model weights and a relatively brief technical report. The problems with this approach are obvious:
- Limited reproducibility. If any part of the data mixture, cleaning rules, or training hyperparameters is missing, third parties may be unable to reproduce a model of the same quality. Discussions about open models whose results cannot be aligned with paper conclusions are common in the community.
- Difficult auditing. A model may contain hidden data contamination, such as evaluation sets leaking into training data, copyright risks, or biased content. With only weights and no visibility into the data, these issues are nearly impossible to trace.
- Barriers to research. Training dynamics, the impact of data on model capabilities, and the “capability emergence” process revealed by intermediate checkpoints are all valuable research materials. A closed training process makes such research impossible.
As a result, the core idea behind a new generation of open-source projects has shifted: instead of releasing only the final artifact, they release the complete Model Flow—from raw corpora, preprocessing code, pretraining data mixtures, midtraining data, and post-training data (SFT/DPO/RL) to the code for each stage, intermediate checkpoints, and training logs.
2. Pretraining: Data Engineering Is the Real Differentiator
2.1 Large-Scale Pretraining and Data Mixtures
The base model in this series was pretrained on a corpus of approximately 6T tokens. Notably, the era of simply “stacking more data” is over. The key to contemporary pretraining lies in data mixture strategy:
- Constrained mixture ratios: Target proportion ranges are set for different sources such as web pages, code, mathematics, and academic papers. During training, the actual consumption of data in each domain is monitored dynamically to avoid prematurely exhausting a data type or letting its proportion drift out of control.
- Quality-aware upsampling: High-scoring subsets after quality filtering are repeatedly sampled (upsampled). Intuitively, reusing data may seem risky, but practice shows that learning high-quality data for 2–3 epochs usually yields more benefit than learning an equivalent amount of low-quality data for only one epoch. The key is controlling repetition rates and deduplication quality.
2.2 Document-Level Processing: PDFs and Scientific Literature
A significant investment in pretraining data involves deep processing of PDF documents. Academic papers, textbooks, and technical reports often exist as PDFs, and direct text extraction frequently produces disordered layouts such as broken formulas, misaligned tables, and interleaved two-column text. The project built a dedicated OCR and document parsing pipeline to convert large volumes of scientific PDFs into well-structured text that preserves formula and table semantics. This is a rare investment among open-source models and an important data source for mathematical and scientific capabilities.
2.3 Midtraining: Injecting Capabilities Between Pretraining and Post-Training
After main pretraining, the model undergoes a midtraining stage of approximately 100B tokens, with targeted reinforcement of code, mathematics, and question-answering data. The logic behind this design is:
- Main pretraining pursues breadth and language ability, with relatively balanced mixture ratios.
- Midtraining focuses capability after the model has acquired basic abilities, using a higher proportion of “reasoning-intensive” data for targeted adjustment near the convergence point.
- Midtraining is usually accompanied by learning rate annealing and context length extension, laying the foundation for later long-context capabilities and post-training.
This two-stage structure of “main training + midtraining” is becoming a standard recipe adopted by both open-source and closed-source models.
3. Post-Training: From Speaking to Reasoning
The flagship reasoning model uses a classic three-stage post-training pipeline:
3.1 Supervised Fine-Tuning (SFT)
High-quality long chain-of-thought demonstration data is used to teach the model to “write out its reasoning process.” The quality of demonstration data directly determines the stability of the reasoning format.
3.2 Preference Optimization (DPO)
After SFT, direct preference optimization is performed using preference pair data (chosen/rejected pairs). The project discloses a data construction idea worth learning from: Delta Learning. Rather than broadly labeling “which answer is better,” it builds preference pairs based on the model’s own incorrect and correct outputs on specific questions. This makes the preference signal point precisely to “specific defects in the reasoning process,” such as a calculation error at a certain step or a missing assumption, rather than general stylistic differences. This fine-grained preference signal can significantly improve the model’s basic reasoning accuracy before the RL stage.
3.3 Reinforcement Learning with Verifiable Rewards (RLVR)
The final stage uses reinforcement learning with tasks whose answers can be verified, such as math problems and code unit tests, as reward signals. The project’s RL framework makes several algorithmic changes worth noting:
- Removing the KL divergence penalty: Traditional RLHF relies on KL constraints to prevent the policy from deviating from the reference model. However, in scenarios where verifiable correctness is the reward, KL constraints can limit the model’s room to explore better reasoning paths. After removal, combined with other stability measures, the model can achieve greater performance gains on math and code tasks.
- Token-level loss aggregation: Loss aggregation is changed from “averaging by sequence” to “aggregating by token,” avoiding long responses being diluted in the gradient and making learning signals from long chain-of-thought samples more sufficient.
At the infrastructure level, asynchronous sampling and training pipelines, rolling weight synchronization, and fine-grained hierarchical reward validation (format reward + correctness reward) are general engineering practices for ensuring efficient and stable RL-stage operation.
4. Full-Stack Data Release: A Valuable Training Specimen
The data release is also complete and layered:
- Pretraining mixture data: Multi-source corpora after cleaning, deduplication, and classification.
- Midtraining data: Focused data packages with higher proportions of code, mathematics, and QA.
- Long-context data: Document-level long samples used in the context length extension stage.
- Post-training data series: SFT demonstration data, preference pair data, and RL verification task data.
Together with intermediate checkpoints released by stage, researchers and engineers can observe how model capabilities evolve as training progresses—for example, at which stage code ability takes shape and which types of data mathematical ability is most sensitive to. This “training dynamics”-level information was previously produced by only a very small number of organizations.
5. Implications for Practitioners and Enterprises
- A much lower reproducibility barrier: Small and medium-sized teams can use public data recipes and code to reproduce or fine-tune domain models with limited compute, instead of groping in the dark during data cleaning.
- Data engineering matters more than model architecture: This project clearly shows that the competitiveness of contemporary large models is reflected more in data mixture, quality filtering, upsampling strategies, and multi-stage curriculum design. Architectural innovation is comparatively incremental.
- Verifiable rewards are a key path to reasoning capability: The three-stage recipe of SFT for format, DPO for defect correction, and RLVR for raising the ceiling, combined with carefully constructed incremental preference data, is a repeatedly validated training paradigm for reasoning models.
- Cloud implementation touchpoints: For enterprises that want to implement similar capabilities in the cloud, object storage for multi-source corpora, container platforms for distributed training, logging and monitoring services for tracking training dynamics, and model serving for hosting inference workloads form a common technology stack combination.
6. Conclusion
The significance of full-stack open source is not merely “releasing a few more datasets.” It turns large model training from a black box into an open scientific practice that is auditable, reproducible, and researchable. When data, code, checkpoints, and logs are all laid out in the open, the community’s knowledge gain upgrades from “another usable model” to “a complete training methodology.” For developers, carefully studying such full-stack technical reports and data recipes is one of the most efficient ways to understand frontier large model training practices today.
