Full-Stack Open-Source LLM Pipeline: A Deep Dive from Pretraining Data to Reinforcement Learning

    This article examines a technical release that opens the entire model flow of a large language model family, from pretraining corpora and data-cleaning pipelines to training frameworks, intermediate checkpoints, post-training data, and reinforcement learning infrastructure. It covers the design of 7B and 32B model variants, key data-engineering decisions such as quality-aware upsampling, scientific PDF parsing, midtraining tradeoffs, and decontamination, as well as a three-stage SFT-DPO-RLVR post-training recipe. Readers will learn how the release reframes open-source models as reproducible production pipelines and what practical lessons it offers for data strategy, training orchestration, post-training, and evaluation governance.

    Full-Stack Open-Source LLM Pipeline: A Deep Dive from Data to Reinforcement Learning

    Introduction: Why “Weights Only” Is Not Enough

    The open-source wave in large language models has benefited many developers, but a long-overlooked problem remains: most so-called “open-source models” release only the final model weights and a brief technical report. For researchers and engineering teams that want to reproduce, improve, or study model behavior in depth, this “half-finished open source” has fundamental limitations:

    • Not traceable: When a model excels or fails in a certain capability, it is difficult to determine whether the cause lies in data, training strategy, or algorithm.
    • Not reproducible: Without training data mixtures, hyperparameter details, and intermediate checkpoints, reproduction costs are extremely high.
    • Difficult to study: Research on interpretability, safety, and data contamination requires information from the complete training process.

    The technical approach discussed in this article is centered on releasing a complete “Model Flow”—that is, all assets from pretraining corpora, data-cleaning pipelines, training frameworks, and intermediate checkpoints at each stage to post-training data and reinforcement learning frameworks. This redefines “open-source model” as “the complete production process of an open-source model” and has paradigm-level significance for the industry.

    1. Overall Design of the Model Family

    The release provides a model family spanning 7B and 32B parameter scales, divided into four variants by purpose, forming a clear product matrix:

    1. Base: A pure pretraining artifact that serves as the foundation for all downstream capabilities.
    2. Think: The flagship version, with long chain-of-thought reasoning capability, aimed at mathematics, code, and complex question-answering scenarios.
    3. Instruct: Aimed at general dialogue and instruction-following scenarios.
    4. RL-Zero version: An experimental version that skips supervised fine-tuning and performs reinforcement learning directly from the base model, used to study the open question of whether “pure RL can elicit reasoning capabilities.”

    This “one family, multiple variants” design is worth learning from: it allows researchers to compare the effects of different post-training paths on the same base, turning the “training recipe” itself into a scientific object that can be experimented on and ablated, rather than a black-box artifact.

    2. Base Model Training: Three Key Data-Engineering Decisions

    2.1 Quality-Aware Upsampling Outperforms Flat Filtering

    The most common approach in pretraining data processing is “flat filtering”: setting a quality-score threshold and discarding documents below it. Experiments in this release show that this approach is not optimal in data-constrained scenarios.

    A better strategy is quality-aware upsampling: instead of a hard cutoff, build a sampling-weight curve based on quality scores—high-quality documents are sampled multiple times, while low-quality documents have a reduced sampling probability but are not completely removed. The intuition is:

    • A hard threshold creates many “false negatives” near the threshold, losing data diversity.
    • Low-quality data still contains scarce domain knowledge (such as niche technical documentation), and completely deleting it can lead to missing long-tail capabilities.
    • A smooth sampling curve is equivalent to “soft shaping” the data distribution, achieving a better Pareto frontier between quality and diversity.

    The engineering implication is: when teams face a decision about “cleaning intensity,” they should prioritize weighted sampling over binary filtering, and determine the shape of the upsampling curve through small-scale ablation experiments.

    2.2 Structured Extraction of Large-Scale Scientific Documents

    Another data innovation in this release is the use of a self-developed document-parsing pipeline to convert massive scientific PDFs into high-quality text. PDFs are a primary carrier of academic knowledge, but their layouts are complex (two-column text, formulas, tables, figures), and directly extracted text is often messy and traditionally considered “unusable.”

    Through vision-language model-assisted layout parsing, the team converted hundreds of millions of pages of scientific literature into well-structured training corpora, significantly enhancing the model’s capabilities in scientific question answering and professional reasoning. This practice shows that the “ceiling” of a corpus is not determined by crawl volume but by parsing quality. For enterprises building internal corpora (such as internal technical documentation, contracts, and research reports), investing in document-parsing infrastructure often yields higher returns than simply expanding crawl scale.

    2.3 Domain Tradeoffs in Midtraining

    After pretraining (about 5.9T tokens) and before post-training, the release inserts a midtraining stage of about 100B tokens, specifically strengthening code, mathematics, and question-answering capabilities. This stage reveals an important engineering pattern—domain tradeoffs:

    • Increasing the mixture weight of mathematics and code data does significantly improve corresponding benchmark scores.
    • But the cost is an observable decline in general question answering (GenQA) and multiple-choice question answering (MCQA) performance.
    • The final data mixture is a compromise found after extensive ablation experiments, not a theoretical optimum.

    This finding has cautionary value for all teams doing domain-adaptation training: midtraining is not a “free capability boost” but a reallocation of capabilities. The recommended approach is to clarify the capability priorities of the target scenario, build an evaluation panel covering multiple domains, scan the mixture space with small-scale experiments, and then launch formal training.

    2.4 Unexpected Benefits of Front-Loading Post-Training Data

    An interesting finding is that mixing a small amount of instruction-following and chain-of-thought format data into the midtraining stage—even before the model has undergone any supervised fine-tuning—improves the base model’s performance across various benchmarks. A possible explanation is that this kind of data helps the model establish a “question-answer” format prior during pretraining, reducing the alignment cost of later post-training.

    2.5 The Complexity of Decontamination

    The release’s candid disclosure of decontamination issues is particularly valuable. The analysis found that benchmark leakage is widespread in many commonly used open-source instruction datasets—variants of many test questions have long been mixed into training corpora. After strict decontamination, scores on some metrics (such as reading-comprehension benchmarks) dropped, but this precisely shows that earlier “high scores” partly came from memorization rather than capability.

    This reminds the industry: the credibility of evaluation numbers is positively correlated with the strictness of decontamination. When evaluating models internally, enterprises should prioritize self-built or private evaluation sets to avoid being misled by the “score-chasing effect” of public benchmarks.

    3. Three-Stage Post-Training Recipe for the Reasoning Model

    The flagship reasoning model uses a three-stage post-training pipeline of SFT → DPO → RLVR, and each stage has technical details worth unpacking.

    3.1 Stage 1: Supervised Fine-Tuning (SFT)

    High-quality long chain-of-thought data is used for cold-start fine-tuning of the base model, teaching it the output format of “think first, then answer.” The key in this stage is the quality and diversity of chain-of-thought data—reasoning steps must be logically coherent and cover multiple reasoning modes such as mathematics, code, science, and common sense.

    3.2 Stage 2: Preference Optimization with Delta Learning (DPO)

    The construction of preference data is one of the highlights of this approach. The traditional method asks annotators or strong models to directly judge “which answer is better,” which is costly and noisy. The Delta Learning idea adopted here is to construct paired samples so that the differences between preference pairs precisely correspond to changes in a desired behavior (such as more rigorous reasoning or less redundancy), thereby focusing preference learning on capability improvement along a single dimension rather than a vague “good or bad judgment.” This controlled preference construction method significantly improves the signal quality of the DPO stage.

    3.3 Stage 3: Reinforcement Learning with Verifiable Rewards (RLVR)

    RLVR (reinforcement learning with verifiable rewards) is the core stage for improving reasoning capabilities: for tasks with deterministic answers such as mathematics and code, programmatic verifiers (answer matching, unit tests) are used as reward signals, replacing reward models that are costly and vulnerable to “reward hacking.” The accompanying reinforcement learning framework makes several improvements in algorithm and engineering:

    • Removing the KL divergence penalty term: Traditional RLHF uses a KL term to constrain the policy from deviating too far from the reference model, but this limits the model’s space to explore new reasoning patterns; after removal, combined with other stabilization methods, it instead achieves stronger reasoning capabilities.
    • Introducing token-level loss calculation: Credit assignment is refined to the token granularity, alleviating gradient variance problems in long chain-of-thought scenarios.
    • Infrastructure optimization: Resource scheduling for inference generation and training is decoupled, improving throughput efficiency for large-scale rollouts.

    These details have direct reference value for enterprises building their own RL training pipelines: the upper bound of RL gains is determined by the algorithm, but whether it can run and run stably is determined by infrastructure.

    4. Asset List for Full-Stack Data Release

    The data assets released in this approach cover every stage of the model lifecycle, forming a reusable “data recipe”:

    Stage Dataset Type Purpose
    Pretraining General mixed corpus Build language and knowledge foundation
    Midtraining Domain-enhanced mixture Strengthen code, mathematics, and QA
    Long-context extension Long-document mixture Expand context-window capability
    Post-training Instruction, preference, and chain-of-thought series Shape alignment and reasoning capabilities

    Together with the released training framework, intermediate checkpoints, and complete training logs, third-party researchers can “enter” at any stage—for example, starting from a midtraining checkpoint for vertical-domain adaptation, or reusing post-training data to train their own base models.

    5. Industry Implications and Practical Recommendations

    This full-stack open-source approach offers several transferable lessons for the industry:

    1. Data strategy: Replace hard filtering with quality-weighted soft sampling, and trade parsing quality for a higher corpus ceiling.
    2. Training orchestration: Acknowledge domain tradeoffs in midtraining, use an evaluation panel to drive mixture decisions, and consider front-loading post-training data.
    3. Post-training: Improving reasoning capabilities depends on a combined approach of “SFT format foundation + controlled preference construction + verifiable-reward RL”; a single stage cannot replace the whole.
    4. Engineering governance: Strict decontamination and public training logs are prerequisites for building evaluation credibility.

    For teams that want to reproduce similar pipelines on their own compute, they can build training pipelines on general-purpose cloud container platforms, object storage, and distributed training scheduling capabilities, focusing their effort on data mixtures and algorithm iteration rather than repeatedly building underlying infrastructure.

    Conclusion

    “Open weights” make a model usable, while “open model flow” makes a model understandable, reproducible, and improvable. This approach demonstrates that every key decision in large-model training—from data sampling curves to RL loss design—can be systematically disclosed and verified. This not only lowers the barrier to large-model research but also sets a new transparency benchmark for the industry.