How to Build a Fully Open LLM: Technical Practices Across Pretraining, Midtraining, and Post-Training
A practical walkthrough of the "Model Flow" approach to open large models, where the goal is to ship a complete, traceable production pipeline instead of a weights file alone. It covers the engineering decisions that matter at each stage of the training lifecycle: quality-aware upsampling and token-constrained domain mixing during pretraining, structured document parsing for PDFs, the trade-offs and pre-injection benefits of a midtraining phase, the three-stage SFT → DPO → RLVR post-training recipe, RL framework improvements such as dropping the KL penalty and moving to token-level loss, and benchmark decontamination as a first-class part of the data pipeline. It closes with a checklist engineering teams can apply to keep their training process reproducible and auditable.
Introduction: Why “Fully Open” Is Worth More Than “Weights Only”
Over the past few years the number of open large models has grown explosively, yet the vast majority publish nothing but the final weights. That practice leaves a long-standing pain point: what outsiders receive is a black-box artifact. They cannot reproduce the training process, and they cannot tell whether a given capability comes from the data, the algorithm, or engineering tuning.
The “Model Flow” idea reframes the goal: instead of delivering a weight file, deliver a complete, traceable production pipeline. In practice it usually contains several layers:
- Final weights: multiple derivatives, such as base, instruction-following, and reasoning variants;
- Full-pipeline data: pretraining corpora, midtraining data, and post-training data, together with their mixture ratios and sampling strategies;
- Training code: the data pipeline, the training framework, and the reinforcement learning framework;
- Intermediate artifacts: key checkpoints from the training run, learning curves, and full training and evaluation logs.
For developers, full openness means you can pinpoint exactly which stage produced a capability gain or a regression. For enterprises, it means you can run continued pretraining or domain fine-tuning on your own data without worrying that the foundation is out of your control. The rest of this article breaks the methodology down along the training lifecycle and highlights the practices most worth borrowing.
Pretraining: The Data Distribution Sets the Capability Ceiling
The central question of pretraining is not “how many tokens did we feed in” but “which tokens did we feed, and in what proportions.” The widely accepted industry experience is that the marginal return of high-quality data far exceeds simply piling on volume.
1. Quality-Aware Upsampling Beats Flat Filtering
Conventional data cleaning relies on flat filtering: set a quality-score threshold and discard every sample below it. The problem is that this decision is binary — a sample is either kept entirely or thrown away entirely — and it ignores the fact that high-quality samples should be seen several more times.
A more refined approach is quality-aware upsampling:
- Score the corpus with a classifier or a set of rules to obtain a continuous quality distribution;
- Skip the hard cutoff and instead build a curve in which higher quality maps to more repeated sampling;
- When the total data budget is constrained, allocate training budget preferentially to high-value samples.
It can be understood as a form of weighted sampling:
# Illustration: the core logic of quality-aware upsamplingdef build_sampling_weights(doc_scores, base_epoch=1.0, max_repeat=4.0):"""doc_scores: quality score of each document (0~1)returns the relative sampling weight of each document within one training pass"""weights = []for s in doc_scores:# map linearly (or piecewise) to [base_epoch, max_repeat]w = base_epoch + (max_repeat - base_epoch) * sweights.append(w)return weights
In practice, the shape of this curve — linear, exponential, or piecewise constant — has to be settled with small-scale ablation experiments. The key insight is that under data constraints, flat filtering wastes high-quality samples that are already scarce, while an upsampling curve substantially improves cost-efficiency.
2. Token-Constrained Domain Mixing
Pretraining corpora are typically a blend of web pages, code, math, academic literature, multilingual text, and more. Domain ratios look simple but are in fact a chain of trade-offs. A common engineering technique is token-constrained mixing: under a fixed total token budget, dynamically adjust the share of each domain that actually enters the training loop, rather than naively stacking raw document counts. This avoids the bias where short-document domains get diluted and long-document domains get over-represented.
3. Parsing and Reusing Structured Documents
A great deal of high-value knowledge lives in PDFs — scientific papers, textbooks, patents. Their value is extremely high, but their raw form is hostile to the model: two-column layouts, formulas, figures, and reference lists severely degrade text extraction quality.
This is why a dedicated document-parsing stage is needed, one that converts a PDF back into structured plain text: identifying body text versus headers and footers, restoring paragraph order, and preserving formula and list semantics wherever possible. The quality of this step often directly determines the ceiling on scientific-reasoning tasks, and it is the most typical illustration of the “garbage in, garbage out” principle.
Midtraining: Rebalancing Capabilities and Injecting Format Priors Early
Between pretraining and post-training, you can insert a midtraining stage: a relatively small, carefully proportioned dataset — usually on the order of hundreds of billions of tokens — used to reinforce the base model’s capabilities. It is cheaper than a full pretraining run, yet has a deeper and longer-lasting effect than post-training.
1. Domain Trade-offs: Not Every Gain Is a Net Gain
The most common pitfall in midtraining is a local optimum that degrades the whole. The typical pattern: after increasing the weight of math and code data, math and code benchmark scores rise noticeably, while general QA (GenQA) and multiple-choice QA (MCQA) regress. The reason is that the model’s capacity and attention are being redistributed — gaining one capability often comes at the cost of another.
So the real objective of a midtraining data mixture is to find a Pareto frontier: an overall optimal balance among code, math, and general QA, rather than topping out on any single metric. This requires a multi-dimensional evaluation matrix and a full evaluation for every candidate mixture.
2. The Upfront Payoff of Post-Training-Style Data
A counterintuitive but important finding: if instruction-following and chain-of-thought (Thinking) format data are introduced early, during midtraining, the base model improves across all kinds of benchmarks — even if no subsequent supervised fine-tuning is done at all.
This suggests that instruction and reasoning formats are not things the model can only learn in post-training; they can be internalized much earlier as an organic part of the pretraining distribution. The lesson for engineering teams is not to think about alignment only when post-training begins — a “format prior” in the data should be injected as early as possible.
Post-Training: A Three-Stage SFT → DPO → RLVR Recipe
Post-training is what turns a base model that “can continue writing text” into an assistant that can converse, reason, and be aligned. The recipe most widely validated today is three stages in sequence.
Stage 1: Supervised Fine-Tuning (SFT)
Train the model on high-quality instruction-response pairs so it learns the conversational format and follows tasks. The quality and diversity of data at this stage directly determine the ceiling for every stage that follows.
Stage 2: Preference Optimization (DPO and related methods)
Direct Preference Optimization (DPO) aligns with human preference using paired preference data — a “better answer” and a “worse answer” for the same prompt — without explicitly training a separate reward model, which keeps the engineering relatively lightweight.
One approach worth borrowing here is Delta Learning-style preference data construction: rather than annotating massive preference datasets from scratch, it builds preference pairs through targeted differential corrections on existing answers, yielding a more efficient signal under a limited annotation budget.
Stage 3: Reinforcement Learning with Verifiable Rewards (RLVR)
For tasks where answers can be verified — math, code — apply reinforcement learning with verifiable rewards (RLVR). Unlike conventional RLHF, which depends on human preference scores, RLVR takes its reward from objective verifiers: whether code passes unit tests, whether a math answer is correct. This verifiable signal is more stable and less prone to reward hacking, and it is a key ingredient in improving reasoning ability.
Engineering Improvements in the RL Framework
Beyond algorithm choice, the engineering implementation of reinforcement learning has a large effect on final quality and training stability. Two categories of framework-level improvements currently being explored in the community are especially worth attention:
Removing the KL divergence constraint term: traditional RLHF uses a KL penalty to pull the policy back toward a reference model and prevent it from drifting too far. In an RLVR setting, however, the verifiable reward already provides a reliable anchor, and dropping the KL term actually frees up the model’s exploration space instead of causing it to converge early onto a conservative policy.
Introducing token-level loss: compared with assigning a single advantage at the sequence level, token-level loss models the contribution of each token at a finer granularity, which helps mitigate inaccurate advantage estimation and diluted gradients in long sequences.
The shared goal of these changes is a better balance between fully exploiting the verification signal and keeping training stable.
Data Decontamination: An Underrated Question of Honesty
Benchmark contamination is a persistent problem in the current open ecosystem. Questions and answers from evaluation sets can slip silently into pretraining or fine-tuning corpora through web crawling, instruction datasets, and similar channels. The result is a model that scores high on leaderboards while having effectively memorized the answers.
A common observation: even in public instruction datasets that appear clean, large numbers of samples overlapping with mainstream benchmarks can still be detected. In other words, decontamination is not optional — it is a necessary condition for evaluation results to be trustworthy.
The cost of decontamination has to be faced squarely: after cleaning, some metrics that depend on “memorization” (extractive QA, for instance) will drop. That is precisely a good thing, because it strips away inflation and restores a picture of the model’s true generalization ability. Practical recommendations:
- Before data enters training, compare it against known evaluation sets using n-gram overlap and near-duplicate detection;
- Establish and publish the contamination-detection process so that anyone reproducing the work can audit it;
- Report both “before cleaning” and “after cleaning” numbers when publishing evaluations, and stay transparent.
Practical Recommendations for Engineering Teams
Taken together, building a reproducible, auditable large-model training pipeline can start from a handful of practices:
- Full-process traceability: preserve not only the final weights but also the data mixture, code versions, intermediate checkpoints, and complete logs, so every step can be traced back;
- Data first: put budget into quality-aware upsampling and domain-mix tuning ahead of blindly scaling token counts;
- Staged capability management: use midtraining to rebalance capabilities, and search a multi-dimensional evaluation matrix for the Pareto optimum across code, math, and general ability;
- Inject format priors early: bring instruction and chain-of-thought format data in as soon as possible to lighten the post-training load;
- Layered post-training: let SFT establish the format foundation, DPO align preferences, and RLVR tackle verifiable tasks;
- Routine decontamination: make contamination detection part of the standard data pipeline, and report evaluation results transparently.
The core conclusion is that a large model’s competitiveness comes less and less from a single breakthrough trick and more and more from systematic coordination across data, algorithms, and engineering. The real significance of fully open models lies in making that coordinated process visible, learnable, and reproducible — which is both a gift to researchers and a practical path for enterprises building autonomous, controllable model capabilities.
