Design Points for a Fully Open LLM Training Pipeline
Open weights alone do not explain where model capability comes from. This article describes a practical “model flow”: what to publish across pretraining, mid-training, and post-training, and how quality-aware sampling, domain mix, decontamination, and a three-stage post-training stack make results reproducible and auditable.
Open weights are only a starting point. If only the final parameters are released, later researchers cannot tell whether capability came from data, the training recipe, or accidental tuning, and they cannot reproduce the experiment under the same conditions. A more complete approach is to publish the “model flow”: the composition of pretraining and mid-training data, post-training samples, training code, key checkpoints, logs, and evaluation protocols. The design below is organized by training stage and is not tied to any specific model family.
1. The base model: when data is limited, change the distribution before cutting more of it
The bottleneck in large-scale pretraining is often not “run a few more steps,” but the density of useful signal. A common practice is to score documents for quality and then drop samples below a threshold—flat filtering. It is simple to implement, but it has two problems. Samples near the threshold are cut off in one stroke, so medium-quality text that covers rare domains is easy to discard by mistake. If high-scoring samples are kept only once, the model still does not see the high-quality distribution often enough.
Quality-aware upsampling maps a score to a sampling weight instead of a binary keep-or-drop decision. In practice, a piecewise or smooth curve works: down-weight or remove low scores, keep mid scores close to their original frequency, upsample high scores according to marginal gain, and set a cap so a few templates do not dominate. The curve should be chosen on downstream tasks in small proxy experiments, not by optimizing perplexity alone. When data is limited, this reallocation usually improves overall capability more than “filtering one more time,” because it keeps coverage while reinforcing the high-quality tail.
Scientific papers, scans, and PDFs with complex layout cannot be used as plain text directly. They need layout parsing, reading-order recovery, structured formulas and tables, and cleanup of headers, footers, and reference noise. Cleaned specialist text helps reasoning and knowledge density, but parsing errors can scramble formula order and become strong noise. Sample the output for human review, and include parsing confidence in the quality score. Do not assume that “it came from a paper” means it is high quality.
2. Mid-training: an explicit tradeoff between domain gains and general ability
After the main pretraining run and before supervised fine-tuning, a shorter mid-training stage is often used to raise the share of code, math, question answering, and long-context material. By this point the model already has general language ability, so a modest amount of targeted data can change behavior sharply. The mix is more sensitive than continuing to pile on general web text.
The empirical pattern is stable: raising the weight of math and code usually lifts the corresponding benchmarks, while general open-ended QA and multiple-choice QA may fall. Causes include distribution shift, answer format being pulled toward code and derivation text, and factual samples being crowded out of the effective batch. There is no single mix that is optimal for every task. State the target first—a general assistant, code, STEM reasoning, or long documents—then search the Pareto frontier on a proxy set.
A practical mixing rule is:
- Keep a share of high-quality general text to limit catastrophic forgetting.
- Use executable or otherwise checkable subsets for code and math, not unfiltered crawls.
- Separate “grounded answers” from “exam-style multiple choice” so the model does not only learn option letters.
- Use short ablations to record how adding or removing each domain moves key metrics, rather than looking only at the final total score.
Very long sequences should not be forced from the start of pretraining. A steadier path is to stabilize capability at a normal length first, then use a short-to-long curriculum in mid-training, coherent in-document samples, and retrieval-style packing, while checking positional extrapolation, the attention implementation, and the memory strategy together. Long-context data should be continuous spans from real long documents. Random concatenation creates fake long-range dependencies.
3. Move instruction and reasoning data earlier
The traditional pipeline leaves instruction following and step-by-step reasoning entirely to supervised fine-tuning. If mid-training already mixes in a small amount of high-quality instructions, chain-of-thought, and grounded solutions, the base model is often more stable on several benchmarks even before formal alignment. The model has already seen the conditional distribution of “question—reasoning—answer,” so post-training does not have to change the output format from scratch.
Moving data earlier does not mean pouring all post-training data into pretraining. The share should stay low, the style should be diverse, and deduplication and decontamination should be strict. Too much templated instruction data, too early, hurts the diversity of language modeling and can write benchmark answers into the training distribution. The goal of the shift is format and reasoning habits, not memorizing test items.
4. Decontamination: a lower score is often the more trustworthy signal
Benchmark text and near-duplicates are common in instruction collections, synthetic QA, and public practice sets. N-gram overlap, embedding similarity, and string matching after normalization catch some leakage, but paraphrases, translations, and reordered options can slip through. After decontamination, some reading-comprehension and extraction metrics drop sharply. That usually means the earlier score contained memorization, not that the recipe got worse.
When reporting results, give the contamination rules, the fraction removed, the before-and-after difference, and whether the training set and the evaluation set used the same normalization. Do not leave leaked samples in the training set to protect a leaderboard score. For historical data that cannot be fully cleaned, mark the conclusion as “possibly overestimated,” and add held-out items written by the team and published after the training cutoff as supporting evidence.
5. Three-stage post-training: supervision, preference, and verifiable reward
A common post-training order for flagship reasoning models is supervised fine-tuning, then preference optimization, then reinforcement learning with a verifiable reward.
Supervised fine-tuning supplies format, safety boundaries, and basic reasoning traces. The data should be short and carefully chosen, covering direct answers, refusals, tool use, and step-by-step derivations. Chain-of-thought should show intermediate steps that can be checked, not long boilerplate. Overlong, repetitive “thinking” templates get amplified in later reinforcement learning.
The preference stage needs paired or ranked data. Rather than asking annotators only to score by feel, build preferences with “delta learning”: on the same prompt, take an answer the current policy already produces as the reference, keep as the preferred sample only those that clearly improve on that reference, and treat regressions, factual errors, or redundant samples as the dispreferred ones. The preference signal then points at the model’s real weaknesses, not at stylistic differences between humans and the model that do not matter. A delta can come from a correct answer, a shorter correct derivation, consistent citations, or a unit test that passes. Open-ended generation whose correctness cannot be judged is a poor fit for high-confidence preferences.
The third stage should run reinforcement learning only on tasks whose reward can be checked automatically, such as a final math answer, code tests, format constraints, and parseable structured output. The verifier must be independent of the policy model, so the model does not score itself highly. Rewards should be sparse but clean: the test passes, the answer matches, or the constraint is satisfied. A vague sense that the output “looks smarter” should not be the main reward.
Several algorithmic and engineering tradeoffs should be written down as reproducible settings, not slogans:
- A KL constraint toward a reference model can limit drift, but if it is too strong it locks the policy near supervised fine-tuning and leaves too little exploration. If KL is weakened or removed later, replace “trust the reward curve” with accuracy on a validation set and checks for regression.
- A sequence-level reward gives every token the same good-or-bad signal. In long reasoning, errors often sit in a few steps. Token-level or span-level losses, and advantages computed only on newly added reasoning spans, reduce gradient noise on unrelated tokens. They are harder to implement and easier to bias, so they need controlled comparisons.
- Relative advantages within a group, truncated sampling, and entropy control should be read together. If reward rises while pass rate falls, the usual cause is reward hacking or length gaming.
- Training logs should record prompt category, accuracy, average length, truncation rate, and why the verifier failed—not only a scalar reward.
6. The release should support a third-party audit
If the goal is reproducible research, the release should include at least: the data-mix ratios and filtering rules for each stage, training and post-training code, several intermediate checkpoints, key hyperparameters, the random-seed policy, evaluation scripts, and a decontamination report. If full source text cannot be released because of copyright or privacy limits, publish a reconstructable recipe, statistical features, and subsets that may be redistributed, and state clearly what cannot be rebuilt.
Do not report only one average score. Break results out at least into general QA, multiple choice, math, code, long context, and safety refusals, and note whether decontamination was applied. Report the domain tradeoffs of mid-training and the ablations of the three post-training stages as separate increments, so every gain is not attributed to the last reinforcement-learning run.
A schematic stage configuration can be fixed in a declarative description for review, instead of being scattered across script arguments:
stages:- name: pretraindata: quality_aware_mixobjective: next_token- name: midtraindata: domain_balanced_mixfocus: [code, math, qa, long_context]- name: sftdata: curated_demonstrations- name: preferencedata: delta_pairs- name: rlreward: verifiable_onlykl: scheduled
7. A checklist before release
Before shipping or open-sourcing, check each item: whether the quality curve beats a flat threshold on proxy tasks; whether the mid-training mix records its negative effect on general QA; whether instruction data moved earlier was deduplicated and decontaminated; whether preference pairs are relative to the current policy rather than to an unrelated reference answer; whether the reinforcement-learning reward can be verified independently; whether KL, length, and token-level loss have ablations; and whether the public materials are enough for someone else to reproduce the main conclusions.
These choices share one purpose: to write down where capability comes from. Quality-aware sampling addresses a shortage of useful tokens. Mid-training trades a controlled domain cost for specialized ability. Decontamination makes scores interpretable. Three-stage post-training optimizes format, preference, and verifiable correctness separately. The value of a complete model flow is that later work can follow the same path and confirm or reject these judgments, instead of only consuming a final weight that cannot be traced.
