How Full-Stack Open-Source LLM Training and Post-Training Work: A Design Breakdown
Large-model open source is shifting from "open weights" to "open everything." This article uses a full-stack open-source model technical report as a reference to break down how such models are actually trained and post-trained, without pointing to any specific model. It walks through pre-training data engineering (token-constrained data mixing and quality-aware sampling), mid-training for code/math/long-context, the three-stage SFT→DPO→RLVR post-training pipeline, the training frameworks and evaluation checklists that make results reproducible, and practical guidance for enterprises that want to build domain capabilities on top of an open base. You will find working Python snippets for sampling plans, preference-pair construction, and RLVR group-normalized advantages, plus reproducible training and evaluation practices.
1. Introduction: How Do Full-Stack Open-Source LLMs Work?
Large-model open source is moving from “open weights” to “open everything.” In the past, a model card, the weight files, and some simple inference code were enough to call a project open source. Today, more and more projects publish data mixtures, training code, post-training recipes, reinforcement learning frameworks, and evaluation methods. This kind of work is often called “full-stack open source.”
For developers and enterprises, the real value is not any single isolated parameter, but the answers to three questions: How do you build a full-stack open-source model? How do you choose the data mixture? And how do you evaluate whether mid-training and post-training actually work?
This article uses a full-stack open-source model technical report as a reference to break down its training design and engineering practices, without pointing to any specific model. The main threads are: pre-training data engineering, mid-training reinforcement, the three-stage SFT→DPO→RLVR post-training pipeline, the training framework and evaluation system, and recommendations for enterprise adoption. The goal is to help readers build a transferable, reproducible path for developing model capabilities.
2. Pre-Training Data Engineering: How Do You Choose the Data Mixture?
Pre-training sets the ceiling on model capability. High-quality data is critical, but it usually accounts for only a small fraction of the total corpus. If you simply sample in proportion to raw token counts, the model will repeatedly see the same batch of high-quality samples across most training steps, which leads to two problems: diminishing marginal returns in later training, and overfitting to specific phrasings. So the data mixture cannot be judged only by “how much is there”; you also have to consider “how many tokens can actually be trained,” “which corpora are worth repeating,” and “how many repetitions are reasonable.”
2.1 Token-Constrained Mixing: Emitting Directly Usable Sampling Probabilities
The core of token-constrained mixing is this: given a total training token budget, fold each data domain’s deduplicated usable tokens, quality score, and repetition cap into a single allocation logic. The example below not only produces a target token count per domain, but also outputs normalized sampling probabilities that can be fed directly into a data sampler.
CORPUS = {"web_text": {"tokens": 2.6e12, "quality": 0.35, "keep_ratio": 0.70, "max_repeat": 1},"code": {"tokens": 4.5e11, "quality": 0.65, "keep_ratio": 0.95, "max_repeat": 4},"math": {"tokens": 2.0e10, "quality": 0.75, "keep_ratio": 0.95, "max_repeat": 8},"science": {"tokens": 5.0e10, "quality": 0.80, "keep_ratio": 0.90, "max_repeat": 6},"encyclopedia":{"tokens": 1.0e10, "quality": 0.85, "keep_ratio": 0.98, "max_repeat": 8},}def sampling_plan(corpus, total_budget, iters=12, eps=1e-12):# Tokens that can be effectively learned after deduplicationusable = {k: v["tokens"] * v["keep_ratio"] for k, v in corpus.items()}# Maximum tokens each domain can contribute: usable tokens * repeat capcap = {k: usable[k] * v["max_repeat"] for k, v in corpus.items()}# Higher quality is prioritized; smaller domains need moderate upweighting,# but are bounded by max_repeatscore = {k: v["quality"] * (usable[k] ** -0.25)for k, v in corpus.items()}total_score = sum(score.values())tokens = {k: min(total_budget * score[k] / total_score, cap[k])for k in corpus}remaining = total_budget - sum(tokens.values())# Redistribute unused budget to domains that still have capacityfor _ in range(iters):if remaining <= eps:breakactive = [k for k in corpus if cap[k] - tokens[k] > eps]if not active:breakwsum = sum(score[k] for k in active)if wsum <= eps:breakchanged = Falsefor k in active:add = remaining * score[k] / wsumadd = min(add, cap[k] - tokens[k])if add > eps:tokens[k] += addremaining -= addchanged = Trueif not changed:breaktotal_tokens = sum(tokens.values())if total_tokens <= eps:raise ValueError("No trainable tokens available, check the corpus configuration")# prob can be used directly as the sampling probability per domainprob = {k: tokens[k] / total_tokens for k in corpus}return prob, tokens, total_tokens
The key point here is not any fixed ratio, but turning the mixture into three checkable conditions: first, how many usable tokens each domain can supply at most; second, how quality affects priority; and third, whether the final output is normalized into probabilities that can be sampled directly. If a small high-quality domain is capped by its repetition limit, the system should flag the need to collect more data rather than raising the repetition count without bound.
2.2 Quality-Aware Sampling, Deduplication, and Document Parsing
Quality-aware upsampling has to account for quality, scale, and redundancy at the same time. High-quality, low-redundancy corpora can tolerate a higher sampling multiplier; low-quality or heavily duplicated corpora should be pushed down. In practice, it is better to use the “tokens remaining after sentence-level deduplication” as the input to the mixture, instead of relying on raw volume alone.
Large-scale corpora usually use two levels of deduplication: document-level near-duplicate removal and paragraph-level deduplication. Document-level deduplication uses methods such as locality-sensitive hashing to find highly overlapping documents; paragraph-level deduplication keeps unique paragraphs and moves high-frequency paragraphs into a shared paragraph pool, sampling them at a uniform frequency. This reduces memorization risk without completely discarding high-value phrasing.
Document parsing is another easily overlooked data pipeline. Much scientific literature still exists as scanned PDFs and needs layout analysis, formula reconstruction, table structuring, reading-order recovery, and OCR proofreading. But note: the quality ceiling of OCR-processed corpora is limited by recognition quality, and language-model infilling can introduce hallucinations. From an engineering standpoint, keep confidence scores, sampling audits, and human review mechanisms in place so the model does not learn incorrect formulas or incorrect reasoning formats.
2.3 Data Configuration and Compliance
The data recipe is best published as a configuration file that records each data domain, its quality score, deduplication ratio, repetition cap, and sampling probability. This lets the community trace “which kind of data a given capability comes from” and makes ablation studies easier. At the same time, data release must come with licenses, provenance fields, privacy screening, and authorization notes for scanned materials. Releasing data without compliance metadata increases the risk of enterprise reuse.
3. Mid-Training: How Do You Fill the Gaps in Code, Math, and Long Context?
A model after pre-training has general language ability, but it is often weak on code, math, multi-step reasoning, and long-text tasks. The reason is that this high-quality data is a small share of the total, and the model has already converged on the general distribution. Mid-training inserts a targeted reinforcement stage lasting roughly ten billion tokens between the main pre-training phase and post-training.
Mid-training data is usually centered on code, math, and high-quality Q&A, while keeping a certain proportion of general corpus to prevent degradation of general ability. Code data strengthens formal expression and structured reasoning, math data trains multi-step computation, and Q&A data turns knowledge into responsive behavior. If long-context data is combined in, you should also construct document-level long samples, cross-paragraph association tasks, and position-aware attention masks, so the model learns to “locate and associate information within long text” rather than merely accepting longer inputs.
For the training schedule, a short-then-long order is recommended: complete most of the tokens with shorter context, then finish with longer context; the learning rate is usually lower than in pre-training to avoid damaging general representations; and keep a general evaluation set as a regression baseline to prevent mid-training from improving a single ability while hurting overall performance.
4. Post-Training: How Do You Build Reasoning Ability with SFT, DPO, and RLVR?
Flagship reasoning models typically use a three-stage post-training pipeline of “SFT→DPO→RLVR.” Here DPO refers to Direct Preference Optimization and DPO-style methods. The three stages solve different problems: SFT solves whether the model can follow instructions, DPO solves which answer it prefers, and RLVR solves whether the reasoning process is reliable.
4.1 Stage 1: SFT for Instruction Following
SFT uses high-quality instruction-response pairs for a cold start. The data should be generated and filtered by a stronger teacher model, covering multiple tasks, difficulties, and formats, while removing low-quality, templated, and abnormally long responses. The goal of SFT is not to make the model “know more,” but to teach it to reliably follow formats and instructions.
4.2 Stage 2: DPO Preference Optimization
DPO-style methods train the model to distinguish better answers from worse ones using pairwise preference data. A more effective approach than simply collecting the “best answer” is contrastive construction: sample multiple candidate answers for the same question, select a “best among the poor” set along evaluation dimensions, and strengthen learnable differences through a repair step. At this point, the chosen sample is positive, the locally repaired rejected sample is negative, and the original weak answer can also be kept as a hard negative.
def critique(prompt, chosen, rejected):"""Return structured defects rather than rewriting the full answer."""return {"missing_steps": ["missing unit conversion"],"wrong_claims": ["wrong boundary-value check"],"format_issues": ["did not output JSON as required"]}def repair(prompt, rejected, defect):"""Have the teacher model fix only the defects, preserving traces of the weak answer."""return edited_answerdef build_preference_pair(prompt, candidates, scorer, margin=0.2):scored = [(c, scorer(prompt, c)) for c in candidates]scored.sort(key=lambda x: x[1], reverse=True)if len(scored) < 2:return Nonebest, worst = scored[0], scored[-1]if best[1] - worst[1] < margin:return Nonedefect = critique(prompt, best[0], worst[0])repaired_rejected = repair(prompt, worst[0], defect)return {"prompt": prompt,"chosen": best[0],"rejected": repaired_rejected,"hard_negative": worst[0]}
More preference data is not necessarily better; clearer differences are better. Preference pairs with high noise and small gaps dilute the training signal. For code tasks you can focus on runnability, for math tasks on step correctness, and for dialogue tasks on factual consistency and format compliance.
4.3 Stage 3: RLVR and Group Normalization
RLVR is reinforcement learning with verifiable rewards. The reward does not depend on human scoring but comes from automatically checkable results, such as exact match of math answers, passing code test cases, or format-validity checks. The typical flow is: sample multiple trajectories for each question, use a verifier to assign rewards, normalize advantages within the group of the same prompt, and then update the model with a clipped policy-gradient method.
import torchdef grouped_reward_advantage(reward, group_ids, eps=1e-6):"""reward: [B], the verifiable reward for each rolloutgroup_ids: [B], multiple samples from the same prompt share the same group_id"""reward = reward.float()group_ids = group_ids.to(reward.device)adv = torch.zeros_like(reward, dtype=torch.float32)for gid in torch.unique(group_ids):idx = group_ids == gidg = reward[idx]if g.numel() <= 1:adv[idx] = 0.0continueadv[idx] = (g - g.mean()) / (g.std(unbiased=False) + eps)return advdef rlvr_loss(logp_new, logp_old, reward, group_ids, mask, clip_eps=0.2):# logp_new / logp_old: [B, T]# reward: [B]# group_ids: [B]# mask: [B, T], counts only the tokens generated by the modeladv = grouped_reward_advantage(reward, group_ids)ratio = torch.exp(logp_new - logp_old.detach())surr1 = ratio * adv[:, None]surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * adv[:, None]return -((torch.min(surr1, surr2) * mask).sum() / mask.sum().clamp(min=1))
Here the shape of the reward and the grouping must be stated clearly: reward is a reward vector over all rollouts in a batch, and group_ids identify which prompt each rollout comes from. You cannot simply treat the whole batch as a single group, otherwise differences in difficulty across questions will contaminate the advantage values.
Removing the KL penalty does not mean relaxing constraints. The clipping mechanism still limits the per-step update magnitude, and KL can be kept as a monitoring metric to observe whether the policy has drifted too far from the initial distribution. Common failure modes include identical rewards within a group, difficulty collapse, verifiers being gamed, and length bias. In engineering, you can use dynamic sampling to filter out non-discriminative samples, along with length and format constraints.
5. Training Framework and Evaluation Checklist: How Do You Prove Training Works?
The other half of full-stack open source is the training framework and evaluation system. Pre-training and mid-training are mainly constrained by compute throughput, while post-training—especially the RLVR stage—is more constrained by generation throughput, weight synchronization, and memory. Reproducibility requires that data order, random seeds, sampling configuration, checkpoints, and evaluation scripts all be under version control.
| Dimension | Pre-training / Mid-training | Post-training, especially RLVR |
|---|---|---|
| Main bottleneck | Compute throughput | Generation throughput, weight sync, memory |
| Parallel strategy | Combination of data, tensor, and pipeline parallelism | Separate deployment of inference engine and training engine |
| Key metrics | tokens/s, loss curve | Sample throughput, in-group reward variance, KL monitoring |
| Reproducibility | Fixed data order and random seeds | Fixed sampling config, log per-round reward distribution |
The training framework is only the foundation; what really determines whether you can iterate is the evaluation checklist. It is recommended to run the following checklist before and after any mixture adjustment, mid-training stage, or post-training stage:
| Evaluation Item | Recommended Practice | Purpose |
|---|---|---|
| Eval set source | Combine public benchmarks, self-built business sets, and domain-specific sets | Avoid testing only public leaderboards and drifting from real scenarios |
| Prompt template | Version-controlled, with fixed templates and parsing scripts | Avoid prompt changes making scores incomparable |
| Metric definition | Clarify definitions such as exact match, pass@1, unit test pass rate, format validity rate | Prevent different teams from interpreting the same score differently |
| Baseline models | Keep the previous-stage model, the open-source base, and the current best model | Determine whether gains come from data, algorithm, or random fluctuation |
| Regression frequency | Must test at the end of each stage; spot-check every few steps during RLVR | Detect degradation or reward hacking promptly |
| Pass threshold | Set a minimum acceptable score and a rollback threshold | Make training decisions auditable and rollbackable |
For teams that want to reproduce or build further on the model, it is best to lock down three things first: data loading order and random seeds, intermediate checkpoints for each stage, and a unified evaluation script. Model capability comes from comparable experiments, not one-off parameter tuning.
6. Data Release, Licensing, and Safety Compliance
Full-stack data release demonstrates a reproducibility commitment better than releasing weights alone. A complete data release system usually includes the pre-training corpus mixture, mid-training data, long-context data, and post-training data. But when enterprises reuse it, they cannot look only at data volume; they also need to check licenses, provenance, privacy screening, and safety filtering.
It is recommended to add the following fields to each dataset: data license, source link or collection method, deduplication version, quality scoring rules, privacy de-identification notes, authorization status of scanned materials, and content safety filtering policy. For self-built enterprise data pipelines, copyright review, privacy compliance, and output content safety filtering should be prerequisites for launch, not after-the-fact fixes.
7. Enterprise Adoption: The Minimum Viable Path
Most enterprises are not suited to full-scale pre-training from scratch; a more feasible path is to start from an open base and gradually inject domain capabilities:
- Choose a reproducible open base, and confirm the model license and evaluation baseline.
- Organize business corpora, and complete deduplication, quality scoring, sensitive-information filtering, and mixture configuration.
- Use a small amount of mid-training data to strengthen code, math, Q&A, or domain-knowledge capabilities.
- Build domain SFT data, DPO preference data, and verifiable-reward tasks, then complete RLVR or rule-based reinforcement.
- Build an acceptance evaluation set, set pass thresholds and a rollback mechanism, and then gradually scale up training.
The core of this path is “evaluate first, then train, then scale.” Without a stable evaluation set, both mixture adjustments and algorithmic improvements are hard to verify; without version-controlled configuration, model iteration degrades into untraceable, ad hoc operation.
8. Summary
The value of full-stack open-source LLMs lies not in any single-point technique, but in turning model production into a traceable pipeline. On the data side, token-constrained mixing, quality-aware upsampling, two-level deduplication, and document parsing turn data quality into a computable problem. On the training side, mid-training reinforcement fills the gaps in code, math, and long context. On the post-training side, SFT, DPO, and RLVR respectively solve instruction following, preference alignment, and reasoning reliability. On the engineering side, training frameworks, evaluation checklists, and compliance governance improve reproducibility.
For developers and enterprises, the most worthwhile takeaway is the closed-loop design method of “data—training—post-training—evaluation.” Running this pipeline end to end in your own business scenario is where the true adoption value of full-stack open source lies.
