Full-Stack Open-Source Language Models: A Complete Build Walkthrough

    This article breaks down the full pipeline behind a full-stack open-source language model series (7B and 32B parameters). It covers the tiers of open-source openness, the data engineering behind the ~5.9T-token base model (token-constrained mixing, quality-aware upsampling, document parsing, midtraining, and long-context handling), the three-stage post-training recipe (SFT to preference optimization to RLVR), training-framework engineering (parallelism, memory budgeting, mixed precision, checkpointing, fault tolerance, and RL-specific optimizations), the evaluation system, and a practical roadmap for teams to reproduce the work. Developers will learn which decisions—data first, staged post-training, and engineering alongside algorithms—most determine a model's quality.

    1. From “Open Weights” to “Full-Stack Transparency”

    In the open-model ecosystem, “open source” is not a single, consistent thing: some releases ship only the final weights, while others don’t even include complete training code. Recently, a research team working with several partner organizations released a full-stack open-source language model series at two parameter scales, 7B and 32B, accompanied by a report of roughly 20,000 words. The defining feature of this release is that the scope of what is public was extended from “the model itself” to “how the model was made.” Similar efforts existed before. For example, in early 2024, a non-profit research organization released a small-parameter model project emphasizing “complete transparency,” publishing the training code, training logs, the full dataset, and hundreds of intermediate checkpoints.

    1.1 Tiers of openness

    The table below orders releases by increasing scope of disclosure, roughly dividing today’s language model releases into five tiers.

    Tier What is open What users can do Main limitation
    L0 Weights only Final weights only Inference, quantized deployment, parameter-efficient fine-tuning Training process cannot be reproduced
    L1 Weights + inference code Adds model definition and inference scripts Deployment and basic evaluation Training dynamics remain a black box
    L2 Weights + training code Adds training framework and post-training code Continue training on your own data Data mixture and quality remain invisible
    L3 Weights + training code + data Adds the data mixing recipe (usually reconstructable samples, not necessarily the original snapshot) Reproduce an approximate data distribution, run ablations Results are still affected by randomness and hardware
    L4 Full-stack transparency (model flow) Adds per-stage weights, mid-training checkpoints, training logs, and data-processing and evaluation code Auditing, controlled experiments, attribution analysis Data licensing and compute barriers remain

    This model series is a typical L4 tier. What it publishes is not merely a “finished product” but a complete “model flow”: how data was collected, cleaned, and mixed; how the model changed step by step during training; and what each post-training stage added—all of it “on the record.”

    1.2 Why full-stack transparency matters

    • Reproducibility: Research conclusions can be independently verified instead of being taken on the authors’ word.
    • Auditability: Training logs and the data mixing recipe can be checked line by line; corpus composition, learning-rate curves, and loss changes are all documented.
    • Controllable modification: Developers can pick a direction from a mid-training checkpoint and continue training, instead of starting pretraining from scratch.
    • Safety governance: The resulting model can be fully inspected, including data licensing, copyright compliance, and safety-evaluation methods.

    1.3 What the model series contains

    Built around a shared base, the series derives several variants:

    • Base version: A model that has only gone through pretraining (and midtraining)—the common starting point for the other versions.
    • Instruct version: A general-purpose assistant with dialogue and instruction-following ability.
    • Thinking version: A reasoning-oriented version targeting math, code, and logical reasoning tasks—the flagship of the series.
    • Pure-RL version: A version that starts directly from the base model and runs reinforcement learning without supervised fine-tuning, used mainly to study “whether reasoning ability can emerge from reinforcement learning alone.”

    This “one base, many forms” organization lets differences between versions be attributed to explicit training steps, which makes it well suited to controlled studies.

    2. The Base Model: Data Engineering Behind 5.9T Tokens

    The base version completed pretraining on about 5.9 trillion (5.9T) tokens. Compared with simply scaling up the corpus, two data strategies deserve more attention: token-constrained mixing and quality-aware upsampling. It also introduced scientific-literature data processed by a document parser and scheduled a dedicated round of midtraining.

    2.1 Token-constrained mixing: budget first, then blend

    Traditional data mixing is usually expressed as “sampling ratios.” But sources differ widely in how many effective tokens they contain, and when the budget approaches the full usable amount of a given source, sampling at the original ratio will “overspend.” Token-constrained mixing reframes the mixing problem as a constrained optimization problem:

    • Objective: maximize the overall quality gain of the mixture;
    • Constraints: each source’s sampled tokens must not exceed its available cap, and all sources combined must exactly equal the total budget.

    The resulting mixture is feasible within the total budget and never over-repeats any corpus. A common engineering approach approximates the gain function with a data-quality classifier or rule-based scoring, then solves for the weights via methods such as Lagrangian duality.

    1. # Data mixing recipe (illustrative, not the real configuration)
    2. pretrain:
    3. budget_tokens: 5_900_000_000_000 # total token budget
    4. sources:
    5. web_crawl: {weight: 0.52, quality_tier: "medium-high"}
    6. code: {weight: 0.20, quality_tier: "high"}
    7. academic_papers: {weight: 0.10, quality_tier: "high"}
    8. math_and_reasoning: {weight: 0.06, quality_tier: "high"}
    9. books_and_reference: {weight: 0.12, quality_tier: "medium"}
    10. constraints:
    11. min_ratio_per_source: 0.01 # lower bound on any single source, so small sources don't get diluted into irrelevance
    12. max_repeat_epochs: 3 # max epochs per source, to control memorization
    13. midtrain:
    14. budget_tokens: 100_000_000_000
    15. focus: [code, math, qa]
    16. lr: {type: "linear_decay", final_fraction: 0.05} # connects with the end-of-pretraining annealing

    2.2 Quality-aware upsampling: spend compute on high-value corpora

    Quality-aware upsampling means letting the sampling probability rise with a data-quality score. A common approach introduces a quality-sensitivity coefficient α:

    sampling weight ∝ sample count × quality score^α

    When α = 0, this degenerates into uniform sampling by data volume; the larger α is, the more sampling concentrates on high-quality samples. The rationale is that high-quality corpora usually yield a higher “learning gain per token,” so under a fixed compute budget this strategy is often more efficient than uniform sampling. However, too large an α easily leads to high-quality corpora being trained repeatedly, causing overfitting and memorization, so a balance with data diversity is needed.

    1. import numpy as np
    2. def mix_tokens(counts, quality, budget, alpha=1.0, cap=0.35):
    3. """
    4. Quality-aware token budget allocation (illustrative implementation)
    5. counts : deduplicated token count available per data source
    6. quality: quality score per data source (0~1, from rules or a small scoring model)
    7. budget : total token budget
    8. alpha : quality sensitivity; 0 means allocate evenly by data volume
    9. cap : upper bound on any single source's share
    10. """
    11. counts = np.asarray(counts, dtype=np.float64)
    12. quality = np.clip(np.asarray(quality, dtype=np.float64), 1e-6, 1.0)
    13. # Step 1: quality-weighted initial weights
    14. w = counts * quality ** alpha
    15. w /= w.sum()
    16. w = np.minimum(w, cap) # cap any single source's share
    17. w /= w.sum()
    18. tokens = w * budget
    19. # Step 2: respect availability limits; flow the shortfall back to sources with headroom
    20. tokens = np.minimum(tokens, counts)
    21. residual = budget - tokens.sum()
    22. while residual > 0:
    23. headroom = counts - tokens
    24. if headroom.sum() <= 0:
    25. break
    26. extra = headroom / headroom.sum() * residual
    27. tokens += np.minimum(extra, headroom)
    28. residual = budget - tokens.sum()
    29. return tokens / counts # actual number of epochs per source

    2.3 Document-parsing quality: why scientific literature deserves separate handling

    Scientific papers, textbooks, and reviews usually exist as PDFs. Their raw text layers have three common problems:

    • Broken reading order: in double-column or complex layouts, the extracted text order may not match the logical order;
    • Layout pollution: headers, footers, references, and figure captions get mixed into the body text;
    • Lost symbols: formulas and tables are shattered during text extraction, severely damaging the semantics.

    A document-parsing pipeline (including OCR, layout analysis, and formula recognition) is responsible for restoring section hierarchy and reading order, transcribing formulas into readable linear expressions, and converting tables into structured text. This step directly determines whether the model can learn proper symbolic reasoning from the literature. For a model known for math and code reasoning, the fidelity of formulas often matters more than the total corpus size.

    2.4 Midtraining: a 100B-token “targeted make-up lesson”

    Near the end of pretraining, the model already has general language and foundational knowledge, but high-density abilities such as code, math, and QA tend to be unevenly distributed. Midtraining refers to a targeted enhancement round between the final stage of pretraining and post-training, using a smaller, more domain-focused corpus, typically on the order of tens of billions to about a hundred billion tokens.

    This series scheduled a round of midtraining on roughly 100B tokens, focusing on three directions:

    • Code: covering multiple mainstream languages, emphasizing cross-file dependencies, real project structure, and “code–documentation–test” pairings;
    • Math: including step-by-step derivations, multiple-solution comparisons, and conversions between natural-language problems and formal expressions;
    • Question answering (QA): including multi-hop reasoning, long-document retrieval-style QA, and factuality-related dialogue data.

    Midtraining is usually accompanied by learning-rate decay, so this stage is also the so-called “annealing” process: the model absorbs high-value data at a lower learning rate, with clear gains and less forgetting. Note that midtraining slightly changes the model’s general ability and “tone/style.” Keeping a proportion of general corpora and doing replay and learning-rate scheduling well are key to controlling the “seesaw” between abilities.

    2.5 Long-context ability: from “can read long text” to “can use long text”

    Many models support very long context windows, but the length over which information is effectively used is usually shorter than the nominal length—a phenomenon often summarized as “lost in the middle.” Common practices in the field include:

    • adjusting the base frequency of rotary position embeddings, or interpolating/extrapolating over position indices;
    • continuing pretraining with samples built from concatenated long documents;
    • constructing training samples like “long document + multi-hop QA” that genuinely require long-range information association.

    This series published a separate long-context data mixing recipe, indicating that long context is treated as an ability that needs dedicated training and evaluation, not a feature obtained simply by enlarging a window parameter.

    3. Post-Training: A Three-Stage Recipe from Base to Usable Model

    Base models are good at “continuing text” but not at “answering.” To make a model follow instructions, output a well-formed reasoning process, and solve complex problems requires a post-training pipeline. The flagship reasoning version of this series uses a three-stage recipe: supervised fine-tuning → preference optimization → verifiable-reward reinforcement learning. A pure-RL version was also built specifically to study the effect of skipping the first step.

    3.1 Stage 1: Supervised fine-tuning (SFT)

    The goal of SFT is to teach the model task formats and reasoning conventions:

    • Data form: high-quality “question–standard answer/reasoning process” pairs, emphasizing the completeness and checkability of the reasoning chain;
    • Training method: next-token prediction over the concatenated input-and-output sample, usually computing loss only on the response part;
    • Key lesson: data that is “lean and high-quality” matters more than data that is “abundant and noisy,” and format consistency directly affects the starting point for subsequent reinforcement learning.

    3.2 Stage 2: Preference optimization (DPO-style methods)

    The preference-optimization stage uses pairwise preference data, with the objective of increasing the log-probability gap of the “preferred response” relative to the “rejected response.” Two approaches are common:

    • Human annotation: annotators rank multiple candidate responses to the same question;
    • Objective feedback: preference pairs are constructed automatically from signals such as whether unit tests pass, whether the answer matches the ground truth, and whether the code passes the compiler and static checks.

    One idea disclosed in the report is called “Delta Learning.” In essence: first let the model sample multiple responses to the same question, then have a judge model or objective signal assign scores, compute each response’s “delta” relative to the group’s average score, and treat positive deltas as positive samples and negative deltas as negative samples—building preference data cheaply and at scale.

    This idea essentially moves the notion of “relative advantage” from the RL stage forward into preference-data construction. It works much like the “within-group relative advantage” in reinforcement learning: what matters is not absolute good-or-bad, but “better or worse than similar choices.” Compared with absolute scoring, relative judgment is often more stable, easier to label, and closer to the psychology of human preference judgments.

    3.3 Stage 3: Verifiable-Reward Reinforcement Learning (RLVR)

    After preference optimization, the model follows instructions well but still has room to improve on complex reasoning tasks. The idea of RLVR is to avoid training a learned reward model and instead judge answer correctness directly with programmatic rules (whether numbers match, whether all unit tests pass, whether a formal-proof checker accepts), then use that result as the reward signal to optimize the policy.

    The typical flow can be summarized as:

    1. For the same batch of problems, sample multiple responses with the current policy;
    2. Score each response with a verifier (usually a 0/1 or piecewise score);
    3. Compute the within-group relative advantage: each response’s advantage = (that response’s score − group average) / (group standard deviation + ε), requiring no separate value network;
    4. Update parameters with a policy-gradient method, combined with clipping (a trust region) to prevent overly large updates.
    1. # RLVR training loop based on group-relative advantage (illustrative pseudocode)
    2. # Distributed setup, memory optimization, and checkpoint resume are omitted
    3. for step, batch in enumerate(dataloader):
    4. # batch: a batch of problems, each with ground-truth answer / unit tests / formal checker
    5. # 1. Rollout: sample G responses per problem
    6. responses = policy.generate(batch.prompts, n=G, temperature=1.0,
    7. max_new_tokens=8192)
    8. # 2. Verifiable reward: rule-based judgment, no reward model to train
    9. scores = verify(responses, batch.ground_truth) # shape: [B, G]
    10. adv = group_relative_advantage(scores) # standardized within group, no value network
    11. # 3. Re-forward: compute token-level log-probabilities under the current policy
    12. logp_new = policy.log_prob(responses) # shape: [B, G, L]
    13. logp_old = rollout_logp.detach() # old-policy log-probs recorded at sampling time
    14. # 4. Clipped policy-gradient objective
    15. ratio = (logp_new - logp_old).exp()
    16. surr1 = ratio * adv.unsqueeze(-1)
    17. surr2 = ratio.clamp(1 - clip_low, 1 + clip_high) * adv.unsqueeze(-1)
    18. policy_loss = -torch.min(surr1, surr2)
    19. # 5. Token-level normalization: average over all valid tokens in the batch, not per-sequence first
    20. token_mask = response_mask.float() # marks non-padding, non-terminator tokens
    21. loss = (policy_loss * token_mask).sum() / token_mask.sum().clamp(min=1)
    22. loss.backward()
    23. optimizer.step()

    3.4 Reward design and “reward hacking”

    The effectiveness of RLVR depends heavily on verifier quality. A verifier that is too weak teaches the model to “guess answers” or produce outputs that are correctly formatted but wrongly reasoned; a verifier that is too strict yields sparse training signals and slow progress. Common improvements include:

    • Piecewise rewards: combining format, process, and result scores to ease signal sparsity during cold start;
    • Difficulty stratification: adjusting sampling temperature and count by problem difficulty, avoiding zero advantage when a group is all-correct or all-wrong;
    • Verifier iteration: expanding the test set and counterexamples with the model’s own failure cases to form a closed loop.

    “Reward Hacking” is an unavoidable problem for RLVR and must be monitored continuously during training: accuracy rising while output length inflates abnormally, formats becoming formulaic, answer phrasing pandering to the verifier’s habits, and so on.

    3.5 The pure-RL version: what happens when you skip SFT

    The pure-RL version in the series starts directly from the base model for RLVR, without supervised fine-tuning. Its value lies in answering a research question: can complex reasoning behaviors (such as self-checking, backtracking, and verification) “grow” purely from outcome rewards, without any demonstrations? Such experiments usually observe a few phenomena:

    • response length grows naturally as training progresses;
    • spontaneous behaviors like self-verification and step-by-step reflection appear;
    • readability and instruction-following may be worse than in the SFT version.

    This result shows that supervised fine-tuning is more about “guiding format and ability,” while reinforcement learning is the “amplification and refinement of ability.” The takeaway for developers is that post-training recipes should be combined flexibly according to the target ability, rather than mechanically copying a fixed pipeline.

    4. Training Framework and Engineering Optimization: Turning “It Trains” into “It Trains Well”

    The framework paired with this series (released separately for the training side and the RL side) is systematically optimized around “less memory, more stable convergence, and easier recovery.” Its design ideas are broadly applicable, and developers should pay attention to the following.

    4.1 Parallelism strategies and the memory ledger

    Large-model training usually requires combining multiple forms of parallelism:

    • Data parallelism: each GPU holds a full model copy, processes a different data batch, then synchronizes gradients;
    • Tensor parallelism: splits a single matrix operation across GPUs to reduce per-GPU weight and activation memory;
    • Pipeline parallelism: splits the model by layer and fills the pipeline with micro-batches;
    • Sequence parallelism: splits a long sequence across GPUs to reduce peak activation memory;
    • Optimizer state sharding: distributes optimizer states across the GPUs in the data-parallel group.

    Memory estimate (for mixed-precision training):

    • each parameter needs about 2 bytes of weights + 2 bytes of gradients + 12 bytes of optimizer state (first moment, second moment, and a master-weight copy), for about 16 bytes total;
    • a 7B model needs about 112GB just for the “model states,” not counting activations;
    • a 32B model needs about 512GB.

    This means that even training a 7B model requires multiple GPUs with parallelism and state sharding; a 32B model basically has to be trained on a multi-machine, multi-GPU cluster.

    4.2 Mixed precision, activation recomputation, and checkpointing

    • Mixed precision: forward/backward compute uses low precision (such as bfloat16) for speed and memory savings, while critical accumulations and optimizer updates keep high precision;
    • Activation recomputation: intermediate activations are not all saved in the forward pass and are recomputed in the backward pass—trading compute for memory;
    • Checkpointing: periodically saving “model + optimizer + data-loader position,” combined with asynchronous and sharded saving to reduce loss from failures.

    This series published hundreds of intermediate checkpoints from the entire pretraining process, a direct product of this mechanism. These checkpoints let researchers observe how abilities evolve across training stages and make “continuing training from any stage” possible.

    4.3 Elastic fault tolerance and checkpoint resume

    For large-scale training, hardware failures and node preemption are the norm, not a smooth run to completion. Engineering-wise, you need:

    • health checks and automatic isolation of faulty nodes;
    • training jobs that support elastic scaling, automatically rescheduling when nodes change;
    • periodic verification of checkpoint integrity to ensure “loss continuity after resume.”

    4.4 Three key RL-framework optimizations

    First, remove the KL-divergence penalty. Traditional RLHF penalizes the KL divergence between the policy and the reference policy to control drift and prevent the model from “going off the rails.” But at the RLVR stage, a common practice is to remove this term, for the following reasons:

    • it constrains the model from “drifting too far,” which simultaneously suppresses exploration;
    • the verifiable reward is itself an objective signal, so the risk from drift is more controllable;
    • after removing the KL term, the objective points more purely at task correctness, and training commonly shows growing response length and steadily rising pass rates.

    Of course, removing the KL term also means other constraints (clipping range, learning rate, group-relative advantage) must replace its stabilizing role—which is why some recipes remove only the KL term and not the trust-region clipping.

    Second, token-level loss. Averaging loss per sequence (first averaging over each response, then over the batch) dilutes the weight of each token in a long response. Token-level loss normalizes uniformly over all valid tokens in the batch, equivalent to making a long response’s “total contribution” proportional to its length, which better matches the objective for “long-thinking” tasks.

    Third, the “silent drift” between rollout and training. Sampling is usually done with an inference engine and training with a training framework, and the two can differ subtly in their numerical implementation. This difference can make “the probability at sampling time” inconsistent with “the probability computed during training,” distorting the importance ratio and manifesting as training instability or even collapse. A common response is to clip or correct the importance ratio (e.g., truncated importance sampling) and monitor the log-probability gap between the training and inference sides.

    5. Evaluation: The Last Piece of the Trust Puzzle

    5.1 Multi-dimensional evaluation

    Evaluation is usually divided into a few categories:

    • Knowledge: subject-matter QA and factuality;
    • Reasoning: math problems and multi-step logical reasoning;
    • Code: function-level completion, algorithm problems, and unit-test pass rates;
    • Long context: long-document retrieval and multi-hop QA;
    • Instruction following: format constraints, multi-turn dialogue, and refusal and safety.

    For reasoning models, “pass@1” is no longer enough to reflect ability; “pass@k” or “avg@k” better shows the ceiling and stability after multiple samples. When evaluating, temperature, sample count, prompt templates, and parsing rules should all be reported, otherwise results are not comparable.

    5.2 Data contamination

    Once public datasets enter the training corpus, evaluation scores inflate. Countermeasures include: performing n-gram overlap detection during the mixing stage and removing high-overlap samples from training; using held-out sets, time-sliced data, or private question sets for cross-validation during evaluation; and reporting the evaluation date and model version to track “leaderboard-chasing” risk.

    5.3 Process evaluation

    Looking only at the final answer cannot tell whether the reasoning is reliable. Process-level metrics can be introduced: comparing answer accuracy with “intermediate-step error rate,” sensitivity to perturbations (change one number and see whether the answer changes accordingly), and the relationship curve between reasoning length and accuracy.

    6. Reproduction in Practice: What Teams Can Do

    The value of this series lies not only in “using it” but also in “doing it the same way.” Here is a practical, actionable roadmap.

    6.1 Start from a “minimal closed loop”

    Don’t chase 5.9T tokens from the outset. It is recommended to first run the full pipeline end to end with a small model (e.g., 0.5B–2B) and a small budget (tens of billions of tokens): data mixing → midtraining → SFT → preference optimization → reinforcement learning → evaluation. After the pipeline works, gradually scale up model size and budget. This path is light on compute but exposes the “pitfalls” in the process early.

    6.2 Four must-dos for data governance

    • Deduplication: exact dedup, near dedup (e.g., MinHash/LSH-based), and cross-source dedup;
    • Filtering: rule-based filtering (length, repetition rate, garbled-text rate) and quality-classifier filtering;
    • Licensing and compliance: distinguish commercially usable from research-only data, and record sources and licenses;
    • Versioning: data mixing recipes, sampling random seeds, and dedup snapshots should all be versioned, otherwise “reproduction” is impossible.

    6.3 Compute and resource planning

    • 7B-scale full-parameter training: start with 8 GPUs at 80GB each, combined with state sharding and activation recomputation;
    • 32B-scale full-parameter training: requires multi-machine, multi-GPU, plus sequence parallelism and checkpoint optimization;
    • Inference deployment: quantize on demand (e.g., INT8/INT4) and use tensor parallelism to spread memory;
    • Evaluation stage: pass@k evaluation for reasoning models is time-consuming and should be planned with dedicated evaluation-cluster resources.

    6.4 Observability of the training process

    • Metrics to monitor: train/validation loss, learning rate, gradient norm, per-task accuracy, and response-length distribution;
    • Alerts: loss spikes, gradient explosions, validation-set degradation, and overly large log-probability deviation between rollout and training;
    • Logs: archive training logs, config snapshots, and checkpoint indices together for after-the-fact attribution.

    At the deployment layer, checkpoints can be stored in object storage, distributed training and inference services can be orchestrated with a container platform, and unified observability can be achieved through logging and monitoring/alerting services. Cloud providers such as Baidu AI Cloud offer these general-purpose cloud capabilities, so teams can build an end-to-end training-to-inference pipeline on that basis without running their own data centers.

    6.5 An evaluation and iteration loop

    Make evaluation a “repeatably executable pipeline”: fix the question set, fix the templates, and fix the decoding parameters, then run it automatically after each training run and compare against historical results. Also keep a “private question set” for cross-validation to avoid the public set being indirectly contaminated.

    7. Limitations and a Rational View

    • Cost barrier: fully reproducing one pretraining run still requires substantial compute, so ordinary teams are better off “continuing training from open checkpoints”;
    • Data licensing: some corpora have licensing restrictions, so publishing a “mixing recipe” is not fully equivalent to publishing the “raw data”;
    • Evaluation limits: public leaderboards carry contamination and overfitting risks, so real business scenarios still need self-built evaluation sets;
    • Behavioral risk: removing the KL term and pursuing long reasoning can lead to verbose responses, format drift, and reward hacking, so output-length limits and format checks are needed in products;
    • Model boundaries: an open model’s capability ceiling is constrained by data and the training recipe, so key business scenarios should retain human review and fallback mechanisms.

    8. Conclusion

    The core contribution of this full-stack open-source model series is not a single “strongest metric” but laying out the entire process—from pretraining and midtraining to post-training—for the community: how data is mixed, how documents are parsed, how the KL term is removed in RL, how token-level loss is normalized, and how rollout drift is corrected. These questions previously stayed inside labs and engineering teams; now they become public knowledge that can be discussed, reproduced, and improved.

    For developers, three methodological points are most worth absorbing:

    1. Data first: the mix, quality, and document-parsing quality often determine a model’s ceiling more than parameter count does;
    2. Staged post-training: SFT is responsible for “being able to do it,” preference optimization for “getting the preference right,” and RL for “doing it better”;
    3. Engineering and algorithms alike: the memory ledger, parallelism strategies, checkpointing and fault tolerance, and rollout consistency are engineering details that directly determine whether the algorithms can work stably on real clusters.

    The significance of full-stack openness lies precisely in letting those who come after avoid repeating past mistakes and continue forward from their predecessors’ checkpoints.