A Deep Dive into Full-Stack Open Source LLMs: Technical Architecture and Training Paradigms

The evolution of Large Language Models (LLMs) is gradually shifting from an “open-weight” approach to a “full-stack open source” paradigm. True full-stack open source goes beyond providing the final model weights; it requires the disclosure of complete datasets, training code, intermediate checkpoints, and training logs across all stages—from pre-training and mid-training to post-training. This end-to-end transparency provides developers with valuable references for deeply understanding LLM training mechanisms, reproducing experimental results, and conducting secondary development. This article will deconstruct the technical architecture of full-stack open source models, exploring the training highlights and engineering practices at each stage.

1. Pre-Training Stage: Fine-Grained Control of Data Distribution

Pre-training is the cornerstone of building a base model, with its core lying in the control of data quality and proportioning. In a full-stack open source technical system, the highlights of the pre-training stage are primarily reflected in the following two areas:

1.1 Quality-Aware Upsampling Strategy

In traditional pre-training data processing, developers often employ a “flat filtering” strategy—setting a quality threshold and discarding data below it. However, this crude truncation method leads to a homogeneous data distribution and severely impacts model performance when data is limited.

A superior approach is “quality-aware upsampling.” This method constructs a smooth upsampling curve based on data quality scores. High-quality data has a higher probability of being sampled, while low-quality data is not completely discarded but participates in training at a lower probability. The advantages of this strategy include:

• Preserving Data Diversity: Avoids the loss of long-tail knowledge caused by strict filtering.
• Optimizing Data Distribution: Allows the model to focus more on high-value information under limited computing power.
• Improving Performance in Data-Limited Scenarios: When high-quality data is insufficient, a smooth upsampling strategy effectively mitigates overfitting and enhances model generalization.

1.2 Integration of High-Quality Scientific Literature

To equip the model with stronger domain-specific knowledge, the pre-training stage heavily incorporates scientific PDF data processed via advanced OCR technologies. This data typically contains complex formulas, charts, and academic terminology. During the cleaning process, developers must resolve PDF layout parsing errors and formula garbling to convert them into high-quality pre-training corpus. This significantly enhances the base model’s foundational capabilities in high-cognitive-threshold scenarios such as scientific research and academia.

2. Mid-Training Stage: Finding the Optimal Balance for Domain Capabilities

After completing basic pre-training, the model undergoes a mid-training stage to specifically strengthen its capabilities in coding, mathematics, and Question Answering (QA). This stage acts as a bridge connecting the general base model to specific capabilities. The technical challenges primarily lie in the trade-offs between domains and the utilization of forward benefits.

2.1 Trade-offs in Domain Capabilities

The data ratio during mid-training directly affects the model’s final capability distribution. Experiments indicate:

• Increasing the weight of math and code data significantly boosts the model’s logical reasoning and programming abilities.
• However, this bias harms the model’s performance on general QA and multiple-choice QA, leading to “catastrophic forgetting.”

To resolve this contradiction, development teams must carefully construct mixed datasets to find the optimal balance. This typically requires dynamic adjustments to the ratios and learning rates of different domain data through multiple controlled experiments, ultimately determining a recipe that strengthens hardcore capabilities while maintaining general generalization.

2.2 Forward Deployment of Post-Training Data

A notable innovative practice is the early introduction of instruction-following and Chain-of-Thought (CoT) data during the mid-training stage. In traditional pipelines, such data is usually only used in the post-training phase. By deploying it earlier, the base model’s performance across various benchmark evaluations improves comprehensively, even without formal Supervised Fine-Tuning (SFT). This “forward warm-up” allows the model to adapt to human instruction formats and logical reasoning processes early on, greatly reducing the alignment difficulty in the subsequent post-training stage.

2.3 Strict Decontamination Processing

Benchmark data leakage is a persistent issue in current LLM training. Many open-source datasets inadvertently contain answers from evaluation sets, leading to artificially inflated benchmark scores. The full-stack open source system emphasizes the necessity of decontamination. Although this may cause a drop in certain metrics, the decontaminated model exhibits more authentic performance and stronger robustness, which is crucial for evaluating the model’s usability in real-world business scenarios.

3. Post-Training Stage: Three-Stage Recipe and Reinforcement Learning Innovations

The post-training stage aims to transform the base model into a flagship model with robust reasoning capabilities. Its core employs a three-stage recipe: SFT -> DPO -> RLVR.

3.1 Three-Stage Training Pipeline

• SFT (Supervised Fine-Tuning): Utilizes high-quality, manually annotated data to teach the model to follow instructions and output formats that meet human expectations.
• DPO (Direct Preference Optimization): Directly uses preference data to align the model without requiring an explicit reward model, enhancing the safety and helpfulness of the outputs.
• RLVR (Reinforcement Learning with Verifiable Rewards): Targets domains with definitive correct answers, such as math and code, using verifiers (rather than subjective reward models) to provide feedback, significantly boosting the model’s deep reasoning capabilities.

3.2 Delta Learning for Preference Data Construction

During the DPO stage, the cost of constructing high-quality preference data is exceptionally high. Technical disclosures reveal an innovative “Delta Learning” mechanism. This mechanism constructs preference pairs by comparing the differences in intermediate states across various reasoning paths. For example, given a complex math problem, the model might generate multiple reasoning paths. Delta Learning can automatically identify the “critical decision nodes” that lead to the final correct or incorrect results, constructing positive and negative preference samples accordingly. This automated data construction method drastically reduces manual annotation costs and improves the targeting of preference data.

3.3 Algorithmic and Engineering Optimizations in the OlmoRL Framework

In the RLVR stage, the reinforcement learning framework undergoes deep optimizations:

• Removing the KL Divergence Term: Traditional PPO algorithms introduce Kullback-Leibler (KL) divergence to prevent the policy from deviating too far from the reference model. However, in scenarios with verifiable rewards, exploring correct answers is more important than maintaining similarity to the base model. Removing the KL divergence term grants the model greater freedom to explore within the logical reasoning space.
• Token-Level Loss Calculation: Traditional loss calculation is typically performed at the sequence level. Introducing token-level loss allows the model to more finely evaluate the contribution of each generated token. This is particularly important for long CoT reasoning, as it can accurately penalize incorrect steps in the reasoning process and reward critical correct deductions.

4. Engineering Practices and Enterprise-Level Deployment Recommendations

The full-stack open source model system is not only a treasure for academic research but also provides a standard paradigm for enterprise-level LLM deployment. When building their own LLMs, enterprises can reference the methodologies above and combine them with cloud-native infrastructure for engineering implementation:

1. End-to-End Data Management: Enterprises should establish a similar data flow management system. From pre-training to post-training, the data ratios, quality scores, and decontamination rules at each stage must be version-controlled. It is recommended to use object storage services to uniformly manage massive training data and trigger automated cleaning pipelines via message queues.
2. Elastic Training Infrastructure: The reinforcement learning stage (such as RLVR) requires extensive parallel environment interactions, demanding highly elastic scaling of computing power. It is advisable to deploy training tasks on container platforms, utilizing their elastic scaling capabilities to handle the fluctuating compute demands of the RL stage, and tracking gradient changes and reward curves in real-time through logging services and monitoring alerts.
3. Intermediate Checkpoint Evaluation: Drawing from the transparency concept of full-stack open source, internal enterprise training should also retain intermediate checkpoints. This not only allows for rollback debugging but also enables a deep analysis of the impact of data ratio changes by comparing model performance across different stages, thereby guiding the next training iteration.

5. Conclusion

The technical architecture of full-stack open source LLMs completely reveals the underlying logic of how contemporary models transition from “fluent conversation” to “deep reasoning.” This is achieved through fine-grained data control (quality upsampling, decontamination), innovative stage sequencing (forward deployment of post-training data, mid-training domain balancing), and the deep evolution of post-training algorithms (Delta Learning, KL-divergence-free RLVR). For developers and enterprises, deeply understanding this full-stack paradigm and applying it within cloud-native engineering practices is the key to building core competitive LLMs.