How Open-Source LLMs Evolved: A Technical Breakdown from Data Scaling to Agentic AI

The pace of open-source large language model development has outstripped most expectations. Looking at the iteration history of this field reveals not only the internal logic of technical progress but also a set of general methodologies behind the continuous capability gains. Rather than discussing individual model versions, this article takes a systematic technical view and breaks the main evolution line into four interconnected dimensions: data scaling, architecture scalability and efficiency optimization, post-training paradigm upgrades, and environment scaling with agentic capability building.

1. Data Scaling: From “Piling Up Tokens” to Structured Design

Early pre-training for large models was relatively straightforward: gather as much public text as possible, clean it, and feed it directly to the model. The core metric at this stage was total token count. First-generation models typically used around 3T tokens of pre-training data, enough to cover web documents, encyclopedias, books, and code, primarily in Chinese and English. In subsequent iterations, teams quickly discovered that simply chasing data volume ran into diminishing returns—models had essentially “read” all the high-quality text on the internet, and continuing to feed in low-quality, duplicated data only increased training costs and could even degrade performance.

Data strategy therefore entered a stage of refined operations. On one hand, token counts kept climbing steadily, from 7T+ to 18T and then to 36T in the latest generation, while language coverage expanded from a handful of mainstream languages to 119 languages and dialects, laying the foundation for multilingual applications. On the other hand—and more importantly—the “structural design” of the data itself produced two notable technical branches:

• Multi-stage pre-training: Instead of mixing all data in a single training run, the process is divided into multiple stages with progressively adjusted data mixtures. A common approach is to start with general corpora for foundational learning, then apply incremental training on logic-dense data such as code and math, and finally introduce instruction and conversational data for final alignment. Staged training allows the model to acquire knowledge along different dimensions at different stages, preventing noisy data from interfering with critical capabilities the way it can in single-run mixed training.

• Deep integration of high-quality synthetic data: The era of relying entirely on human annotation or natural corpora is over. The current mainstream approach is to train a strong “teacher model” that generates structured samples with reasoning traces, then use filtering mechanisms to extract high-quality subsets for the training set. The biggest advantage of synthetic data is controllability—it enables the precise construction of sample types that are scarce in natural corpora, such as long-chain reasoning samples and multi-step tool-calling samples.

2. Architecture Evolution: From Dense to Highly Sparse MoE and Hybrid Attention

Architectural iteration is the physical foundation of model performance gains. First-generation models used a standard Decoder-only Transformer, combined with proven components such as SwiGLU activation, RMSNorm normalization, and RoPE relative position encoding. Given the compute available at the time, this design balanced training stability with inference efficiency.

The core tensions driving architectural evolution have always centered on two questions: How do you control inference costs as parameter counts grow? And how do you maximize model expressiveness within a fixed compute budget?

From MHA to GQA: The First Relief from Memory Pressure

The original Transformer uses multi-head attention (MHA), where each head maintains its own Key/Value cache. As context length grows, KV cache memory usage expands linearly, directly limiting the maximum sequence length a model can handle. GQA (Grouped-Query Attention) emerged in response: Query heads are grouped, and heads within a group share the same set of Keys and Values. This significantly reduces KV cache usage while incurring almost no accuracy loss on most tasks, and GQA has become a standard configuration in virtually all subsequent versions.

Dense and MoE: Two Parallel Routes

For scaling total parameters, the industry split into two technical routes:

• Dense models: Every token activates all parameters, so compute grows linearly with parameter count. The advantages are training stability and predictable inference behavior, making dense models suitable for latency-sensitive scenarios.
• MoE (Mixture of Experts) models: FFN layers are split into multiple expert sub-networks, and each token activates only a small subset of experts. This achieves the effect of “large total parameters, small actual compute.” A typical configuration is a model with 30B total parameters but only 3B activated per token, which significantly outperforms a dense model of comparable activated size.

Notably, MoE architectures suffer from training instability at high sparsity—some experts may “starve,” meaning they rarely receive gradient updates. Later versions introduced more refined load-balancing losses and expert initialization strategies to ensure all experts are adequately trained.

Hybrid Attention: The Next Architectural Direction

The latest architectural exploration goes beyond traditional attention. Hybrid Attention architectures combine standard softmax attention with linear attention (such as Gated DeltaNet): tokens requiring precise positional awareness and local association use traditional attention, while global information aggregation and long-range dependency modeling go through the linear attention channel to save compute. This parallel hybrid design essentially lets the model control its “attention budget” at a finer granularity, seeking a better trade-off between compute cost and expressiveness.

3. Post-Training: From Single-Round Alignment to Multi-Stage Reinforcement Learning

Pre-training determines a model’s knowledge ceiling; post-training determines its behavior and usability. Early post-training pipelines were simple: SFT (supervised fine-tuning) plus RLHF (reinforcement learning from human feedback). SFT taught the model the format of “speaking,” while RLHF made outputs better match human preferences. Both have limitations: SFT depends on a limited pool of annotated data, making models prone to “recitation-style” responses, while RLHF requires extensive human preference labeling, which is costly and struggles to cover long-tail scenarios.

Combining Scaled SFT with Multi-Stage RL

The first step in the evolution was a massive expansion of SFT. Some versions now use more than one million supervised samples covering instruction following, role-play, structured output, multi-turn dialogue, and other formats. But this is not simply a matter of adding annotation headcount—automated construction methods play a large role, such as model self-generation, rewriting, and expansion to grow the training corpus.

After that, the reinforcement learning stage became layered:

• Offline RL: No environment interaction; the model learns preference rankings directly from a fixed set of candidate samples. A typical example is DPO (Direct Preference Optimization), which optimizes the policy directly through an implicit reward function, eliminating the need to train an explicit reward model and greatly simplifying the training pipeline.
• Online RL: The model continuously samples new responses during training, a reward model scores them, and gradients are propagated back in real time. A typical example is GRPO (Group Relative Policy Optimization), which replaces absolute reward values with group-relative advantage estimates, effectively reducing reward model variance.

Result-Verification RL: The Key to Deeper Reasoning

In domains like math and code, where outcomes can be automatically verified, RL based on result verification has proven effective at significantly deepening model reasoning. Its core logic no longer relies on human preference labels; instead, multiple candidate reasoning paths are constructed, and only paths whose final answers are correct receive positive reinforcement. The model gradually learns to “search” internally among multiple solution approaches and converges on strategies with higher average success rates. This paradigm is one of the sources of today’s “slow thinking” capabilities.

Unifying Thinking and Non-Thinking Modes

An important product-level innovation is unifying “fast answer mode” and “deep reasoning mode” within a single model. The model has a built-in Thinking Budget parameter that controls whether deep reasoning is engaged and caps the token budget for reasoning. When the budget is 0, the model answers directly; when the budget is large, the model first generates an internal reasoning chain before making a decision. This design hands the choice to the user, balancing the low-latency needs of everyday interactions with the high-accuracy needs of complex problems.

4. Environment Scaling and Agentization: From “Writing Code” to “Getting Things Done”

If the capabilities of the first three generations of models were mainly about “understanding” and “generation,” the current competitive focus has shifted to “action.” The industry consensus is that the core competitiveness of next-generation LLMs lies in agent capabilities—whether a model can autonomously execute multi-step tasks, call tools, and adjust its strategy based on environment feedback in real or simulated environments.

Agentic Coding: From Generating Code to Operating Code

Traditional code generation produces a static code snippet. Agentic coding, by contrast, requires the model to operate within a complete software engineering environment. To train this capability, teams built training infrastructure with tens of thousands of parallel execution sandboxes. During training, the model must complete the full loop: understand requirements → write code → run tests → read error messages → fix code → test again. This process essentially transfers the iterative debugging skills of software development to the model. It is less about “writing code” and more about “doing development.”

Long-Horizon Agent RL addresses the credit assignment problem in long-chain tasks—in an operation sequence of hundreds of steps, how do you identify which step caused the final success or failure? This requires RL algorithms with stronger temporal attribution capabilities.

AgentWorld: Language World Models Cut Sampling Costs

Online RL sampling in real environments is extremely expensive, especially for code execution and browser interactions, where each trial requires waiting hundreds of milliseconds or even seconds for real feedback. To ease this bottleneck, frontier work is building “language world models”—using a language model itself to simulate tool execution results and environment feedback. For example, when an agent performs a file read operation, instead of calling the real file system, the world model generates simulated feedback such as “read succeeded, file contents are…”. This approach essentially moves the RL trial-and-error process into a virtual space, enabling near-real-time sampling and training and greatly improving training throughput.

5. Summary and Outlook

Looking back at this evolution, one clear underlying logic emerges: every leap in model capability is essentially a breakthrough against a constraint in some dimension. Data constraints were broken by multi-stage pre-training and synthetic data; compute constraints were eased by MoE and hybrid attention; annotation constraints were bypassed by result-verification RL and preference optimization; and environment interaction constraints were weakened by parallel sandboxes and world models.

For developers and researchers, the value of understanding this trajectory is that you no longer need to be distracted by a flood of version numbers. Focus on four core variables: the composition and scale of the training data, the sparsity and attention form of the model architecture, the reinforcement learning objectives in post-training, and the depth of the model’s interaction with the real environment. Master the direction of these four levers, and no matter what new version appears, you can quickly locate its technical position and assess its potential impact.