How Large-Scale Reinforcement Learning Drives Capability Leaps in Foundation Models

1. Background: Open-Weight Models Are Rapidly Approaching the Closed-Source Frontier

Over the past two years, a clear theme has emerged in the foundation model industry: the gap at the pretraining stage is gradually converging, while capability differentiation at the post-training stage has become the new competitive focus. Composite intelligence indices from third-party evaluations show that the gap between leading open-weight models and the strongest closed-source models has narrowed from double digits to single digits. Some open-weight models had already ranked among the global leaders in evaluation scores before their weights were officially released, becoming some of the highest-scoring open-weight models available.

The technical logic behind this phenomenon deserves closer analysis: the capability leap does not come from larger pretraining scale, but primarily from sustained investment in large-scale reinforcement learning during the post-training stage. This article analyzes the topic across five dimensions: training paradigms, engineering practices, capability characterization, benchmark interpretation, and deployment recommendations.

2. Why RL Post-Training Has Become the Core Engine of Capability Leaps

2.1 The Division of Labor Between Pretraining and Post-Training

Pretraining compresses knowledge through next-token prediction over massive corpora, determining the model’s knowledge ceiling. However, for “how to do things correctly”—step-by-step reasoning, tool use, following complex instructions—the pretraining objective itself does not directly optimize these behaviors. The post-training stage fills these gaps through two phases: supervised fine-tuning (SFT) and reinforcement learning (RL). SFT establishes basic behavioral patterns, while RL solidifies correct behaviors into stable policies through trial and error.

2.2 Verifiable Rewards: From “Alignment” to “Capability”

Early RLHF primarily addressed human preference alignment. Recent engineering practice shows that when reward signals can be programmatically verified, the leverage of reinforcement learning on capability improvement far exceeds expectations:

• Code tasks: Using unit test pass rates as rewards—objective and irrefutable;
• Math tasks: Using correctness verification of final answers as rewards;
• Tool calling: Using task completion and parameter format compliance as rewards.

This type of “verifiable reward” shifts the optimization target from “what humans think is good” to “what is objectively correct,” enabling the model to solidify effective reasoning chains through thousands of trials rather than merely learning to produce pleasing expressions.

2.3 Scale Is the Key Variable

Small-scale RL typically yields only a few percentage points of improvement, and inconsistently. Only when compute investment is substantially increased and training environments and iterations are expanded in tandem does capability improvement take on “leap” characteristics. Industry observers generally agree that there is a significant positive correlation between post-training RL compute investment and final capability—this is precisely why leading teams continue to increase investment in this area.

3. Multi-Domain Joint Training: Engineering Practices Across Hundreds of Downstream Task Domains

3.1 Task Domain Coverage and Ratio Scheduling

Post-training for frontier models is typically no longer limited to a single domain, but involves joint training across hundreds of downstream task domains. Typical coverage includes:

• Code generation and real-world software engineering bug fixing;
• Mathematical reasoning and logical deduction;
• Agent tasks: multi-step tool calling, long-horizon planning, environment interaction;
• Knowledge QA, long-text understanding and summarization;
• Multi-turn instruction following and safety alignment.

The core challenge of joint training is ratio scheduling: if one domain’s weight is too high, other domains degrade (catastrophic forgetting); if too low, the gains are insignificant. In practice, dynamic ratio scheduling is commonly used, adaptively adjusting sampling probabilities based on each task domain’s pass rate curve, giving “slower-learning domains” more training opportunities.

3.2 Reward System Design

• Rule-based rewards: Deterministic rules for programmatically verifiable components (test passage, answer matching, format compliance);
• Model-judged rewards: Open-ended tasks (writing, summarization) scored by high-quality judge models;
• Combined process and outcome rewards: Focusing on both final results and constraints on intermediate reasoning steps.

Particular attention must be paid to preventing “reward hacking”: models may learn to exploit verifier loopholes, such as hardcoding test cases or producing content that appears compliant but is actually empty. Common countermeasures include decoupling training from verification, periodically updating verifiers, and introducing adversarial probe tasks.

3.3 Training Infrastructure

Large-scale RL places far greater demands on infrastructure than SFT:

• Sampling throughput: Large volumes of parallel rollouts are needed to generate candidate trajectories; inference-side compute is often several times that of the training side;
• Asynchronous pipelines: Decoupling environment interaction, reward computation, and parameter updates to avoid compute idle time caused by synchronous waiting;
• Trajectory governance: Quality filtering and deduplication of experience replay to prevent low-quality trajectories from polluting the policy.

These workloads typically run on high-performance GPU clusters, with container platforms orchestrating mixed training and inference workloads, object storage managing massive corpora and checkpoints, and monitoring and alerting systems ensuring stability during long training runs.

4. Three Manifestations of Capability Leaps: Code, Agent, and Knowledge

4.1 Code Capability

Code is the domain where rewards are most verifiable, so RL gains are most direct. Models can now not only generate code at the standalone function level, but also approach the strongest closed-source models on real repository-level bug-fixing tasks. This requires comprehensive capabilities in cross-file understanding, bug localization, and minimal-change fixes.

4.2 Agent Capability: The Most Significant Dimension of Improvement

Agent tasks comprehensively test planning, tool calling, error recovery, and long-context management—making them the most significantly improved dimension in this round of capability leaps. Key enablers include:

• Million-token context windows: Capable of holding complete trajectories of long-horizon tasks, large tool return results, and multi-file code context—this has become a general configuration trend for frontier models;
• Structured tool calling: Outputting standardized parameters through function calling protocols, significantly reducing parse failure rates;
• Reflection and retry: Automatically correcting tool invocation methods after errors, rather than abandoning the task.

A minimal tool-calling agent loop can be summarized as:

# Illustrative code: minimal tool-calling agent loop
SYSTEM = "You are a rigorous assistant: reason step by step first, call tools when external information is needed, then give the final answer."
def run_agent(query: str, max_steps: int = 8) -> str:
    messages = [{"role": "system", "content": SYSTEM},
                {"role": "user", "content": query}]
    for _ in range(max_steps):
        resp = llm.chat(messages)              # Call the LLM
        if not resp.tool_calls:                # Model considers information sufficient
            return resp.content                # Output final answer
        for call in resp.tool_calls:           # Execute tools requested by the model
            observation = execute(call.name, call.arguments)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": observation,        # Feed back observation results
            })
    return "Maximum steps reached; task could not be completed within the allowed iterations."

The value of this loop lies in the fact that the model corrects its strategy based on real observations at each step—precisely the behavioral pattern specifically optimized during the reinforcement learning stage.

4.3 Knowledge Capability

By injecting high-density knowledge corpora during post-training and combining with retrieval augmentation, model performance on knowledge-intensive tasks improves in tandem. However, note that the ceiling for knowledge-related capabilities is still determined by pretraining; post-training’s role is “activation and organization,” not “creating something from nothing.”

5. Benchmark Interpretation: How to Reasonably Assess Intelligence Indices and Benchmark Scores

5.1 Composition and Interpretation of Composite Intelligence Indices

Third-party intelligence indices typically aggregate multiple benchmark categories—code, math, agent, knowledge—with weights, reflecting balanced capability but also masking individual weaknesses. When interpreting, note the following:

1. Look at distribution, not just total score: Different models may show strengths and weaknesses across agent and knowledge dimensions;
2. Pay attention to score stability: The same model’s scores fluctuate across different sampling temperatures and prompts;
3. Beware of benchmark contamination: Whether training data includes evaluation sets directly affects score credibility.

5.2 Industry Implications of the Narrowing Gap

When the gap between open-weight and closed-source models narrows to single digits, it means that for most general tasks, open-weight models have become credible alternatives. For enterprises, the direct value is reflected in three areas: private deployment is feasible, sensitive data need not leave the premises; domain fine-tuning based on open weights enables differentiated capabilities; and the technology stack is autonomous and controllable, avoiding single-vendor lock-in.

6. Enterprise Deployment Recommendations

1. Evaluate first: Build private evaluation sets aligned with your business, covering real task distributions—avoid relying solely on public leaderboards;
2. Adopt in phases: Start with scenarios with higher fault tolerance, such as code assistance and internal knowledge bases, then expand to core business;
3. Infrastructure selection: Training and inference workloads differ greatly; leverage general-purpose cloud platform capabilities in combination—for example, container services for workload orchestration, object storage for corpora and checkpoints, and model inference services for online traffic (cloud platforms such as Baidu AI Cloud offer these general-purpose categories of capabilities);
4. Balance cost and latency: Long contexts and multi-step agents significantly amplify inference costs; control overhead through context trimming, result caching, and tiered model routing;
5. Security and compliance: Open weights do not mean no responsibility—content safety, access control, and operation auditing mechanisms are still required.

7. Outlook

The scaling of post-training reinforcement learning has only just entered deep waters. Foreseeable directions include: automatic generation and curriculum design of task environments, finer-grained process reward modeling, multi-agent collaborative training, and incorporating verifiers themselves into the learnable scope. The gap between open-weight and closed-source models will likely narrow further, with the competitive focus shifting from “single-point scores” to a comprehensive contest of “engineering usability, cost efficiency, and safety controllability.” For developers, now is the best window of opportunity to build evaluation systems and accumulate agent engineering experience.