Large-Scale RL Post-Training: How Open Models Are Closing the Gap with Closed-Source Frontiers

1. A Paradigm Shift: The Main Battlefield Moves from Pre-Training to Post-Training

Over the past two years, the capability gap between open-weight large models and closed-source frontier models has visibly converged. Composite intelligence indices published by various third-party evaluation organizations show that the total-score gap between leading open-weight models and the strongest closed-source models has narrowed substantially, with the former reaching or approaching the level of the strongest closed-source models on code, Agent, and knowledge benchmarks.

The core driver of this change is not simply parameter stacking but a shift in training paradigm. As high-quality pre-training corpora approach exhaustion and the marginal returns of pre-training continue to decline, the industry has redirected compute toward post-training, especially large-scale reinforcement learning (RL). The underlying logic can be summarized in three points:

First, the base gap was never large. Open-weight bases at the trillion-parameter scale show no generational gap in pre-training quality compared with closed-source models; the historical gap mainly came from the disparity in post-training investment.

Second, post-training scales remarkably well. Adding multiples of RL compute on the same base continues to raise scores on reasoning, code, instruction following, and other dimensions, exhibiting a pattern akin to “post-training scaling.”

Third, methodology diffuses far faster than data barriers. Pre-training advantages depend on proprietary corpora, whereas RL post-training algorithms, reward design, and engineering practices spread rapidly through the community—making the open camp the most direct beneficiary.

2. The Technical Core of Large-Scale RL Post-Training

2.1 Algorithm Evolution: From Preference Alignment to Verifiable Rewards

The RL post-training technology stack has iterated rapidly over the past two years, and understanding this evolutionary thread is a prerequisite for understanding the capability leap:

The classic three-stage RLHF+PPO pipeline: supervised fine-tuning first, then reward model training, and finally policy optimization with PPO. Its advantage is generality; its drawback is that the reward model is easily “over-optimized” by the policy—the model learns to please the reward model rather than genuinely improve quality.

DPO-style offline preference optimization: skips the explicit reward model and constructs the loss directly from preference pairs. Engineering is simple and stable, but exploration capability is limited, making it hard to break through the ceiling of preference data.

GRPO-style group-relative methods: sample a group of responses to the same prompt and replace the independent baseline with intra-group relative advantage, eliminating the large critic network and significantly reducing memory and compute overhead. This is particularly suited to large-scale parallel rollout and has become one of the current mainstream choices.

Verifiable rewards (RLVR): for tasks that can be automatically judged—code (whether unit tests pass), math (whether the answer matches exactly), tool calls (whether JSON is valid and parameter types are correct)—objective signals serve directly as rewards. This mechanically compresses the room for “gaming the reward” and is key to the dramatic gains in code and reasoning capabilities.

2.2 Multi-Domain Joint Training: Countering the “Capability Seesaw”

Reinforcement in a single domain easily causes the model to fix one thing and break another—code gets stronger while writing degrades. The common engineering approach is joint sampling across hundreds of downstream task domains with dynamically adjusted quotas: saturated domains are downsampled and weak domains upsampled; the reward signal uses a hybrid system of “verifiable reward + preference reward + format reward”; and supervised fine-tuning data is periodically replayed to suppress catastrophic forgetting. This multi-domain joint training turns capability improvement from “single-point breakthroughs” into “overall elevation,” and is the direct reason for the systematic rise in composite intelligence index scores.

2.3 Infrastructure: Rollout Throughput Determines the Training Ceiling

An often-underestimated fact: the bulk of RL post-training compute is not in gradient updates but in massive rollout sampling. Training frameworks therefore typically decouple “generation” from “update”: the generation side uses a high-throughput inference engine, relying on general optimizations such as continuous batching and paged memory management to maximize sampling throughput; the training side runs asynchronously in a pipeline with the generation side, with weights synchronized via chunked broadcast; Agent-oriented tasks also require a sandboxed tool execution environment (code interpreter, retrieval, browser simulation, etc.) to ensure feedback is real and verifiable. The core loop can be abstracted as:

for step in range(total_steps):
    prompts = curriculum.sample(batch, weights=domain_weights)
    # Generation side: high-throughput engine runs parallel rollout; tools execute for real in the sandbox
    trajectories = engine.rollout(policy, prompts, tools=sandbox)
    # Reward side: verifiable tasks use rules; the rest go to the preference model
    rewards = [verifier(t) if verifiable(t) else rm.score(t)
               for t in trajectories]
    # GRPO-style update: intra-group relative advantage replaces an independent critic
    advantages = group_relative(rewards)
    policy.update(trajectories, advantages)
    # Dynamically rebalance per-domain sampling weights to prevent the capability seesaw
    domain_weights = rebalance(domain_weights, rewards)

3. How to Properly Interpret “Intelligence Indices”

Composite intelligence indices typically aggregate benchmarks across code, math, knowledge QA, instruction following, multilingual ability, and Agent tasks into a weighted total score for easy horizontal comparison—but three caveats apply when interpreting them. First, static question banks carry saturation and data-contamination risks, so scores may be inflated. Second, different organizations use different dimension weightings, so the same model may rank several places apart on different leaderboards. Third, a leaderboard measures a slice of general capability, not performance in a specific business scenario.

A practical recommendation is to build a small evaluation suite for your own business and compare using the same “weighted aggregation” idea:

weights = {"code": 0.3, "agent": 0.3, "knowledge": 0.2, "instruction": 0.2}
score = sum(run_suite(model, dim) * w for dim, w in weights.items())

Treating third-party indices as a “first screen” and your own evaluation as the “final review” is a more robust approach to model selection.

4. Why Agent Capabilities Improve Most

An Agent task is essentially long-horizon sequential decision-making: multi-turn tool calls, state maintenance, error recovery. This is naturally isomorphic to the RL objective of “maximizing long-term return,” so RL post-training yields far greater gains for Agents than for single-turn QA. This shows up at three levels: first, format stability of tool calls—JSON validity and parameter correctness are directly incorporated into the reward; second, task decomposition and planning—trajectory-level rewards encourage the model to break down before executing; third, self-correction—trajectories that proactively retry and correct the path after a failed execution receive higher returns, and the model gradually learns the “detect error—backtrack—switch path” behavior pattern. Combined with context windows of millions of tokens, an Agent can maintain more complete state memory within a single session, reducing external retrieval rounds—hence the improvement in task completion rates.

5. The Engineering Reality of Long Context: Fitting It In vs. Using It Well

“Fitting it in” and “using it well” are two different things for long context. The former relies on techniques such as positional encoding extrapolation, attention sparsification, and KV cache compression, solving memory and compute efficiency problems; the latter is constrained by phenomena such as “low utilization of mid-context information” and requires long-context post-training coordinated with retrieval augmentation. A common engineering strategy is tiered memory: recent trajectories kept in full, historical trajectories summarized and compressed, and key facts written to a retrieval store and recalled on demand. On the cost side, note that longer context means higher KV cache memory usage, so cache quantization and reuse strategies are needed to control inference overhead.

6. Developer Implementation Practices

6.1 Model Selection Checklist: confirm the commercial terms of the weight license; use third-party indices for a first screen and your own evaluation for the final review; assess inference cost in long-context scenarios.

6.2 Inference Optimization: weight quantization (INT8/INT4) and KV cache quantization, speculative decoding acceleration, and separated scheduling of long and short requests are the current general combination for reducing cost and improving efficiency.

6.3 Deployment Forms: enterprises can host open-weight models as containers on self-built GPU clusters, or leverage general infrastructure capabilities from cloud platforms such as Baidu AI Cloud (GPU containers, object storage, logging services, and monitoring/alerting) to reach production—model artifacts placed in object storage, inference services elastically scaled as containers, and link observability handled by logging and monitoring. The selection criteria at this layer are throughput, stability, and cost, not model capability itself.

6.4 Security and Compliance: when deploying production-grade Agents, configure content safety guardrails, output auditing, and tool-call allowlists to prevent the model from performing unauthorized operations during long-horizon decision-making.

7. Conclusion

The root cause of open-weight models approaching the closed-source frontier is that the center of gravity of capability barriers has shifted from “data and parameters” to “post-training methodology and compute engineering”—and the latter is far more replicable than the former. For developers, leaderboard scores are a starting point, not the endpoint; to truly capture the dividends of RL post-training capability, you need a solid evaluation system, fine-grained inference optimization, and robust deployment engineering. The next stage of competition will shift to real task completion rates in Agent scenarios—that is the final battlefield on which the quality of intelligence indices will be tested.