Reinforcement Learning Post-Training: How Open-Source LLMs Are Closing the Gap with Closed-Source Models

1. Shifting Focus: From Pre-Training Scale to Post-Training RL

In the first half of LLM development, capability improvements were almost entirely driven by scaling laws in the pre-training phase: larger parameters, more data, and increased compute. However, as the utilization of high-quality public corpora approaches its limits and marginal returns diminish, the industry’s focus has shifted to the post-training phase. Recent model releases demonstrate a highly consistent technical choice: without significantly changing the base model, developers are investing several times—or even dozens of times—more compute into post-training RL. This involves large-scale joint training across hundreds of downstream task domains, including coding, mathematics, agents, and general knowledge, ultimately leading to a comprehensive leap in overall intelligence.

The difference between this “joint training” and traditional “stage-by-stage fine-tuning” is notable. Traditional approaches often proceed with instruction tuning, followed by preference alignment, and finally patching individual capabilities. Tasks easily interfere with each other, leading to catastrophic forgetting. Joint training, conversely, mixes hundreds of task domains into a single training recipe, balancing capability growth through dynamic sampling and reward weighting, preventing the “whack-a-mole” effect.

For developers, the implication is that under the same base model, differences in “post-training recipes” are sufficient to create a perceptible capability gap. Evaluating a model solely by parameter size is no longer sufficient to judge its true capability.

2. Technical Core: How RL Post-Training “Manufactures” Capability

Post-training RL is not a single algorithm, but a set of training paradigms organized around “reward signals.” Based on the source of rewards, they can be categorized into three types:

1. Preference Rewards (RLHF): Human annotators or weak supervision models rank responses. A reward model is trained first, followed by using RL to optimize the policy model. This is suitable for open-ended dialogue, tone, safety, and other dimensions that are difficult to quantify.
2. Verifiable Rewards (RLVR): Tasks like code unit testing, math answer checking, and output format validation have objective criteria. Rewards can be calculated directly by rules without needing a reward model. This type of signal is the cleanest and hardest to “game,” making coding and math the areas where post-training RL yields the most significant gains.
3. Process and Trajectory Rewards (Agent): For multi-step tool-calling tasks, rewards are defined over the entire execution trajectory—whether the task is closed-loop, whether steps are redundant, and whether boundary-violating operations occurred.

A simplified verifiable reward training flow is as follows (conceptual illustration, not a specific implementation):

for batch in train_loader:
    prompts = batch["prompts"]
    # 1. Sampling: The policy model generates K candidate responses for the same prompt
    responses = policy_model.generate(prompts, num=K, temperature=0.8)
    # 2. Reward Calculation: Verifiable tasks use rule-based scoring; non-verifiable tasks fall back to reward model scoring
    for prompt, resp in zip(prompts, responses):
        if prompt.task == "code":
            reward = run_unit_tests(extract_code(resp))
        elif prompt.task == "math":
            reward = 1.0 if answer_matches(resp, prompt.gt) else 0.0
        else:
            reward = reward_model.score(prompt, resp)
    # 3. Advantage Estimation: Uses in-group relative performance instead of an external value network to reduce training costs
    advantages = group_relative_advantage(rewards)
    # 4. Policy Update: Applies distribution constraints anchored to the frozen base model to prevent language capability degradation
    policy_model.update(prompts, responses, advantages, ref_model=frozen_base)

Two engineering details are worth emphasizing. First, the “in-group relative advantage” approach allows multiple responses to the same question to serve as references for each other, drastically reducing the reliance on an additional value network and making large-scale parallel training computationally feasible. Second, during training, the frozen base model must be used as an anchor to apply distribution constraints (e.g., KL penalties). Otherwise, the model is prone to overfitting on RL signals, leading to language degradation and broken output formats.

3. The Open vs. Closed Source Gap: Why It Narrowed Rapidly

The “intelligence index” maintained by several third-party evaluation organizations (a weighted score combining dimensions like code, math, knowledge QA, and agents) has long been a benchmark for measuring the gap between open-weight and closed-source models. Looking at trends over the past year, a clear shift is occurring: the score gap between top-tier open-weight models and the strongest closed-source models has narrowed from double digits to single digits. In coding, knowledge, and certain agent benchmarks, open-weight models have reached or approached the level of the strongest closed-source models.

The technical reasons behind this are not mysterious:

• Diffusion of Training Methodologies: “Recipes” like verifiable rewards and large-scale joint training are fundamentally methods, not proprietary assets. Once proven effective, they are rapidly replicated across the industry.
• Scaled Supply of Inference Compute: The bottleneck for RL post-training has shifted from “data” to “sampling and inference compute.” The widespread availability of cloud GPU resources has made it affordable for open-source teams to conduct training at comparable scales.
• Iterative Amplification by the Open Ecosystem: Community-driven fine-tuning, quantization, and deployment optimizations of open-weight models further amplify the practical usability of base capabilities.

For enterprises, this means the old assumption that “open source will inevitably lag behind closed source by a generation” is failing. However, the single-digit gap also reminds us that differences still exist on the most difficult tasks. Model selection must be based on the actual performance of your specific tasks, not just leaderboard rankings.

4. Agent Capabilities: The Stronghold of RL Gains

Among various capabilities, Agents (autonomous planning, multi-step tool calling) represent the area with the most significant gains from post-training RL. This is because the feedback format of agent tasks naturally aligns with RL:

• Environment as Feedback: Whether a tool call succeeds or whether a task is ultimately completed is inherently a computable reward signal.
• Trajectory-Level Optimization: RL optimizes the entire decision chain, not single-step text quality, perfectly matching the “multi-step planning + dynamic correction” nature of agents.
• Failure Samples Have Value: SFT can only learn from successful trajectories, whereas RL can extract negative gradients from failures to learn “what to avoid.”

An illustration of trajectory reward design is as follows:

def trajectory_reward(traj):
    """Agent trajectory reward design example: results-first, process-supplementary"""
    reward = 0.0
    if task_completed(traj.final_state):          # Result reward: task must be closed-loop
        reward += 1.0
    reward -= 0.05 * traj.num_steps               # Step penalty: suppress ineffective actions
    if traj.has_forbidden_action():               # Boundary penalty: e.g., dangerous commands
        reward -= 0.5
    reward += 0.1 * traj.tool_call_success_rate   # Process reward: encourage accurate tool usage
    return reward

When evaluating agent capabilities, developers should focus on four quantifiable metrics: tool selection accuracy, parameter filling correctness, multi-step task completion rate, and error recovery rate (whether the model can self-correct and retry after a tool error). Many evaluations only look at end-to-end success rates, which can mask the problem of “low first-try success rate, surviving through repeated retries”—a shortcoming that most severely impacts user experience and costs in production environments.

5. Trillion Parameters and Million-Token Context: Engineering Realities

Two hard metrics presented by current top-tier models—parameter scales moving toward trillions and context lengths expanding to millions of tokens—have a direct impact on deployment choices:

1. Trillion parameters does not mean undeployable. Models at this scale almost exclusively use sparse Mixture-of-Experts (MoE) architectures, activating only a small fraction of parameters during inference. Combined with quantization and KV cache optimizations, open-source inference solutions can deliver acceptable throughput on multi-GPU servers. However, memory and bandwidth remain hard constraints. When estimating resources, base calculations on “activated parameters + concurrent throughput,” not total parameters.
2. Million-token context does not mean it should be fully utilized everywhere. Long context significantly reduces the complexity of slicing and retrieval in multi-document tasks, but inference costs scale linearly with token count. In practice, a more common strategy is a hybrid architecture of “long context + retrieval augmentation”: recall is handled by the retrieval system, while the model context only carries content that truly requires global understanding.
3. Cost Structure Dictates the Route: Self-deploying an open-weight model represents “fixed cost + operational cost,” whereas calling a closed-source API represents “variable cost based on usage.” The higher the daily call volume and the stricter the data compliance requirements, the more obvious the comprehensive advantage of self-deployment.

6. Four Practical Recommendations for Developers and Enterprises

• Don’t just look at a single index ranking. The intelligence index is a weighted average that can mask uneven capability distribution. The correct approach is to select 3-5 public benchmarks closest to your business for horizontal comparison, and then build a custom evaluation set with hundreds of real business samples for retesting.
• Use a “capability radar chart” instead of a “total score” for decisions. Score coding, math, knowledge QA, tool calling, and multilingual capabilities separately, and increase the weight of dimensions that are critical to your business while penalizing shortcomings.
• Select deployment paths based on data sensitivity tiers. For prototyping with public data, prioritize using hosted APIs to accelerate iteration. For production pipelines involving core data, consider self-deploying based on open-weight models, leveraging cloud service providers like Baidu AI Cloud for general GPU compute, container platforms, and logging/monitoring infrastructure to lower the operational barrier.
• Reserve interfaces for capability upgrades. Model iteration speed has outpaced the adaptation speed of most business systems. It is recommended to abstract model calls into a unified gateway layer, shielding against differences in tool calling formats and context lengths across different models, facilitating seamless future replacements.

7. Conclusion

The single-digit intelligence index gap between open-weight and closed-source models is fundamentally the result of “methodology diffusion + compute equalization,” with large-scale RL post-training acting as the core engine of this catch-up effort. For developers, what truly matters is not the “ranking,” but understanding the technical logic behind the capability leap and establishing a proprietary, reproducible evaluation and selection system. As the gap narrows to single digits, the decision-making power is shifting from model vendors to teams that understand both business and engineering.