Post-Training with Reinforcement Learning: The Path to Upgrading Open-Source LLM Programming Capabilities

1. Background: The “Last Mile” for Open-Source Models is Post-Training

Over the past year, the parameter scale of open-source foundation models has grown from hundreds of billions to trillions. In “exam-style” tasks like knowledge QA and code completion, the gap with top-tier closed-source models has narrowed significantly. However, placing these models into real-world engineering environments—asking them to independently log into a terminal for troubleshooting, maintain a code repository over multiple continuous rounds, or generate an interactive 3D scene from a single prompt—immediately reveals a stark difference.

A representative observation: the initial version of a massive open-source foundation model could outperform several closed-source models on “geek-level” tasks like operator optimization and compiler principles. Yet, on terminal benchmarks that test fundamental command-line skills, it scored just over 10%. This “imbalanced proficiency” points to a clear conclusion: the generational gap lies not in pre-training, but in post-training. Pre-training determines what a model “knows,” while post-training determines what a model “can do”—whether it can stably invoke tools, plan multi-step tasks, and maintain goal alignment in long-horizon loops. These capabilities must be solidified during the post-training phase using reinforcement learning (RL).

2. Three-Month Evolution Review: A Two-Track Race

Reviewing recent public developments from a leading open-source model team, their efforts can be categorized into two parallel tracks:

• External Tooling Track: Continuously iterating on intelligent programming tools, embedding command-line capabilities into graphical interfaces, building in browsers with visual annotation, and offering dual plan/goal modes to engineer for long-horizon Agent tasks.
• Internal Model Track: Rather than rushing to build a larger next-generation foundation model, the team spent nearly a hundred days on RL post-training for the existing foundation model, covering over a hundred domains.

There were two interesting side notes during this period: the team used the digits of Pi as an easter egg to tease the version number, which the community instantly understood; and an anonymous model topped third-party arenas, was identified by the community through its output style “fingerprint,” and was then officially announced. This rhythm of “anonymous blind testing first, official release second” is becoming the standard practice for top teams to validate model capabilities.

To summarize the technical strategy of this upgrade: retain the full-parameter foundation skeleton, spend compute on joint RL re-training, and significantly reduce API prices. This is a pragmatic route of “not piling on parameters, but piling on post-training.”

3. Technical Breakdown: What Exactly Does RL Post-Training Do?

1. Verifiable Rewards: Code is RL’s Natural Home

Math and coding tasks have become the main battlegrounds for post-training because they offer objective, automatically verifiable result signals: does the program compile, do unit tests pass, is the terminal task completed? Compared to reward models dependent on human preferences, Verifiable Rewards (RLVR) are harder to pollute by “sycophancy.” An illustrative reward function structure looks like this:

def reward(task, trajectory) -> float:
    # 1) Hard result reward: fail directly if tests do not pass
    if not run_tests(task.repo, trajectory.patch):
        return -1.0
    r = 1.0
    # 2) Process reward: tool usage efficiency, number of invalid retries
    r += 0.2 * tool_efficiency(trajectory)
    # 3) Anti-reward hacking: heavily penalize tampering with assertions or copying test cases
    if tampered_assertions(trajectory.patch):
        r -= 2.0
    return r

(The above is an educational illustration; the reward design in real systems is far more complex.)

2. Multi-Domain Joint Training: Preventing “Gain One, Lose Three”

If RL is applied solely to code, the model will easily regress in other capabilities—catastrophic forgetting and reward hacking are the two major scourges of post-training. The industry standard is joint training across over a hundred domains: code, math, multi-turn instruction following, tool invocation, writing, and safety alignment enter the reward system simultaneously. A unified KL constraint is used to hold the policy in check, preventing the model from sacrificing general capabilities to maximize a single reward. This also explains why the upgraded version saw a massive leap in programming ability while its third-party comprehensive intelligence index simultaneously grew by double digits, placing it in the global first tier.

3. Agent Environment and Toolchain Synergy

Realizing the benefits of post-training requires an environment. Sandboxes, terminals, browsers, and visual feedback form a closed loop for training and evaluation: the model executes commands in the sandbox, observes errors, modifies and retries, and the complete trajectory directly becomes the RL training signal. On the tool side, the desktop GUI exposes this capability to users unfamiliar with the command line; the dual plan/goal modes correspond to “task decomposition” and “goal maintenance,” which are key designs to keep long-horizon Agents from drifting off target.

4. General Engineering for Training Stability

The engineering difficulty of RL post-training lies in stability: reward curve oscillations, sudden policy collapse, and VRAM explosions from long trajectories are commonplace. The industry commonly uses curriculum learning (assigning tasks from easy to hard), asynchronous rollout infrastructure, and a mix of rule-based and model-based rewards to flatten the curve. There is no “black magic” here; it is mostly system engineering effort.

4. Evaluation Perspective: How to Scientifically Measure 3D and Frontend Capabilities

This upgrade’s performance on the visual generation side is particularly impressive: generating interactive 3D scenes from a single prompt, and outputting complete, runnable frontend pages in one go. To measure such capabilities, it is recommended to build your own evaluation framework rather than relying solely on leaderboards:

Dimension	Focus	Common Methods
Functional Correctness	Can the code compile/run, do tests pass	Automated sandbox execution
Interaction Completeness	Are buttons, animations, state transitions usable	Headless browser traversal
Visual Fidelity	Deviation of layout and color from expectations	Screenshot comparison + vision model scoring
Code Quality	Readability, component separation, maintainability	Static analysis + manual spot-checks
Comprehensive Intelligence	Cross-domain reasoning and instruction following	Third-party anonymous blind testing

Practical advice: leaderboards should only be used to narrow down candidates. Final selection must use your own business’s desensitized real-world tasks (e.g., 20 to 50 typical tickets) to build a private evaluation set, focusing heavily on three metrics: pass rate, average token consumption, and end-to-end latency.

5. Cost Perspective: Why Flagship Capabilities Can Be Slashed

Post-training does not change the parameter scale, so the inference cost structure remains the same, yet prices can be significantly reduced (this upgrade saw API prices drop by roughly a third). There are three reasons behind this:

1. Continuous inference-side optimization: Techniques like quantization, speculative decoding, KV cache reuse, and continuous batching compress the cost per token.
2. Competition from open weights: Users can take the weights for private deployment at any time, meaning pricing power is no longer held unilaterally by the vendor.
3. Economies of scale: As usage volume increases, unit inference costs are further amortized.

It is also important to remind developers: Agent scenarios have a token amplification effect—a single long task may generate dozens of rounds of tool calls, consuming far more tokens than a simple Q&A. Cost governance should be tackled from three angles: prompt and context caching, model routing based on task complexity (sending simple tasks to smaller models), and batching with budget alerts.

6. Developer Implementation Advice

1. Rely on your own evaluations for model selection, not just leaderboards; focus on testing long tasks and terminal operations, as these areas differentiate models the most.
2. Toolchain division of labor: Use CLI for batch processing and automation, and GUI for exploration and review; manage long-horizon tasks using the dual plan/goal modes to prevent target drift after multiple iterations.
3. Deployment models: Open weights support private deployment, making them suitable for teams with compliance and data isolation requirements. If using the cloud, platforms like Baidu AI Cloud offer general capabilities such as container services, GPU compute, object storage, and monitoring, allowing you to quickly build evaluation and inference environments without setting up your own data center.
4. Progressive replacement: Run shadow traffic with the new model first, compare it against the live model for one to two weeks, and then gradually switch over by task category to minimize risk.

7. Conclusion

The signal from this upgrade is clear: the competitive focus between open-source and closed-source has shifted from “who has the bigger foundation” to “who has the more aggressive post-training”. Pre-training determines the lower bound, post-training determines the upper bound; open weights determine whether enterprise users can truly hold these capabilities in their own hands. For developers, now is a great time to experience flagship-level programming capabilities at an extremely low cost—but step into the arena with your own evaluation sets and cost ledgers, rather than just blind faith in leaderboards.