Advanced Post-Training: How Reinforcement Learning Unlocks LLM Potential

Over the past couple of years, the competitive focus of large language model (LLM) development has undergone a quiet but profound shift: moving from “larger pre-training” to “deeper post-training.” A recent upgrade practice by a leading team on their trillion-parameter flagship model provided a landmark footnote to this trend. Without launching a new round of massive pre-training and entirely reusing the existing foundation model, they significantly increased compute investment in the post-training phase and expanded reinforcement learning (RL) to hundreds of downstream domains. Consequently, the model’s capabilities in real-world programming tasks, professional document analysis, 3D modeling, and complex tool invocation saw remarkable leaps.

The engineering implications of this case are worth deconstructing by all practitioners: The potential stored in the foundation model during pre-training is far from being fully mined during post-training; what increasingly determines whether a model is “usable and useful” is the depth and quality of its post-training.

I. Why Post-Training Has Become the Main Battleground for Capability Leaps

1. Diminishing Marginal Returns in Pre-training. High-quality public corpora are approaching the limit of extraction, and the marginal value of incremental data continues to decline. The cost of a single pre-training run for a trillion-parameter model is exorbitant, leaving very limited room for trial and error. While the Scaling Law still holds, the data and compute investment required for an equivalent capability improvement are growing super-linearly. Continuing to aggressively scale up on the pre-training side is rapidly losing its cost-effectiveness.

2. Foundation Capability is “Potential,” Post-Training Determines “Usability.” What pre-training inherently endows a model with is a “probability distribution” of knowledge and capability. Post-training—comprising Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Reinforcement Learning with Verifiable Rewards (RLVR)—determines how much of this potential can be invoked stably and controllably. Practice repeatedly proves that the same foundation model, after undergoing different depths of post-training, can exhibit capability gaps large enough to obscure the scale difference of the foundation models themselves.

3. The Significant Leverage Effect of Post-Training. For most teams, a high-quality post-training iteration (taking weeks to months with manageable compute) delivers capability improvements in vertical domains that significantly outweigh the returns of investing the same cost into pre-training. This is the fundamental reason why the strategy of “reusing the foundation model and深耕 post-training” is being adopted by an increasing number of teams.

II. Post-Training Technology System: A Three-Tier Progression

Tier 1: Supervised Fine-Tuning (SFT). This shapes the model’s basic behavioral patterns through high-quality instruction data and chain-of-thought samples: instruction following, output formatting, and reasoning chain organization. SFT delivers quick results, but its capability ceiling is limited by the data distribution—it is essentially “imitation,” where the model only learns methods present in the data and struggles to surpass the demonstrator.

Tier 2: Reinforcement Learning from Human Feedback (RLHF). This involves training a reward model to fit human preferences, then updating the policy model using algorithms like PPO. It excels at handling tasks with high subjectivity that are difficult to verify programmatically, such as writing style and safety. The risk is that reward model biases will be amplified by the policy, requiring continuous human preference data回流 for calibration.

Tier 3: Reinforcement Learning with Verifiable Rewards (RLVR), the Current Core Engine. In objectively verifiable domains like code, math, and tool usage, programmatic signals—such as unit test pass rates, compilation execution results, exact answer matches, and tool invocation success—are used directly as rewards. The reward signal is clear, low-noise, and requires no manual annotation, making it the prerequisite for large-scale compute deployment in post-training. A typical training loop looks like this:

# RLVR training loop (pseudo-code)
for batch in task_sampler(
        domains=["code", "math", "doc_analysis", "tool_use", "ui_agent"],
        difficulty="adaptive"):          # Adaptive difficulty sampling
    responses = policy.generate(batch.prompts, temperature=0.7)
    # Verifiable rewards: unit test pass, compilation success, format compliance, tool goal achievement
    rewards = [verifier.score(p, r) for p, r in zip(batch.prompts, responses)]
    # KL penalty constrains the policy from deviating from the base distribution, protecting general capabilities
    loss = ppo_objective(policy, responses, rewards,
                         ref_model=frozen_base, kl_coef=0.05)
    optimizer.step(loss)

III. Scaling to Hundreds of Domains: Three Major Hurdles to Cross

1. Reward Signal Design. Verifiable domains use execution results directly as rewards. Semi-verifiable domains (like document analysis) can combine a tiered verification of “rule validation + a stronger model acting as a reviewer + manual spot-check calibration.” Purely subjective domains require vigilance against reward hacking—manifested as verbose outputs and increased sycophantic phrasing—which is typically mitigated by introducing length penalties and pairwise comparisons.

2. Task Sampling and Curriculum Design. When mixing training across hundreds of domains, the domain ratio directly dictates capability trade-offs (e.g., too many code tasks might squeeze out document analysis capabilities). In practice, dynamic sampling often targets a difficulty zone where the “model success rate is around 20% to 80%”: if it’s too easy, gradients vanish; if too hard, rewards are all zero. Simultaneously, a certain proportion of general tasks must be retained to prevent capability forgetting.

3. Training Stability. Gradient direction conflicts across multiple domains need to be mitigated through domain-level loss weighting or gradient clipping. KL constraints continuously limit the degree of deviation between the policy and the foundation model, preventing the model from “accommodating the reward” at the expense of generality.

IV. Four Pillars of Infrastructure: The Invisible Moat of Post-Training

A widely underestimated point in the industry is that the upper limit of post-training is largely determined by infrastructure.

1. Efficient Training Frameworks. The compute contention between rollout generation and parameter updates is the core contradiction in RL training efficiency. Mature frameworks perform heterogeneous scheduling: using high-throughput engines on the inference side for batch generation, synchronously updating on the training side, bridged by an accumulated trajectory buffer, and mixing inference and training GPUs to significantly boost overall compute utilization.
2. Long-Context Inference. The significance of a million-token context lies not in “fitting it in,” but in “using it well”—requiring validation across multi-dimensional tasks like cross-document retrieval, long-range consistency, and needle-in-a-haystack tests. Engineering relies on optimizations like paged KV Cache, context parallelism, and chunked prefill. Coupled with native visual understanding, models can directly read interface screenshots and document layouts, a key component for Agents perceiving real environments.
3. Sandbox Environments. Tasks like operating terminals, running code, and driving 3D modeling software or game engines must run in isolated sandboxes: file system and network isolation, resource limits, and execution timeouts. The sandbox is also a reward source—task execution results (compilation success, test passage, rendered artifacts meeting specs) directly convert into verifiable rewards. The fidelity gap between the sandbox and the real production environment determines the migration loss of training signals.
4. Task Scheduling Systems. Long-horizon, multi-step tasks exceeding dozens of steps require reliable orchestration: state persistence between steps, failure retries, parallel sub-task dispatch, and timeout circuit breaking. The scheduling system must serve both batch task construction and trajectory recycling on the training side, as well as ensure the stable operation of Agent services on the inference side.

V. How to Validate Capabilities: Moving from “Benchmarking” to “Working”

1. Real-Terminal Programming: End-to-end tasks in a real shell environment, including dependency installation, debugging, and multi-file modifications. Leading models have entered the 60%+ range on such benchmarks, with the true watershed being the ability to “not crash” throughout a multi-step task.
2. Professional Document Analysis: Long-document Q&A and structured extraction in fields like finance and law, focusing on numerical precision, table understanding, and cross-page consistency. This capability directly relies on the combination of long-context and RLVR.
3. Tool Invocation and 3D Modeling: Whether a model can drive professional software to complete modeling, scene setup, and rendering based on natural language specs is the touchstone for testing “language-to-operation” mapping capabilities.
4. Long-Horizon Software Engineering Tasks: Testing planning, memory, and self-repair capabilities, serving as a comprehensive examination of all abilities.

A practical recommendation: when building in-house evaluations, prioritize accumulating “real business tasks + automatically verifiable acceptance criteria” rather than chasing public leaderboard rankings.

VI. Inference-Side Optimization: Accelerating Without Capability Loss

Another noteworthy direction is a model paradigm of “same capability, faster speed”: without altering model capabilities, inference optimizations like speculative decoding (drafting by a small model, parallel verification by a large model), quantization, and KV Cache reuse can boost output speed by 2 to 3 times. The key is establishing an equivalence regression validation loop to prevent the model from becoming “faster but dumber.” For interactive Agents, output speed directly impacts task success rates—delay continuously accumulates in long tasks, increasing the probability of user interruption.

VII. Enterprise Implementation Recommendations

1. Foundation Model Selection: Prioritize foundation models whose post-training potential has been proven, and that are fully open or plan to open their weights, preserving room for private deployment and secondary post-training.
2. Data and Rewards Over Compute: First polish the acceptance criteria and verifier quality, then scale up training. This is the investment sequence with the highest ROI.
3. Leverage Cloud Infrastructure: Building an RL training and sandbox system from scratch is costly. Teams can utilize platforms like Baidu AI Cloud, leveraging its mature capabilities in GPU compute, model fine-tuning services, and containerized isolated environments to focus their energy on the true differentiators: domain data and reward design.
4. Iterative, Evaluation-Driven Approach: Iterate the “data -> training -> evaluation” loop on a weekly basis to avoid directional risks associated with large, one-time investments.

Conclusion

The upgrade path of “keeping the foundation model, deepening post-training” signals that LLM competition has entered a new phase: Pre-training determines the potential ceiling, while post-training determines the degree of realization. For the vast majority of teams, the truly controllable and replicable dividend lies in post-training—designing rewards well, building realistic sandboxes, and grounding evaluations in reality are the only ways to fully unlock the potential of trillion-parameter models.