Post-Training: Unlocking the Potential of Trillion-Parameter Foundation Models

1. Paradigm Shift: Focus Moves from Pre-training to Post-training

Recently, a notable trend has emerged in the LLM industry: flagship model iterations from various teams no longer necessarily involve a completely new, larger-scale pre-training run. Instead, they reuse existing foundation models and achieve capability leaps by significantly increasing post-training compute. Some teams even adopt a versioning scheme where the base model version remains unchanged while minor versions are incremented, signaling that the improvements primarily come from post-training rather than the base model itself.

This shift is not temporary but an inevitable result of the convergence of two technical curves:

1. Diminishing marginal returns of pre-training. The incremental availability of high-quality public corpora is becoming limited. Continuing to scale up data and parameters leads to exponential cost growth, while the capability gains become increasingly marginal.
2. The gap between base capabilities and usable capabilities. Trillion-parameter base models absorb vast amounts of knowledge and reasoning patterns during pre-training, but these capabilities exist in a “latent” state dispersed across the parameters. A model might occasionally solve a complex reasoning problem, but it cannot do so consistently or on command.
3. Efficiency differences in downstream capability injection. Teaching a base model to “reliably call tools, write code in an execution environment, and complete multi-step tasks” by piling on more pre-training data is highly inefficient. Optimizing directly on the reward signals of target tasks using reinforcement learning (RL) provides a much shorter path.

In short: Pre-training determines what a model “knows and can think of,” while post-training determines what a model “can reliably do.” When the base model scale reaches the trillion-parameter level, the efficiency of post-training in unlocking potential is often higher than building a larger base model.

2. How Post-Training “Unlocks” Potential: Mechanism Breakdown

2.1 Three-Layer Technical Stack

Modern post-training systems typically consist of three layers, each serving a distinct purpose:

• Supervised Fine-Tuning (SFT): Shapes model behavior using high-quality instruction-response data, addressing basic issues like instruction following, output formatting, and task paradigms. It answers the question, “What does a合格的 response look like?” (What does a passing response look like?)
• Preference Optimization (RLHF/DPO, etc.): Introduces human or model preference signals among multiple candidate outputs to optimize helpfulness, harmlessness, and expression quality. It answers, “Which one is better?”
• Reinforcement Learning with Verifiable Rewards (RLVR): In domains with objective verification methods like math, code, and tool calling, it uses compilation results, unit tests, and rule validation as rewards to directly optimize task success rates. It answers, “How to truly get it right?”

These three are progressive: SFT brings behavior up to the passing line, preference optimization makes it “satisfactory,” and RLVR makes it “objectively correct.” The primary source of current capability leaps is the large-scale application of this third layer.

2.2 Reinforcement Learning is “Filtering” rather than “Inculcating”

The industry often uses “emergence” to describe the appearance of LLM capabilities. From a post-training perspective, a more specific explanation can be offered: pre-training has already compressed the vast majority of the “knowledge raw materials” needed for problem-solving into the parameters. What RL does is filter out the correct reasoning paths from the massive solution space using reward signals and solidify them into stable policies.

This also explains why post-training can bring significant improvements without altering the base model at all—the raw materials are already there; what was missing was a stable extraction method. The value of RL lies not in teaching the model new knowledge, but in transforming “accidentally getting it right” into “consistently getting it right.”

2.3 From Single Domain to Hundreds of Downstream Domains

Early RL post-training focused on domains like math and code, where it is easy to construct verifiers. A recent clear trend is extending RL to hundreds of downstream domains: document analysis, professional Q&A, multi-step software engineering, graphics and modeling tasks, etc. The core challenge of domain scaling is only one: the availability and consistency of reward signals. Any domain where a reliable verifier can be constructed can replicate the path of “RL bringing a leap.” This also means that a significant portion of the engineering effort for post-training teams is actually spent designing and maintaining verifiers for various domains.

3. Key Technical Components

3.1 Reward System: Hard Verifiers First, Generative Rewards as Backup

Robust reward design typically follows these principles:

• If it can be objectively verified, never use model scoring: Code relies on compilation and unit tests, math relies on final numerical values, and structured tasks rely on rule validation.
• Open-domain tasks use generative reward models as backup, but the reward model itself must be calibrated to prevent score drift.
• Always defend against reward hacking: Models will look for loopholes in the reward function, such as bypassing tests to directly assert expected outputs or leaking answers during reasoning. Common countermeasures include randomizing test cases, leak detection, process review, and reward normalization.

def code_reward(traj) -> float:
    """Example reward function for a code task"""
    if not compiles(traj.code):
        return -1.0                     # Negative reward for compilation failure
    if leaks_answer(traj.trajectory):
        return 0.0                      # Answer leakage is considered cheating
    passed, total = run_unit_tests(traj.code)
    return passed / total               # Score based on test pass rate

3.2 Long-Context Reasoning: From “Fitting It In” to “Using It Well”

Flagship model context windows have grown from tens of thousands of tokens to the million level, but “fitting it in” and “using it well” are two different things. On the training side, issues like long-sequence data ratios, sequence parallelism, and positional encoding extension need to be resolved. More importantly, “deep assessment points” must be deliberately placed within the training data—putting key information at varying depths within a long document to force the model to learn precise attention allocation, rather than only looking at the beginning and end. On the inference side, techniques like KV cache management and attention sparsification must be coordinated to ensure acceptable throughput and latency under long contexts.

3.3 Sandboxed Agent and Tool Training

Agent capabilities are a focus of the current post-training round, and their training methods differ significantly from traditional RL:

• Multi-turn interactive reinforcement learning: Rewards no longer come from a single-turn response, but from the final result after the model takes multiple actions in an environment. Environmental feedback (compilation errors, test failures, tool errors) itself constitutes a rich learning signal.
• Real tool integration into sandboxes: Professional tools like code executors, 3D modeling software, CAD tools, and game engines are integrated into the execution environment via standardized interfaces. The model learns to combine tools through real feedback.
• Long-horizon task scheduling: Hour-long, multi-step software engineering tasks cannot be covered by a single reward. They need to be broken down into verifiable milestones and shaped using a combination of “process progress rewards + final result rewards.”

# Simplified training loop for RL with verifiable rewards (illustrative)
for batch in domain_task_stream:          # Sample tasks from hundreds of downstream domains
    prompts = sample_tasks(batch, n_variants=k)
    rollouts = policy.generate(prompts)   # Parallelly sample multiple trajectories in the sandbox
    rewards = [verifier.score(r) for r in rollouts]
    adv = group_relative_advantage(rewards)  # Intra-group relative advantage to reduce variance
    policy.update(prompts, rollouts, adv)    # Policy gradient update

3.4 Infrastructure: The Invisible Moat of Post-training

Post-training systems are fundamentally different from pre-training in engineering, with several key points:

• Decoupling generation and training: Trajectory sampling (inference workload) and policy updates (training workload) are placed in separate clusters, connected by an asynchronous pipeline. Either side can scale independently.
• Large-scale sandbox concurrency: Agent training requires simultaneously running thousands of isolated execution environments, involving rapid container creation, snapshotting, recycling, and fault isolation.
• Elasticity and fault tolerance: Long-cycle RL training must support checkpoint resumption and failed task rescheduling; otherwise, hardware failures will directly devour days of training progress.
• End-to-end health monitoring: Real-time curves of metrics like mean reward, entropy, KL divergence, and response length distribution are the primary indicators for determining whether training has collapsed or reward hacking has occurred.

It can be said that while pre-training’s moat lies in data and compute, post-training’s moat lies significantly in infrastructure: environments, verifiers, and scheduling systems.

4. Evaluation: How to Measure the True Gains of Post-training

Improvements brought by post-training must be measured with more stringent methods:

1. Beware of static benchmark failures. Overlap between training data and public evaluation sets (data contamination) will severely overestimate capabilities. Strict deduplication and regular updates of evaluation sets are required.
2. Prioritize real-task evaluation. Compared to multiple-choice static benchmarks, completing coding tasks in a real terminal environment, analyzing real complex documents, and executing multi-step engineering tasks better reflect a model’s deliverable capabilities.
3. Value both process and outcome. Looking only at the final pass rate can mask the “luck” component. It should be combined with process metrics like task step count, tool calling efficiency, and failure recovery ability.
4. Online performance closed-loop. Continuously convert failure cases from real-world online usage into new training and evaluation tasks, forming a data flywheel.

5. Engineering Practice Recommendations

For teams planning to build or introduce post-training capabilities, here are a few actionable recommendations:

• First, identify the bottleneck layer: If the model lacks knowledge, prioritize continued pre-training or Retrieval-Augmented Generation (RAG). If it lacks behavioral paradigms, do SFT. Only use RL if it has the behavior but is unstable and cannot improve accuracy. Applying RL where it shouldn’t be used is a common waste of resources.
• Take a progressive route: First use SFT to pull the task success rate to a visible level, then introduce RL for optimization. Running RL directly on a cold-start policy is highly inefficient for exploration.
• Tilt engineering investment towards rewards and data: The quality of the verifier determines the upper limit of RL. Having domain experts participate in designing verification rules yields far greater returns than simply piling on training compute.
• Leverage cloud platform toolchains: For example, taking advantage of LLM services provided by cloud platforms like Baidu AI Cloud. These services typically cover data management, model fine-tuning, evaluation, and deployment hosting. Combined with GPU resource scheduling, object storage, logging services, and monitoring alerts from the container platform, teams can focus their self-build efforts on reward design and data closed-loops—the areas that truly make a difference.

6. Future Trends

Looking ahead, three trends are largely certain:

1. The proportion of post-training compute will continue to rise. Training compute budgets will shift from “pre-training dominated” to “pre-training and post-training equally important,” or even post-training taking the majority share.
2. Agent RL will become the main battlefield. Environment construction, tool ecosystems, and long-horizon task training will become the new focus of competition for model teams.
3. The self-improvement closed-loop will gradually take shape. The massive amount of trajectories produced by models in real tasks will be filtered by verifiers and flow back as training data, forming a “usage—feedback—retraining” flywheel.

For developers, this signifies an important shift in perspective: The base model version number will no longer be the only focus; the depth of post-training on minor versions of the same base model is the true source of capability differences. Understanding the technical logic of post-training means understanding the source of competitiveness for the next generation of LLMs.