From Foundation Models to Agents: A Full-Stack Look at RL Post-Training for LLMs

1. The Shift in Competition: Post-Training as the Main Engine of Capability Gains

Over the past two years, the industry’s understanding of where large model capabilities come from has changed structurally. On the pre-training side, high-quality corpora are increasingly hard to obtain, and simply scaling data and parameters yields diminishing marginal returns. On the post-training side, reinforcement learning (RL) and related methods continue to deliver steady gains in domains where results are verifiable, such as code, math, and agents. A basic consensus has emerged: the foundation model sets the ceiling of potential, while post-training determines how much of that potential is actually realized.

Recent flagship model iterations show three highly consistent characteristics: first, compute investment in the post-training stage has multiplied; second, context windows have broadly expanded to the million-token level; third, the focus of capability has shifted from “single-turn answer quality” to “end-to-end completion rate on long-horizon agent tasks.” These three directions are not isolated selling points but an interlocking technical whole—long context provides “working memory” for long-horizon tasks, while RL post-training converts environment feedback into stable, reliable behavioral policies.

2. Large-Scale RL Post-Training: An Underestimated Systems Engineering Problem

“Just throw more compute at it and it will improve” is the most common misreading of RL post-training. RL training is essentially a continuously running loop of “sample—evaluate—update.” To support large-scale joint training across hundreds of downstream domains, any weak link will leave newly added compute idle. It can be examined layer by layer across the stack:

1. RL framework layer: decoupling sampling from training. In generative RL, sampling is autoregressive generation with dynamic, fragmented load; training is batch gradient computation with relatively stable load. Running the two serially leaves large amounts of compute waiting. The mainstream approach is an asynchronous pipeline: the sampling engine and training engine are deployed independently, weights are synchronized periodically, and partial trajectory retention (continuing to use unfinished trajectories when weights update) reduces waste.
2. Long-context inference layer: RL sampling is itself inference. Long-horizon task trajectories often span hundreds of thousands of tokens. KV cache memory management (such as paged management), prefix cache reuse, and sequence parallelism directly determine sampling throughput, and therefore RL iteration speed.
3. Training efficiency layer: tuning combinations of data parallelism, tensor parallelism, pipeline parallelism, and other strategies; overlapping communication with computation; and recomputation strategies for ultra-long sequences determine effective utilization per unit of compute.
4. Sandbox infrastructure layer: RL for agent tasks requires models to execute code, terminal commands, and file operations in real environments. Sandboxes must simultaneously provide strong isolation, second-level creation and destruction, snapshot recovery, and high-concurrency scheduling. Once environment execution slows down, the entire training loop is dragged down.
5. Scheduling system layer: training, inference, and environment interaction are three task types with completely different load characteristics. They require unified resource scheduling and elastic scaling to efficiently “stitch together” fragmented compute.
6. Environment quality platform layer: this is the most easily overlooked layer, yet the one that most affects returns. RL effectiveness depends heavily on task environment quality: whether tasks are solvable, whether rewards are verifiable, and whether there are exploitable loopholes (reward hacking). A shared platform is needed for quality checks, version management, and deduplication of environments.

The following pseudocode summarizes this loop:

# Large-scale RL post-training main loop (pseudocode)
initialize policy model π, task environment pool E (all quality-verified)
while not converged:
    envs    = scheduler.allocate(E)            # scheduler allocates environments and compute
    trajs   = rollout_engine.sample(π, envs)   # async sampling: interact with sandbox environments
    rewards = verifier.score(trajs)            # verifiable rewards: scored by final outcome
    trainer.update(π, trajs, rewards)          # policy optimization + weight sync
    env_qc.audit(E)                            # environment quality inspection and elimination

Key conclusion: the conversion rate from RL compute to capability returns depends on environment quality and engineering efficiency, not on total compute itself. This also explains why teams investing similar amounts of compute can end up with vastly different capability results.

3. Long-Horizon Agent Capabilities: From “Answering Right” to “Getting It Done”

What is a long-horizon task? Unlike single-turn Q&A, a long-horizon task spans dozens of steps, multiple tools, and a longer time horizon. It requires the model to plan and decompose, maintain state, reflect on itself, and recover from errors. The industry typically characterizes it through three capabilities:

• Multi-step execution: decomposing goals into executable steps and maintaining consistent context and state across a long chain;
• Tool use: efficiently orchestrating external capabilities such as search, code execution, file read/write, and terminal commands;
• End-to-end problem solving: using the final outcome (tests passing, task closed) as the acceptance criterion rather than intermediate output that “looks good.”

Long context is a prerequisite for long-horizon tasks. From code engineering and financial research to 3D content creation, these scenarios share large input scale, long execution chains, and objective acceptance criteria. Models must not only “hold” millions of tokens but also maintain deep retrieval and multi-hop reasoning over ultra-long inputs. Notably, “being able to hold it” and “being able to reason over it” are two different things—deep utilization of long context also requires dedicated training.

Coding agents are the most typical touchstone. A complete engineering loop is: read the codebase → locate the problem → modify → run tests → fix based on feedback → verify again. Any error in any step requires the model to discover and correct it autonomously. Compared with static code completion, tasks completed in real terminal environments under resource constraints are closer to real production requirements, and have therefore become a mainstream way to assess coding agents.

4. How to Benchmark Coding Agents Scientifically

The evaluation system determines the quality of our understanding of model capabilities, and deserves its own discussion.

A credible benchmark should have four elements:

1. Real environment: evaluation in an executable, interactive terminal or engineering environment, not one-shot static generation;
2. Strict acceptance: each task comes with dedicated tests and an independent verifier to prevent “output looks correct” from slipping through;
3. Data leakage prevention: tasks and test cases are managed in isolation to avoid inflated scores from training data contamination;
4. Reproducibility: environments can be rebuilt and results rerun, so scores are comparable.

The design focus of long-horizon benchmarks is to cover multiple engineering domains and emphasize “multi-step execution, tool use, and end-to-end solving,” using task completion rate rather than single-point accuracy as the core metric. The evaluation main loop looks roughly like this:

# Coding agent evaluation main loop (pseudocode)
for task in benchmark.tasks:                  # tasks cover multiple engineering domains
    sandbox = Sandbox(task.setup)             # rebuild real environment in isolated sandbox
    history = []
    for step in range(task.max_steps):        # multi-step execution, not single-turn generation
        obs    = sandbox.observe()
        action = agent.decide(obs, history)   # planning / tool calls / terminal commands
        result = sandbox.execute(action)
        history.append((obs, action, result))
        if sandbox.finished(result): break
    score = verifier.check(task.tests, sandbox)  # independent verifier for strict acceptance
report.aggregate(scores, by="task_category")     # completion rate by domain

Three suggestions for interpreting scores: First, comparing the improvement magnitude and task distribution of different versions of the same model over time is more informative than chasing a single leaderboard across models. Second, beware of inflated scores caused by reward hacking and insufficient test coverage. Third, for developers, the most reliable approach is to build a private evaluation set from real business tasks, run small-traffic canary comparisons, and then make decisions.

5. Inference Speed: Experience and Cost at Equal Intelligence

Agent scenarios have an easily overlooked amplification effect: in multi-step tasks, per-inference latency is multiplied by the number of steps. For a task requiring dozens of interaction steps, even a few seconds of extra latency per step significantly degrades the end-to-end experience. This is why flagship models commonly offer a “high-speed version”—several times faster output while maintaining the same level of intelligence.

Common acceleration paths include:

• Speculative decoding: a small model drafts while a large model verifies in parallel, accelerating generation while keeping the output distribution unchanged;
• Quantization: low-precision quantization of weights and KV cache to reduce memory footprint and memory access overhead;
• Operator and kernel optimization: more efficient attention implementations and fused operators;
• Serving-layer scheduling: continuous batching and prefix cache reuse to improve overall throughput.

One caveat: when evaluating a “high-speed version,” confirm that its capability on the target task has not regressed—speed, quality, and cost form a triangle, and responsible engineering optimization should achieve “faster without getting dumber.”

6. Million-Token Context: Value, Challenges, and Common Paths

On the value side, ultra-long context directly opens up three scenario types: repository-level code understanding and refactoring, deep analysis of long documents and long research reports, and task memory across long conversations.

On the challenge side, the quadratic complexity of attention, the memory pressure of KV cache, and “deep reasoning” over long text (evolving from needle-in-a-haystack to multi-needle retrieval and cross-paragraph relational reasoning) are all hard problems.

Common technical paths include sparse and linearized attention variants, sequence parallelism based on ring-communication ideas, KV cache compression and tiered caching, and combinations of retrieval augmentation with context engineering (truncation, summarization, structured organization). In practice, the combination of “long context + precise context management” is often more effective than simply piling on length.

7. A Developer’s Perspective: An Action Checklist for Evaluation and Deployment

1. Build a private evaluation set: extract tasks from real business scenarios and define machine-verifiable acceptance criteria (tests passing, results comparable);
2. Use agent completion rate as the core metric: also track average step count, token cost, and failure mode distribution, rather than only single-turn output quality;
3. Do context engineering well: truncate, summarize, and structure ultra-long inputs, control noise, and improve effective information density;
4. Design robust tool interfaces: clear, structured error reporting can significantly improve an agent’s self-correction rate;
5. Establish checkpoints and rollback mechanisms: in long-horizon tasks, allowing “return to the most recent correct state” is often more cost-effective than blind retries;
6. Make good use of cloud toolchains: mainstream cloud platforms generally provide model services, GPU compute, containers and function computing, monitoring and alerting, and other general capabilities. Taking Baidu AI Cloud as an example, developers can use these general categories of capabilities to quickly build a complete pipeline of “evaluation—comparison—canary—launch”; for teams with data compliance or private deployment needs, open-weight models offer another path through local deployment and domain fine-tuning.

8. Conclusion

Looking beyond “which new model just scored higher on some benchmark” reveals more valuable signals: RL post-training is being industrialized—environment quality platforms, sandbox infrastructure, and heterogeneous scheduling systems are becoming standard equipment for leading teams; evaluation is becoming systematized—real environments, strict acceptance, and long-horizon tasks are becoming industry consensus; agents are moving from demos to production—completion rate, cost, and stability are replacing “impressive demos” as core metrics.

For developers, rather than chasing single-point scores, it is better to understand the technical logic behind them: capability leaps are never the product of a single breakthrough, but the result of systematic evolution across algorithms, engineering, and evaluation systems. Building your own evaluation standards and mastering engineering methods for context and toolchains is the most certain capability when facing rapid model iteration.