Post-Training RL and Long-Horizon Agents: The New Core of LLM Evolution

1. From “Bigger” to “More Capable”: The Focus of Model Upgrades Has Changed

If you look at flagship model iterations over the past two years side by side, a clear pattern emerges: the competitive center of gravity is shifting from pretraining parameter scale toward post-training compute investment and the construction of agent execution environments. The latest flagship models generally share three characteristics. First, parameter scale has entered the trillion-level range, and context windows have expanded to the million-token level. Second, post-training now incorporates large-scale reinforcement learning (RL), extending RL from general-purpose “alignment” to hundreds of downstream domains. Third, execution infrastructure is being built around these models at the same time—sandboxes, scheduling, and managed runtimes—so that models are not just able to answer questions, but able to get work done.

In terms of capability distribution, code generation, long-horizon task execution, financial research, and 3D creation have become key areas of focus. These areas share common traits: long task chains, dependence on tool calls, and verifiable outcomes. In other words, they are naturally suited to reinforcement learning and naturally suited to delivery in agent form. Understanding this round of upgrades requires examining three things separately: how post-training RL works, how infrastructure supports it, and how long-horizon agents are deployed in practice.

2. Post-Training Reinforcement Learning: From “Alignment Tool” to “Capability Production Line”

2.1 A Shift in Role

In the early days, RL played a mainly alignment-focused role in large models: using human feedback to suppress harmful outputs and unify style. It was essentially a gatekeeper. In this latest wave, RL has become a capability production line. Whenever a domain has verifiable results—whether unit tests pass, whether terminal commands achieve their goals, whether financial data is computed correctly, or whether a 3D model renders successfully—you can construct an environment, define rewards, and let the model acquire skills through trial and error.

This class of methods is often described as reinforcement learning with verifiable rewards. It complements traditional approaches based on human preference reward models: the former is suitable for tasks with objective standards such as code, math, and tool calling, while the latter is suitable for more subjective tasks such as writing style and safety boundaries.

2.2 What “Scaling to Hundreds of Downstream Domains” Really Means

Extending RL to hundreds of downstream domains is much more than adding a few more training tasks. It involves the convergence of three engineering tracks:

• Environment construction: Each domain needs an interactive task environment, including initial states, tool interfaces, and evaluation scripts. Once the number of environments reaches scale, building and maintaining them becomes a data engineering effort in its own right.
• Reward design: You need to prevent the model from gaming the reward function while keeping the signal dense enough for long task chains to learn from intermediate feedback.
• Mixing ratios and curriculum: When training jointly across hundreds of domains, task difficulty gradients and sampling ratios across domains significantly affect final capabilities, requiring extensive ablation experiments.

2.3 Engineering Challenges in Joint Training

• Gradient conflict: The optimal policy directions of different domains may contradict each other. This needs to be mitigated through sampling ratios, parameter isolation, or regularization.
• Catastrophic forgetting: Strengthening one domain can damage general capabilities. This requires adding capability-retention terms to the reward and continuously running regression tests.
• Training signal-to-noise ratio: A single trajectory in a long-horizon task has extremely high variance, and advantage estimation noise is significant, placing higher demands on algorithm implementation and infrastructure stability.

A simplified training loop can be understood as follows:

# Simplified multi-domain RL training loop (pseudocode)
policy = init_policy(base_model)
for step in range(train_steps):
    domain = sample_domain(domain_pool, weights=curriculum)  # Sample domain by curriculum weighting
    traj  = rollout(policy, domain.env, sandbox=SandboxPool) # Interact in isolated sandbox
    reward = domain.verifier.score(traj)                     # Verifiable reward
    buffer.add(traj, reward)
    policy.update(buffer.sample(batch), clip=eps)            # Policy optimization
    regression_test(policy, general_bench)                   # General capability regression

3. The Underappreciated Other Half: Infrastructure for Scaling RL

Public discussion often focuses on algorithms, but what determines whether RL can scale is frequently infrastructure. A serious flagship-level RL overhaul typically involves six types of systems:

1. RL training frameworks: The core challenge is making generation and training workloads cooperate efficiently. The inference side produces trajectories in batches, while the training side consumes experience and updates parameters. Both sides need asynchronous pipelines, hot weight updates, and KV cache reuse; otherwise compute utilization suffers significantly.
2. Long-context inference optimization: Million-token contexts mean attention and KV cache overhead rise sharply. Sequence parallelism, paged caching, and selective attention are needed to make long-trajectory training economically feasible.
3. Training efficiency engineering: Large-scale parallel sharding strategies, memory optimization, elastic scaling, and failure recovery directly determine training throughput and stability.
4. Sandbox infrastructure: Agent tasks need to execute code in real terminal, filesystem, and network environments. A sandbox must simultaneously address isolation security, resource limits such as CPU, memory, and time budgets, and snapshot recovery for fast resets after failure.
5. Scheduling systems: Massive mixed training and evaluation workloads require unified task orchestration, priority preemption, and resource quota management.
6. Environment quality inspection platforms: This is the most easily overlooked piece. The environment itself is a form of data. If evaluation scripts misjudge results, tool interfaces are unstable, or task descriptions are ambiguous, RL will faithfully learn the wrong signal. Environments therefore need automated validation, sampling checks, and version management.

The short version: algorithms determine the upper bound of RL capability, while infrastructure determines whether that upper bound can actually be reached.

4. Long-Horizon Agents: From Single-Turn Q&A to Continuous Operation

4.1 Why Long-Horizon Tasks Are Hard

Single-turn Q&A and long-horizon tasks are fundamentally different types of difficulty. Building a data pipeline that runs across multiple days, performing multi-step engineering refactoring, or conducting ongoing financial research involves at least four categories of challenges:

• Context expansion: Historical steps and tool outputs accumulate continuously, quickly exceeding the window or diluting attention.
• Error accumulation: Small early deviations propagate through multiple steps and cause overall failure.
• Goal drift: External conditions change during execution, requiring dynamic replanning.
• Irreversible operations: Actions such as deleting files or committing code in a real environment cannot simply be rolled back.

4.2 Harness: The Skeleton Outside the Model

The industry consensus is becoming clear: long-horizon capability depends not only on the model but also on the execution framework wrapped around it, often called the harness. The harness handles context management, including history compression, summarization, and key state retention; tool protocols, including unified call formats, timeout retries, and result validation; checkpointing and resumable execution; error recovery, including automatic retry, alternative paths, or escalation to humans; and end-to-end observability.

Exposing this harness as cloud infrastructure—in effect, a managed agent service—is one of the most industrially significant moves in this round of upgrades. Enterprises do not need to build their own always-on runtime, sandbox clusters, and monitoring systems. They can run long-horizon tasks directly in the cloud and have multiple agents collaborate by role—for example, one handling retrieval, one handling coding, and one handling review.

# Long-horizon agent main loop with checkpointing and error recovery (pseudocode)
state = load_checkpoint(task_id) or init_state(goal)
while not state.done and within_budget(state):
    plan = planner.revise(state)                    # Dynamically replan based on latest observations
    obs  = harness.execute(plan.next_step, sandbox=sandbox)
    state.merge(obs)
    if state.step % CHECKPOINT_EVERY == 0:
        save_checkpoint(task_id, state)             # Resumable execution
    if obs.failed:
        state = recovery.handle(state, obs.error)   # Retry / alternative path / escalate
finalize(task_id, state.summary)

5. Inference Economics: High-Speed Variants and Prefix Caching

5.1 High-Speed Variants: Same Intelligence, Different Latency

Releasing a high-speed variant alongside a flagship model has become standard practice. Intelligence remains the same, output speed improves by roughly 2–3x, and the unit price is about twice that of the standard version. The general technical approaches behind such variants include speculative decoding, where a small model drafts and the large model verifies in parallel; quantization and low-precision inference; KV cache reuse; and more aggressive continuous batching. The selection logic is straightforward: latency-sensitive interactive scenarios such as code completion and real-time conversation should prioritize the high-speed variant, while throughput-oriented offline batch processing is more cost-effective with the standard version.

5.2 Prefix Caching: The Cost Lever of the Agent Era

Most model services use a three-part pricing scheme: cached input, uncached input, and output. Cached input usually costs an order of magnitude less than uncached input because the KV cache for identical prefixes can be reused, eliminating redundant prefill computation. For agent scenarios, this mechanism is highly valuable—every step of a long-horizon task carries a large volume of highly repetitive historical context, and the prefix hit rate directly determines the cost curve.

# Impact of prefix caching on multi-turn agent session cost (relative price, illustrative)
P_IN, P_HIT, P_OUT = 1.0, 0.1, 5.0           # Relative unit price: uncached input / cached input / output
prompt, hit, out = 100_000, 90_000, 2_000    # Tokens per turn (illustrative)
cost_no_cache = prompt * P_IN + out * P_OUT
cost_cache    = hit * P_HIT + (prompt - hit) * P_IN + out * P_OUT
print(cost_no_cache, cost_cache)             # At 90% hit rate, input cost can drop severalfold

Corresponding engineering practice: put stable system prompts, tool definitions, and domain knowledge at the front of the context and keep them byte-level stable—even a single whitespace change can invalidate the cache—while placing volatile content at the end and reusing session prefixes across turns whenever possible.

6. Evaluation: How to Judge Code and Agent Capability

• Real terminal environment benchmarks: In a resource-constrained real terminal, models act as coding agents to complete complex tasks. Each task has dedicated tests and objective scoring. These benchmarks do not just test whether the model can write a piece of code, but whether it can actually get things done on a real machine, including environment exploration, dependency installation, multi-command orchestration, and failure recovery.
• Long-horizon code benchmarks: These cover multiple engineering domains and focus on multi-step execution, tool calling, and end-to-end problem solving, making them closer to real development work than single-turn code generation.

Public results show that leading models have moved into the 60%+ range on such benchmarks, and the gap between the top tier has narrowed to around one percentage point. That itself is an important signal: code agent capability is moving from “demo-ready” to “engineering-ready.” When interpreting scores, note three points: pay attention to saturation—once scores approach 60%, leaderboard differentiation declines and failure modes matter more than total scores; guard against inflated scores from data leakage; and differences at the one-percentage-point level may fall within evaluation variance.

Before enterprise adoption, it is advisable to build a small internal evaluation set: tasks sourced from real work orders, objective and reproducible scoring scripts, environments consistent with production, multiple runs per task with averaged results, and retained failure trajectories for root-cause analysis.

7. Enterprise Adoption Paths and Selection Guidance

1. Capability-based routing: Send simple tasks such as format conversion and summarization to small models or high-speed variants, and send complex tasks such as architecture design and multi-step engineering to flagship models. A routing layer keeps costs within a reasonable range.
2. Hybrid context strategy: Long context is not a replacement for retrieval-augmented generation (RAG). Keep stable domain knowledge in the prefix and benefit from cache discounts, while injecting frequently changing knowledge dynamically through retrieval.
3. Prefer managed runtimes for long-horizon tasks: Long-horizon tasks have high requirements for availability, checkpoint recovery, and observability, and the cost of self-building is often underestimated. You can assemble a runtime from general cloud capabilities—model services, containers and function compute, object storage, message queues, logging, and monitoring and alerting. Platforms such as Baidu AI Cloud also offer general-purpose model services and agent hosting capabilities, which can serve as a middle ground between fully self-built and fully purchased solutions.
4. Address security and compliance up front: Sandbox isolation, least privilege, human confirmation for destructive operations, and full audit trails are baseline requirements before agents enter production.

8. Conclusion

The message from this round of flagship model upgrades is clear: the race over parameter count and context length alone is giving way to a systemic competition built on post-training RL plus execution infrastructure. For developers, three things deserve serious attention: reshape business tasks into verifiable forms; treat long-horizon runtime engineering details such as caching, checkpointing, and error recovery as first-class concerns; and replace leaderboard worship with internal evaluation that mirrors production. Model capabilities are iterating faster and faster, but these engineering methodologies will have a much longer lifespan.