Beyond Pretraining: Post-Training RL and Long-Horizon Agents as the New LLM Upgrade Paradigm

1. From “Stacking Pretraining” to “Stacking Post-Training”: The Shift in Competitive Focus

For the past several years, improvements in large model capabilities depended primarily on expanding pretraining scale: more data, more parameters, longer training time. But as high-quality public corpora approach their limits and marginal returns diminish, the industry’s competitive focus has clearly shifted to the post-training stage—that is, building on a base model through supervised fine-tuning, reinforcement learning, process rewards, and similar methods to polish “general knowledge” into “usable capability.”

Recent flagship models from multiple vendors show three highly consistent signals:

• Post-training compute has increased significantly, with investment even exceeding that of the previous generation several times over;
• Reinforcement learning is no longer confined to easily verifiable domains such as math and code, but has expanded to hundreds of downstream task domains;
• The capability selling point has shifted from “smarter Q&A” to “agents that can keep working”: coding, financial research, 3D tool invocation, long-duration task execution.

This confirms a judgment: the base intelligence gap between models is narrowing, and what truly differentiates experience is the degree to which post-training “cashes in” intelligence as productivity. Million-token context is also gradually becoming standard on flagship models, providing a memory carrier for long-horizon tasks.

2. Spreading RL Across Hundreds of Domains: Where Is the Difficulty?

Scaling reinforcement learning to hundreds of downstream domains essentially requires solving three problems: where environments come from, how rewards are computed, and how training stays stable.

1. Environment and sandbox infrastructure

The “training ground” for RL is an interactive environment. Code tasks require containers that can compile and run, tool invocation requires simulatable interfaces, and long-horizon tasks require workspaces that can save and restore state. Behind this is a full sandbox infrastructure: fast spin-up, resource constraints, network isolation, and traceable execution. An environment quality-check platform is responsible for filtering out tasks that are unsolvable, ambiguous, or whose rewards can be “gamed”—otherwise the model learns opportunistic strategies, and evaluation scores are inflated while actual capability is poor.

2. Reward design and verifiability

Verifiable rewards are currently the most effective route: whether unit tests pass, whether a terminal task reaches its target state, whether a financial conclusion is consistent with the data source. For domains that are difficult to verify automatically, a hybrid approach combining rule-based scoring and model judging is used, with consistency checks to suppress reward noise.

3. Training frameworks and scheduling systems

Parallel RL across hundreds of domains means large-scale rollouts, heterogeneous task scheduling, and efficient transfer of long-context KV state. At the framework level, the challenges are sample throughput, memory reuse, and checkpoint resumption; at the scheduling level, the challenge is balancing sample ratios across domains to prevent catastrophic forgetting and capabilities trading off against one another.

It is fair to say that post-training for this generation of models is no longer just a contest of algorithms, but of the overall engineering capability of an “RL factory.”

3. How Coding and Agent Capabilities Are Measured Scientifically

Coding is the domain where post-training returns are most direct, and it has also given rise to stricter evaluation paradigms.

1. Terminal-environment benchmarks

Unlike traditional “complete the code given a function signature” tasks, the new generation of benchmarks places models in real, resource-constrained terminal environments: given a task (such as fixing a complex defect or building a service from scratch), the model must autonomously execute commands, read and write files, install dependencies, and run tests, with each task paired with dedicated verification tests. These tests measure end-to-end delivery capability rather than single-point completion. From public results, the gap between leading models on such benchmarks has narrowed to single-digit percentage points, and “terminal-level” coding capability is becoming a flagship standard.

2. Long-horizon code evaluation

Another category of evaluation focuses on multi-step execution and tool invocation: task cycles span dozens or even hundreds of interaction rounds, assessing whether a model can maintain plan consistency, self-correct after errors, and ultimately solve the problem end to end. Compared with one-shot generation, long-horizon evaluation is closer to real engineering.

A typical long-horizon agent main loop can be abstracted as:

# Long-horizon agent main loop (illustrative)
while not task.finished and step < MAX_STEPS:
    observation = env.snapshot()          # Collect environment state: terminal, files, logs, etc.
    plan = model.reason(context, observation)   # Reasoning and planning
    action = model.propose_action(plan)         # Generate command or tool call
    result = sandbox.execute(action)            # Execute in restricted sandbox
    verifier.check(result)                      # Validate intermediate results
    memory.update(plan, action, result)         # Write experience for later steps to reference

What affects actual performance is often not the quality of single-step reasoning, but context management (state compression, retention of key information), failure retry strategies, and verifier design—collectively known as the agent’s Harness (execution framework).

4. Same Intelligence, Several Times the Speed: How High-Speed Variants Work

New model releases are often accompanied by “high-speed variants”: with intelligence level essentially unchanged, output speed is increased several-fold. Common technical approaches include:

• Speculative decoding: a small model drafts while a large model verifies in parallel, producing multiple tokens per forward pass;
• Prefix caching and KV reuse: stable prefixes such as system prompts and long documents are computed only once, then reused directly by subsequent requests;
• Quantization and operator fusion: compress memory and compute overhead under controlled accuracy loss;
• Continuous batching and scheduling optimization: improve GPU utilization and reduce queueing latency.

For developers, the value of high-speed variants is especially pronounced in interactive scenarios (coding assistants, real-time conversation): first-token latency and generation speed directly determine how “responsive” the experience feels; while in offline batch tasks, the standard version usually offers better cost-effectiveness.

5. Understanding Token Pricing: Cache Is the Cost Lever

Mainstream APIs today generally adopt three-dimensional pricing: cached input, uncached input, and output. The price difference among the three is typically an order of magnitude—output is the most expensive, uncached input next, and cached input the cheapest. The core of cost optimization is therefore increasing cache hit rate:

• Place stable content (system prompts, tool definitions, knowledge base prefixes) at the very front of the context;
• Reuse session prefixes across multi-turn conversations as much as possible, avoiding rewriting history each turn;
• For long-document Q&A, adopt a “document prefix cache + multi-question reuse” pattern.

A simple cost estimation illustration:

def estimate_cost(p_hit, p_miss, p_out, tok_hit, tok_miss, tok_out):
    """Price unit: CNY per million tokens (example values, for demonstrating the calculation only)"""
    return tok_hit/1e6*p_hit + tok_miss/1e6*p_miss + tok_out/1e6*p_out
# Assumption: cached input 1.5 CNY, uncached input 15 CNY, output 70 CNY
cost = estimate_cost(1.5, 15, 70,
                     tok_hit=4_000_000, tok_miss=1_000_000, tok_out=800_000)
print(f"Estimated cost: {cost:.2f} CNY")   # 77.00 CNY

For the same call volume, if the hit rate rises from 20% to 80%, input cost can drop by more than half. In agent-style long tasks (where context repeatedly carries history), caching strategy can affect total cost even more than the model’s unit price itself.

6. Harness in the Cloud: Managed Agents Become New Infrastructure

Another trend worth watching is the emergence of “managed agent” services: vendors package the execution framework that supports long-running agents—sandboxes, scheduling, state management, observability and alerting—as cloud infrastructure offered externally. Typical capabilities include:

• 7×24 long-horizon tasks: tasks run across days, with checkpoint recovery support;
• Multi-agent collaboration: multiple agents divide work, communicate, and aggregate results;
• Environment isolation and security: code execution and tool invocation are all completed within controlled sandboxes;
• Full-link observability: reasoning, actions, and results at each step are traceable and replayable.

This effectively lowers the engineering barrier for agent applications from “building a complete execution infrastructure yourself” to “calling a managed environment.” Mainstream cloud platforms (such as Baidu AI Cloud) also generally offer general-purpose components such as containers, function computing, and monitoring and alerting alongside model services, which can likewise be combined into a similar execution foundation.

7. Four Takeaways for Developers and Enterprises

1. Choose models by “capability density” rather than single-point scores: focus on evaluation dimensions close to production, such as long-horizon tasks, tool invocation, and terminal-level coding;
2. Choose high-speed for interactive scenarios and standard for offline scenarios: allocate versions by latency sensitivity to optimize both experience and cost;
3. Write cache design into the architecture: layered prompts and stabilized prefixes are the highest-return cost optimization measures;
4. Agent capability = model + Harness: the execution framework, verifiers, and state management also determine success; when evaluating managed services or self-built solutions, prioritize sandbox security, observability, and checkpoint recovery.

Conclusion

The main line of this round of model upgrades is very clear: the intelligence established by pretraining is being “cashed in” through large-scale post-training and reinforcement learning as productivity that can sustain work. For developers, understanding the post-training paradigm, evaluation methods, and cost structure has more long-term value than chasing single leaderboard rankings—because what determines real-world results has always been the combined optimum of model, engineering, and cost.