Scaling RL and Long-Horizon Agents: The New Technical Frontier of LLMs

1. Shifting Focus: From “More Knowledgeable” to “More Capable”

A new generation of flagship LLMs shares several common characteristics:

• Parameter scales are entering the trillion-level, but the focus is no longer on the parameters themselves, rather on real-world workload performance.
• Context windows universally reach the million-token magnitude, supporting entire code repositories, lengthy research reports, or massive logs within working memory.
• Capabilities focus on high-value tasks: code generation and debugging, 3D creation, financial research, and long-horizon Agent execution.

This reflects a clear technical consensus: models already possess sufficient “knowledge.” The bottleneck lies in the ability to “translate knowledge into reliable actions.” The production method for these capabilities relies precisely on large-scale RL during the post-training phase.

2. Scaling RL: From Alignment Tool to Capability Pipeline

Early RL primarily served alignment (e.g., RLHF). Today, it has expanded to hundreds of downstream domains, becoming a direct production pipeline for capabilities. Scaling this involves three core issues:

1. Where do reward signals come from?
Unlike subjective conversational preferences, tasks like coding, mathematics, and tool usage inherently possess “verifiable rewards”: passing unit tests, successful compilation, and reproducible calculation results. Extending these verifiable rewards to more domains (such as structural validation for financial reports or renderability checks for 3D models) is a prerequisite for RL scaling.

2. How to jointly train across hundreds of domains without mutual degradation?
Multi-task RL faces a typical “seesaw” problem: improving domain A might degrade domain B. Common engineering practices include balanced domain sampling, scheduling tasks by reward difficulty, and constraining update magnitudes per domain using reference policies.

3. Compute structure changes.
During RL training, the inference volume generated by rollouts far exceeds the gradient updates themselves. Inference and training are deeply coupled. A typical RL iteration can be abstracted as follows:

# RL Training Main Loop (Illustrative)
for batch in task_stream:
    # 1. Large-scale rollout: policy generates complete trajectories in environments
    trajectories = policy.rollout(batch.environments, n_per_task=k)
    # 2. Verifiable rewards: scoring via test cases/compilers/rule validation
    rewards = verifier.score(trajectories)
    # 3. Policy update: online policy RL algorithms (e.g., PPO and variants)
    stats = policy.update(trajectories, rewards)
    # 4. Quality monitoring: reward anomalies, entropy collapse, reward hacking detection
    monitor.log(stats, reward_hacking_signals)

3. Infrastructure Overhaul: Six Components of RL Infrastructure

Supporting large-scale joint training requires a systematic overhaul of the entire infrastructure chain. The industry generally focuses on the following components:

1. RL Frameworks: Decoupling rollout and training into an asynchronous pipeline. Rollout and training clusters scale independently to prevent the generation phase from blocking the update phase.
2. Long-Context Inference: Tuning combinations of paged KV cache management, tensor/pipeline/sequence parallelism to ensure rollout throughput with million-token contexts.
3. Training Efficiency: Utilizing mixed precision, activation recomputation, and optimizer state sharding to reduce per-step costs.
4. Sandbox Infrastructure: Providing strongly isolated environments for code execution, file operations, and network access. It must support snapshots and fast recovery. Agent trajectories can easily span hundreds of steps, and any environmental crash must be replayable.
5. Scheduling Systems: Co-locating training and rollout workloads to improve overall GPU utilization.
6. Environment Quality Inspection Platforms: Often the most overlooked yet rate-limiting component of RL. Systematic patrols are needed to ensure tasks are solvable, rewards cannot be “cheated” (reward hacking), and evaluation data has not leaked into pre-training corpora.

Empirically, the upper bound of RL gains is determined by environment and reward quality, not just the algorithm. Environment quality platforms transform “building environments” from manual craft into an industrial process.

4. Long-Horizon Agents: Evaluation and Runtime Design

Regarding evaluation, the industry is shifting from static Q&A benchmarks to dynamic task benchmarks. Representative examples include terminal-environment coding evaluations: engineering tasks are deployed in real, resource-constrained command-line environments with specialized tests per task to prevent models from “gaming” via memorization. Also notable are long-horizon code evaluations focusing on multi-step execution, tool calling, and end-to-end problem-solving rather than single-function completion. Publicly, top models on these benchmarks are approaching the strongest closed-source models, with gaps narrowing to around 1 percentage point. More importantly, the evaluation methodology itself—real environments + resource constraints + specialized verification—is becoming the universal paradigm for Agent evaluation.

Regarding runtime, long-horizon tasks rely on a “Harness” (Agent runtime framework). Its core responsibilities include:

• Context management: Compressing continuously accumulating observations and tool results into a finite window.
• State externalization and checkpointing: Persisting task state to support resume-from-checkpoint.
• Error recovery: Retrying failed commands and planning alternative paths.
• Multi-agent collaboration: Task splitting and result aggregation.

A minimal Agent main loop looks like this:

# Agent Runtime Main Loop (Illustrative)
while not task.finished():
    observation = environment.observe()
    context = memory.pack(system_prompt, observation, task_plan)
    action = model.decide(context)
    result = sandbox.execute(action)      # Isolated execution
    memory.update(action, result)
    if result.failed and retry_budget > 0:
        plan.revise(result.error)
    if checkpoint_due():
        state.persist()                    # Resume from checkpoint

A key practical insight: letting the model externalize its “progress” (e.g., maintaining a task list file) and writing state to the file system rather than the conversation history significantly improves the stability of long tasks.

5. Inference Services and Costs: Engineering Implications of Caching and Fast Tiers

New-generation model services generally exhibit two structural changes:

First, the “Fast Tier”: At the same intelligence level, output speed is increased several times through inference-side optimizations. Common methods include speculative decoding, higher server-side parallelism, and quantized deployment. While implementations vary, the product logic is consistent: trading higher unit prices for availability in latency-sensitive scenarios.

Second, a three-tier pricing structure: Cache-hit input, cache-miss input, and output are priced separately. This profoundly impacts Agent workloads, as long-horizon tasks typically consume far more input tokens than output. Estimating with illustrative unit prices (actual pricing depends on the platform):

Assumptions: Cache-hit input $0.01 / 1M tokens, Cache-miss input $0.10 / 1M tokens, Output $0.50 / 1M tokens
A turn interaction: 500k input tokens (450k hit prefix cache), 10k output tokens
Cost with cache hits = 0.45 * 0.01 + 0.05 * 0.10 + 0.01 * 0.50 = $0.05
Cost without cache   = 0.50 * 0.10               + 0.01 * 0.50 = $0.10
(Using equivalent relative proportions; difference scales similarly)

(Note: Translating the core logic—the cost difference can scale by multiple times depending on cache hit rates.)

The engineering conclusion is highly specific: stable content like system prompts, tool definitions, and code repository indices should be placed at the very beginning of the prompt prefix, and prefix jitter should be avoided. The cache hit rate directly dictates the marginal cost of an Agent.

6. Managed Agents: Infrastructure-izing Model Capabilities

A more notable trend is model providers offering the Harness itself—the runtime supporting long-running Agents—as a cloud service. Enterprises can run 24/7 long-horizon tasks and enable multi-agent collaboration without building their own orchestration layers. Similar to the evolution of Serverless, the hardest operations to maintain (sandboxes, scheduling, state management) are pushed down into platform capabilities.

For enterprise developers, the underlying technical foundation can be composed of general cloud capabilities: container platforms host the runtime, task scheduling services manage long-horizon tasks, message queues decouple multi-agent communication, object storage persists artifacts, and logging and monitoring services ensure observability. Cloud providers (such as Baidu AI Cloud) already offer these general categories of capabilities, which can serve as the foundational layer for Agent systems. Whether to adopt fully managed Agent services depends on a team’s trade-off between control and operational overhead.

7. Five Practical Recommendations for Developers

1. Prioritize evaluation over leaderboards: Sample tasks from your real workflows to build internal evaluation sets; use leaderboard scores only for initial screening.
2. Focus on long-horizon metrics: Multi-step completion rates, tool calling success rates, and checkpoint recovery capabilities are better predictors of production performance than single-turn Q&A accuracy.
3. Test cache hit rates before calculating unit prices: At identical price tags, differences in hit rates can lead to multiple times the actual cost variance.
4. Include open weights in selection criteria: Some providers now commit to releasing full model weights shortly after launch. Plan ahead for private deployment and compliance assessments.
5. Minimize sandbox permissions: Long-horizon Agents have file system and network access. Least privilege, network whitelisting, and artifact auditing should be default configurations.

Conclusion

From pre-training races to RL scaling, from single-turn Q&A to long-horizon Agents, and finally to the infrastructure-ization of the Harness, LLMs are evolving along the main thread of “reliably completing real work.” For developers, understanding this paradigm shift means the basis for selecting models has changed, the method for calculating costs has changed, and the architecture for building applications is changing. Aligning your evaluation systems, cost models, and runtime designs with the long-horizon task paradigm early on will secure an advantage in the next phase of model capability leaps.