Engineering Long-Context AI Agents: Post-Training RL Practices

1. From “Good Answers” to “Task Completion”: The Core Shift in Model Upgrades

Recent flagship LLM releases feature trillion-scale parameters, million-token context windows, and targeted enhancements in coding, 3D creation, financial research, and long-horizon agentic capabilities. Concurrently, “high-speed” variants have been introduced, boosting output generation speeds by 2x to 3x without compromising intelligence. Observing these iterations reveals a clear narrative: the allocation of post-training compute has fundamentally shifted.

Over the past two years, most teams focused compute on making models “sound more human.” Today, compute is increasingly directed toward making models “continuously work in real environments”—executing multi-step tasks, repeatedly calling tools, and maintaining stability over sessions lasting dozens of minutes or even hours. This represents a paradigm shift from “conversation quality” to “task completion rate,” explaining why coding and Agent capabilities have become the primary battlegrounds in model evaluations. Cloud providers are following suit; for instance, platforms like Baidu AI Cloud are integrating LLM services, agent orchestration, and managed runtime environments into unified capabilities that developers can combine on demand.

2. The New Post-Training Paradigm: Scaling RL Across Hundreds of Domains

2.1 From SFT to Cross-Domain Joint RL

Early post-training relied on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align outputs with human preferences. While effective for “dialogue,” this paradigm falls short for “long-horizon tasks.” The correctness of a completed task is difficult to capture via single-turn preference scoring; models must instead trial-and-error within an environment and receive verifiable feedback.

Consequently, new-generation models generally expand RL from a few alignment domains to hundreds of downstream areas: coding (using unit tests and compilation results as rewards), financial research (data verification and fact-checking), 3D creation (rendering results and geometric constraints), and tool calling (task terminal state determination). These “verifiable rewards” are prerequisites for scaling RL—reward signals must be objective and reproducible to prevent training drift.

2.2 Infrastructure Upgrades for Large-Scale RL

Cross-domain joint RL demands significantly more from infrastructure than SFT. A single sampling step often entails a complete closed loop: “generate — execute — evaluate — return.” Industry practices include:

• RL Training Framework Refactoring: Asynchronous decoupling of sampling and training to prevent environment execution from stalling GPUs.
• Long-Context Inference Optimization: Rollout sequences frequently exceed hundreds of thousands of tokens, requiring efficient attention implementations and KV cache management.
• Training Efficiency Improvement: Optimizing compute utilization across large-scale clusters to reduce pipeline idle time.
• Sandbox Infrastructure: Code execution, web browsing, and tool calling are performed in isolated, secure, and resettable environments.
• Scheduling Systems: Task-level scheduling to distribute massive environment instances across heterogeneous resources.
• Environment QA Platform: Continuously auditing task difficulty and reward correctness to prevent “reward hacking.”

A conceptual main training loop is shown below (pseudocode):

# Pseudocode: Cross-domain RL training main loop
for batch in domain_mix_sampler():          # Mix hundreds of domain environments by ratio
    traj = agent.rollout(batch.env, max_steps=64)  # Think-Act-Observe multi-step execution
    reward = batch.verifier.check(traj)     # Verifiable reward: test/validation/terminal state
    stats = rl_trainer.update(traj, reward) # Policy gradient update
    env_qa.audit(batch, stats)              # Environment QA: difficulty and reward correctness

Domain ratios are critical. When mixing hundreds of domains for training, an imbalanced ratio will cause the model to overfit domains where rewards are easily given. A mature approach involves difficulty-adaptive sampling for each domain and continuous monitoring of pass-rate curves.

3. Evaluating “Can It Work”: Terminal and Long-Horizon Benchmark Design

3.1 Terminal Evaluations: Realistic and Resource-Constrained

To measure the true capability of coding agents, the industry is shifting toward “real terminal environment” evaluations. Models are given a resource-constrained command-line environment and asked to perform composite tasks like installing dependencies, fixing bugs, and running tests, with specific validation for each task. Public results show that new-generation models now commonly score above 60% in such evaluations, narrowing the gap with top closed-source models to within 1 percentage point—indicating that “independent terminal operation” is moving from demo to practical usability.

3.2 Long-Horizon Code Evaluations: Multi-Step, Tools, and End-to-End

Another category is long-horizon code evaluation, covering various engineering fields and focusing on three aspects:

• Multi-step Execution: Tasks are broken down into dozens of sub-steps; can the model maintain its goal without drift?
• Tool Calling: Parameter accuracy for file I/O, command execution, and retrieval operations.
• End-to-End Resolution: The complete closed loop from requirement to runnable deliverable.

Industry data shows that new-generation models typically improve by 4 to 6 percentage points over their predecessors. While seemingly marginal, the significance for Agent applications is profound: failure rates in long-horizon tasks propagate step-by-step, meaning any improvement in per-step success rate is exponentially amplified across the entire chain.

3.3 Three Considerations for Evaluation Design

1. Leakage Prevention: Tasks and test cases must not appear in public corpora, or evaluations will be systematically inflated.
2. Variance Control: Agentic tasks have high randomness; run multiple times to calculate means and report confidence intervals.
3. Process and Results Equally Weighted: Evaluate not only the final pass rate but also steps taken, token consumption, and wall-clock time, as all three constitute the cost perspective.

4. High-Speed Variants: Trade-offs for 2-3x Decoding at Equal Intelligence

High-speed product variants typically rely on one or more of the following techniques:

• Speculative Decoding: A small model drafts outputs verified by the large model, accelerating decoding while maintaining the output distribution.
• Quantization and Sparsification: Reducing numerical precision on the inference side to compress memory and bandwidth overhead.
• KV Cache Optimization: Paged management and prefix reuse to reduce redundant computation under long contexts.
• Continuous Batching and Scheduling: Increasing throughput while keeping Time-to-First-Token (TTFT) latency low.

When is it worth paying roughly double the unit price? The rule of thumb is: it is always worthwhile when the LLM operates within a “multi-turn loop”. In interactive coding or long-horizon agent tasks, decoding time is multiplied by the number of loops. A 2x to 3x speedup drastically reduces task wall-clock time, and the cost of human waiting usually far exceeds token costs. Conversely, for offline batch processing or latency-insensitive scenarios, the standard version is more cost-effective.

5. Million-Token Context: Capabilities and Costs

A million-token context allows “entire code repositories” or “long-term research reports” to directly enter the model’s field of view, but it is not free:

• Attention computation scales with sequence length. Even with sparse engineering implementations, long-sequence inference costs remain significantly higher than short-sequence inference.
• KV cache memory grows linearly with context, serving as the primary bottleneck for long-context services.
• Positional encoding extrapolation and “lost in the middle” phenomena still require training optimizations and retrieval augmentation to mitigate.

Practical recommendations: Place stable system prompts and frequently used documents at the beginning of the context to hit the cache; place retrieved “mutable content” at the tail; structurally summarize ultra-long materials before expanding them as needed, rather than treating the million-token window as a bottomless pit.

6. The Economics of Three-Tier Token Pricing

New-generation model APIs generally adopt a three-tier pricing structure: Cache Hit Input, Cache Miss Input, and Output. At common industry volumes, Cache Hit Input costs about 1/10th of Cache Miss Input, while Output is the most expensive. Let’s estimate using illustrative numbers (for explaining pricing logic, not actual quotes):

Assume an agent session: System prompts and prefix documents total 200K tokens, multi-turn loops accumulate 1M new input tokens, and 100K output tokens. Illustrative unit prices: Hit Input $0.20 per million tokens, Miss Input $2.00 per million tokens, Output $8.00 per million tokens.

def cost(hit_m, miss_m, out_m, p_hit=0.2, p_miss=2.0, p_out=8.0):
    return hit_m*p_hit + miss_m*p_miss + out_m*p_out
print(cost(0.2, 1.0, 0.1))   # Stable prefix, cache hit: $2.40
print(cost(0.0, 1.2, 0.1))   # Prefix drift, all miss: $3.20

The per-call difference is modest, but in production systems with tens of thousands of daily calls, the cost disparity from prefix stability can amount to tens of thousands of dollars per month. This yields three engineering rules:

1. Stable Prefixes: System prompts, tool definitions, and common documents should be placed at the very front without dynamic concatenation.
2. Dynamic Content at the Tail: Content that changes per turn should be placed at the end of the context.
3. Reuse Session Context: For conversations that can be continued, do not replay the full history from scratch every time.

7. Harness as a Service: The Cloudification of Agent Infrastructure

Another signal from this upgrade cycle is the emergence of “managed agent” services. The runtime frameworks (harnesses) supporting long-running agents are now provided as cloud infrastructure, supporting 24/7 long-horizon tasks and multi-agent collaboration. This effectively shifts the engineering focus of agent applications from “orchestration code” to “operational capabilities”:

• Isolation and Security: Tasks execute in independent sandboxes; file systems and networks are auditable and resettable.
• Persistence and Checkpoint Resumption: Long-task states are persisted, enabling recovery from checkpoints after failures.
• Observability: Step-level trajectories, tool calls, and token consumption are logged end-to-end for easy replay and attribution.
• Multi-Agent Collaboration: Role division (e.g., planner, executor, reviewer) and message bus integration.

An enterprise evaluation checklist should include: sandbox isolation level, task timeout and retry semantics, cost quotas and circuit breaking, trajectory export formats, and integration with existing general cloud capabilities like CI/CD, object storage, logging services, and monitoring alerts. Providers like Baidu AI Cloud also offer LLM services and agent orchestration capabilities that can be combined with the aforementioned components, making them worth including in architecture design assessments.

8. Conclusion: Directing Compute Toward “Models That Can Work”

From dialogue to long-horizon tasks, the evolutionary path of LLMs is becoming increasingly clear: post-training compute is concentrating on cross-domain RL, evaluations are leaning toward real terminals and multi-step engineering, and product forms are extending into high-speed inference and managed agents. For developers, what needs upgrading is not just model selection, but an entire engineering methodology—context and cache management, cost accounting, evaluation design, and runtime operations. Whoever streamlines this methodology first will gain real delivery capabilities on new-generation models, rather than merely impressive demos.