Long-Horizon Agents: A Practical Guide to RL Post-Training and Multi-Model Routing

    This article examines what long-horizon tasks require from models, how large-scale RL post-training affects long-horizon execution, and where multi-agent collaboration commonly breaks down. It also covers long-context and high-speed inference considerations, plus a practical routing and risk-control approach for allocating models by task complexity and role.

    Long-horizon tasks impose three requirements on models: they must keep making progress, retain key information in context, and keep costs controllable. Drawing on practice from multi-agent collaborative development, this article explains the relationship between large-scale RL post-training and long-horizon execution capability, analyzes common pain points, and presents a routing and risk-control method that can be directly adapted.

    1. Post-Training Determines Long-Horizon Task Performance

    Pretraining teaches a model a large amount of foundational knowledge, while post-training mainly affects instruction following, complex reasoning, and tool-calling capability. Large-scale RL post-training requires substantial compute, and the training system must also support multiple downstream domains at the same time. Therefore, every layer of the underlying technology stack needs optimization.

    First, communication and memory in the RL framework. Once models reach trillion-parameter scale, the communication overhead caused by tensor parallelism and pipeline parallelism increases noticeably. Some teams adjust parallelism strategies and loss functions to reduce capability degradation during training. The effectiveness of such methods depends on the specific model and data, and must be validated with your own evaluation set.

    Second, long-context computation. Attention computation grows rapidly with sequence length. Technologies such as FlashAttention and Ring Attention can process longer inputs under limited memory, but they address resource ceilings; they do not automatically improve a model’s ability to understand long text.

    Third, code execution sandboxes and scheduling. Agents need to run code in an environment and read the results. The sandbox must balance high throughput, low latency, and isolation, while the scheduling system scales resources on demand so that compute utilization for RL training remains stable. Both self-built clusters and public cloud container platforms can serve this role. When choosing, the main comparisons are isolation capability, cost, and the team’s operations capability.

    2. Three Pain Points in Multi-Agent Collaboration

    A common architecture is “Main Agent plus Worker Agent”: the main agent decomposes tasks and dispatches them, while multiple workers execute subtasks in parallel. This structure can increase parallelism, but it also exposes the following problems.

    1. Insufficient execution willingness. Some models finish long tasks too early, giving a brief conclusion instead of continuing to make progress. One common explanation is that reward design teaches the model to end tasks quickly with short answers. There are two types of mitigation. On the training side, process supervision (PRM) can provide feedback on intermediate steps. On the application side, you can detect whether output is too short or missing key steps, and retry or switch models when it does not meet the criteria.

    2. Context bloat and compression loss. Multi-agent communication quickly increases context. When approaching the window limit, frequent compression operations such as /compact can easily lose code details and historical decisions, causing subsequent steps to deviate from the goal.

    3. Token cost. High-end models are capable but consume quota quickly. “High-low pairing” is a common practice, and the key is to allocate budget and models by role. The table below lists the differences between the two types of roles.

    Dimension Main Agent Worker Agent
    Model High-end large model for planning and complex reasoning Lightweight model for executing clear subtasks
    Input Task goal, global constraints, progress summary A single subtask, necessary code snippets, and interface definitions
    Token budget Higher share, with a per-call cap Per-subtask caps, with concurrency control during batch execution
    Suitable tasks Decomposition, solution selection, result review Code generation, format conversion, single-file edits
    Failure handling Replan, or escalate to human confirmation Retry once; if it still fails, hand it to the Main Agent

    How should you choose the Main Agent and Worker models? Run candidate models on the same batch of tasks, and record success rate, average token count, and number of manual modifications. Once a Worker model reaches an acceptable success rate on a certain type of subtask, assign that type of subtask to it. The selection basis should be results on your own tasks, not parameter scale.

    How do you troubleshoot information loss during context compression? Run the same task once before compression and once after, then compare whether key file names, interface definitions, and decision records are preserved in the output. A safer approach is to write key decisions to an independent state file. Compression then applies only to the dialogue history, and the state file can be read when needed.

    3. Long Context and High-Speed Inference

    New-generation models generally support million-token contexts and can read a large codebase or a long financial report in one pass. But being able to read an entire codebase does not mean the model can accurately use every piece of information in it. A model’s ability to extract information from the middle of long text is usually weaker than at the beginning and end, a phenomenon often called “lost in the middle.” Retrieval-augmented generation (RAG) therefore remains valuable: retrieving highly relevant snippets first and then placing them in context is usually more reliable than stuffing all materials into the context at once.

    Long-horizon tasks require multiple rounds of reflection and tool calls, and the output time of each round accumulates. The value of high-speed models is that they shorten the wait time for each round. Some vendors claim output speed can improve by 2 to 3 times, but such numbers are comparable only under the same benchmark, hardware, input and output lengths, and batch size. Before launch, test on your own tasks and judge results by end-to-end latency rather than single-output speed alone.

    4. Engineering Practice: Routing and Risk Control

    The following example illustrates the idea of allocating models by task complexity. The code is directly runnable and uses the invoke interface.