Upgrading Large Language Models: Practices in Long-Horizon Agent Training

1. Core Direction of Model Upgrades: From “Answering” to “Executing”

In the recent round of large-scale model upgrades, the most notable shift is not merely the increase in parameter count, but the transition of core capabilities from short-term text generation to long-horizon task execution. Compared to previous generations, newer models demonstrate a pronounced “continuous working” characteristic in scenarios such as code generation, agent invocation, financial research, and 3D content generation.

This shift means models no longer simply provide a single output for a single input. Instead, they can maintain task goals over extended contexts, plan steps, invoke tools, and adjust strategies based on intermediate results. For developers, the criteria for evaluating model capabilities have also evolved: beyond the accuracy of single-turn Q&A, it is crucial to focus on the model’s stability and recoverability in real-world working environments.

Ultra-large parameter models typically utilize dense or Mixture of Experts (MoE) architectures, incurring high training and inference costs. Therefore, a widely adopted industry approach is to scale post-training compute and apply reinforcement learning (RL) across more vertical domains. The goal is to acquire more stable policies for a broad range of downstream tasks, rather than overfitting to a few benchmarks.

Furthermore, these models generally support a context window of up to a million tokens. This capability is critical for long-horizon agents. During prolonged tasks, the system must continuously retain environmental feedback, historical actions, intermediate plans, and constraints. Without sufficient context, the model is prone to losing key information midway through a task, leading to inconsistent actions or redundant efforts.

2. Post-Training and the Expansion of Reinforcement Learning

The traditional LLM training pipeline typically involves pre-training, supervised fine-tuning (SFT), and preference alignment. However, recent practices indicate that relying solely on supervised samples is insufficient to cover the strategic diversity required for complex agent tasks. An increasing number of teams are integrating large-scale reinforcement learning into the post-training phase, allowing models to interact with real or simulated environments to learn the closed loop of “take action — receive feedback — revise strategy.”

A primary challenge in scaling RL is constructing a sufficient number of downstream task environments with controllable quality. Common approaches include:

1. Code Execution Environments
   Providing the model with an isolated execution sandbox to run test cases, observe error messages, and iteratively modify code.

2. Command-Line Task Environments
   Requiring the model to complete tasks such as installing dependencies, modifying configurations, and troubleshooting faults within a resource-constrained terminal environment.

3. Research and Analysis Environments
   Supplying the model with structured data, documentation, and retrieval tools to perform information gathering, comparison, and report generation over an extended period.

4. 3D Content Creation Environments
   Enabling the model to generate and adjust 3D objects via script interfaces or parametric tools, and to determine compliance based on rendering results.

In these environments, RL reward signals are no longer just text matching scores, but quantifiable metrics such as task completion, test passage, and resource limits. This feedback closely mirrors real-world working scenarios and effectively enhances the model’s long-horizon stability.

Concurrently, training infrastructure must be upgraded to support large-scale RL. For instance, it requires capabilities for multi-environment concurrent execution, long-duration task scheduling, automatic retries for failed tasks, and environment health checks. Without these engineering配套设施, it is difficult to truly scale RL across hundreds of downstream domains.

3. Key Designs for Long-Horizon Agent Systems

There are distinct architectural differences between long-horizon agents and standard conversational systems. A long-horizon agent typically comprises the following modules:

• Model Core: Responsible for task understanding, plan generation, and action decision-making.
• Tool Invocation Layer: Encapsulates capabilities such as code execution, file operations, web retrieval, and data access.
• State Storage: Preserves task progress, intermediate nodes, and execution context.
• Scheduler: Determines when to continue, retry, or stop.
• Observation and Feedback: Monitors execution results, resource consumption, and anomaly states.

In practical development, the most frequent challenge is whether the system can resume from the point of failure if a task aborts midway, rather than starting over. Therefore, state persistence and execution snapshots are critical. A robust practice is to record the execution state before and after key steps, allowing the model to determine the recovery point based on historical records upon task interruption.

Another critical issue is multi-agent collaboration. In complex tasks, a single agent might not simultaneously handle planning, execution, and validation. A common practice is to split roles: one agent decomposes requirements, another implements specific steps, and a validation agent checks the results. This structure reduces the decision-making pressure on a single model during long tasks but imposes higher requirements on communication protocols and task orchestration.

Additionally, security boundaries cannot be ignored. When allowing a model to execute commands, modify files, or access the network, strict permission policies and audit logs must be enforced. Especially in cloud-hosted environments, each task should run in an isolated sandbox with secondary confirmation for sensitive operations.

4. Capability Evaluation in Code and Terminal Environments

When evaluating coding capabilities, looking solely at static problem scores is no longer sufficient. A more realistic evaluation method is to deploy the model into a terminal environment with a clear start and end point to complete multi-step tasks. For instance, a task might require the model to deploy a service on a simulated server, fix a configuration error, or pass a suite of automated tests.

This type of evaluation typically focuses on the following dimensions:

• Completion Rate: Whether the model can achieve the task objective within the time limit.
• Step Efficiency: Whether there is a large amount of redundant or invalid operations.
• Resource Consumption: Whether memory, disk, and network are used reasonably.
• Error Recovery: Whether the model can identify the cause of a failure and adjust its strategy.
• Tool Usage Rationality: Whether the most appropriate commands or interfaces are selected, rather than blindly attempting options.

According to public information, the performance of the new generation of ultra-large parameter models on such terminal tasks is approaching the advanced levels of current closed-source models. Although benchmarks differ in their criteria, the trend is clear: models are transitioning from “writing code snippets” to “independently completing project-level tasks.”

This evaluation methodology is also applicable to internal enterprise tools. Teams can build private task sets, solidifying typical faults, deployment processes, or refactoring tasks from real projects into repeatable test environments. The evaluation results obtained this way reflect the model’s adaptability to specific business needs far better than any public leaderboard.

5. Inference Efficiency and Cost Control

The inference cost of LLMs in long-horizon tasks is often much higher than that of single-turn Q&A. An agent task lasting tens of minutes or even hours can generate a context of hundreds of thousands or even millions of tokens. Therefore, inference speed and cost control are mandatory considerations for deployment.

Common optimization strategies in the industry currently include:

1. Cache Hits
   Caching identical or similar prefixes to reduce redundant computation.

2. Context Compression
   Summarizing or pruning historical messages, tool outputs, and intermediate states while retaining key information.

3. Tiered Responses
   Employing model instances with different response speeds for different tasks. For example, using a faster path for simple judgments and a slower, more precise path for complex reasoning.

4. Asynchronous Execution
   Splitting parallelizable sub-tasks for concurrent execution to shorten overall completion time.

5. Output Length Constraints
   Limiting model output length during planning and summarization phases to avoid generating verbose content that drives up costs.

Furthermore, for long-running tasks, it is not advisable to let the model hold the full context indefinitely. A more rational approach is to divide the task into multiple stages, generating a state summary at the end of each stage to serve as the prerequisite information for subsequent stages. This controls the context length and facilitates recovery if the task is interrupted.

6. Deployment Recommendations for Enterprise Developers

For teams looking to integrate these capabilities into internal systems, consider the following steps:

1. Build Evaluation Sets from Real Tasks

Do not rely solely on public benchmarks. It is recommended to collect real requirements from R&D, IT operations, and data analysis departments, and transform them into regression-testable task templates. Each task should include clear inputs, expected results, and pass conditions.

2. Establish a Controlled Execution Environment First

Before allowing the model to manipulate files or execute commands, establish an isolated sandbox. Set resource limits and timeouts for each execution, and maintain comprehensive operation logs. If feasible, snapshot file changes to enable quick rollbacks.

3. Design Clear Task Termination Conditions

The most common issue with long-horizon tasks is infinite looping or invalid attempts. Set a maximum step count, maximum duration, and failure threshold for each task. Once a limit is triggered, transition to manual takeover or an automated retry process.

4. Implement a Result Validation Mechanism

After the model completes an action, an independent validation module should check the results. For instance, whether the code passes tests, the document conforms to formatting, or the data is complete. The validation module can be rule-based or handled by another model, but avoid letting the executor simultaneously act as the judge.

5. Prioritize Observability

The execution process of a long-horizon agent must be trackable and traceable. It is recommended to log the inputs, outputs, tool invocations, and state changes for every step, routing this information into a unified logging platform. This not only facilitates troubleshooting but also provides data for subsequent model optimization.

7. Future Technical Directions Worth Tracking

Based on current trends, LLMs still have significant room for improvement in the direction of long-horizon agents. The following areas warrant continuous attention:

1. More Stable Long-Term Memory

Existing models primarily rely on the context window to retain information, but even large windows have limits. Establishing a readable and writable external memory structure, and enabling the model to retrieve and update memory at appropriate times, is critical for making long-horizon agents practical.

2. Finer-Grained Process Supervision

Compared to scoring only the final result, providing phased feedback during task execution helps the model learn more rational paths. How to construct process supervision data cost-effectively remains an open question.

3. Stronger Tool Discovery and Composition Capabilities

The number of tools in real-world environments can be vast. Models need the ability to quickly filter, compose, and generate invocation sequences from tool descriptions. This places higher demands on the diversity of training data and the standardization of tool descriptions.

4. More Reliable Self-Inspection and Repair

Whether a model can actively backtrack, undo actions, and attempt new solutions upon realizing it has entered an error branch is a key indicator of agent maturity. Such capabilities typically require explicit “failure-recovery” scenarios to be designed during training.

5. Edge-Cloud Collaboration

Not all phases of a complex task need to be executed in the cloud. For latency-sensitive or data-sensitive steps, local execution can be considered; for steps requiring high compute or large-scale retrieval, the cloud is more appropriate. Edge-cloud collaboration will provide a more flexible architectural choice for model deployment.

Overall, the new generation of ultra-large parameter models has made significant progress in coding capabilities, long-horizon task execution, and multi-environment interaction. For technical teams, rather than chasing the specific performance numbers of a particular product, it is more valuable to first establish internal task environments, evaluation systems, and execution safety mechanisms. After all, no matter how powerful a model is, it requires a reliable workflow to translate into genuine productivity.