The Real Source of Long-Horizon Agent Capability: Post-Training, Environments, and Reward Design

1. The Phenomenon: Sudden Leaderboard Gains Are Not the Same as Capability Leaps

Recently, several models have made clear progress on long-horizon agent benchmarks such as Terminal-Bench, SWE-Marathon, and Agents’ Last Exam. In particular, a number of open-weight models have approached or even matched some frontier closed models on certain leaderboards. This has led to a familiar narrative: one model has now “comprehensively surpassed” another.

Such judgments are usually oversimplified. Agent capability is not a static property of a single model. It is the result of a system composed of the model, the runtime environment, tool calls, context management, budget control, and automated verifiers. In other words, the same model can produce significantly different scores under different evaluation frameworks and tool configurations. The more useful questions are therefore not about who ranks first, but rather:

1. What are these evaluations actually measuring, and are the scores reliable?
2. Where does the gap still exist between open-weight models and frontier closed models?
3. What have recent post-training systems done correctly to produce improvements on long-horizon tasks?

2. The Evidence Boundary: What Is Officially Disclosed and What Is Reasonable Inference

Before discussing specific techniques, it is important to define the evidence boundary. Public information currently includes official blog posts, technical reports, papers, and leaderboard results. However, an official release narrative is not the same as full technical disclosure. To distinguish the factual basis, this article uses three types of statements:

• Officially disclosed: Training designs, experimental settings, or data explicitly stated in the original source.
• Experimentally supported: Findings backed by controlled experiments or ablations in papers, but valid only within the scope of the published experiments.
• Inferred judgment: Explanations based on publicly known mechanisms, which should not be treated as confirmed internal implementations by any vendor.

There are also two temporal boundaries. First, leaderboard data is time-sensitive and may change. Second, some models are positioned as open-weight at release, but the weights are only made public after safety evaluation and hardening. At a given point in time, such models should not be considered fully reproducible open models.

This layering of evidence matters because the agent evaluation and training space contains a large amount of engineering detail that is not fully disclosed. A score may reflect a longer runtime budget, a more permissive retry policy, or a better-matched tool set, rather than improved model reasoning alone.

3. What Agent Evaluations Actually Measure: A Score Is Not a Single Capability Value

3.1 Evaluation Tasks Usually Mix Multiple Capabilities

Long-horizon agent evaluations typically require a model to complete a task spanning many steps, such as modifying a code repository, operating a terminal, querying documentation, or fixing test failures. These tasks are not simple question answering. They involve:

• Goal understanding and task decomposition.
• Environment exploration and information retrieval.
• Command generation and tool invocation.
• Error handling and retry strategies.
• Long-term context memory and state tracking.
• Final result verification.

A leaderboard score can therefore improve because of gains in any of these areas. For example, a higher score on a benchmark may come from stronger reasoning, but it may also come from tool-call formatting that better matches the evaluation harness, or from more stable generated trajectories.

3.2 Harness, Tools, and Budget Have a Major Impact on Scores

In agent evaluation, the harness is the runtime framework that connects the model to the environment. The same model can score quite differently if the harness allows more turns, provides clearer error messages, or permits partial retries. Likewise, whether tool definitions are complete, whether environment feedback is immediate, and whether context is truncated properly all affect final performance.

This means we cannot use a single leaderboard to declare that one model is broadly ahead. A more precise statement would be: a given model completed a certain proportion of a specific class of tasks under a specific harness, tool set, and budget.

3.3 The Fundamental Difference Between Long-Horizon Tasks and Short-Form QA

Short-form question answering is closer to a static capability test, while long-horizon agent tasks are closer to a dynamic process test. The latter examines not only what the model knows, but also how it continues to act in an uncertain environment. Improvements on long-horizon tasks should therefore be traced to training data distribution, environment interaction frequency, and reward signal design, rather than simply attributed to a larger base model.

4. Where the Gap Usually Lies Between Open-Weight Models and Frontier Closed Models

4.1 Base Capability Gaps Are Narrowing, but the Engineering Chain Still Matters

Public evaluations show that some open-weight models are already close to frontier closed models in coding, math, and general reasoning. That does not mean overall capability is fully aligned. The gap often appears in:

• Stable attention and information retention over long contexts.
• Error recovery in complex multi-step tasks.
• Fine-grained understanding of tool return results.
• Robustness in open-ended environments.
• Stability of safety boundaries and refusal policies.

These capabilities are difficult to capture fully through a single evaluation score.

4.2 The Gap Is Increasingly About Systems and Ecosystems

The advantage of frontier closed models usually comes not only from model weights, but also from mature tooling ecosystems, APIs refined through heavy real-world traffic, well-developed context management strategies, and feedback loops accumulated over time. In comparison, open-weight models may perform well in the upstream engineering chain, but still require more investment in deployment, adaptation, monitoring, and rollback.

Therefore, deciding whether a model is suitable for enterprise agent scenarios requires looking beyond benchmark scores. It also requires assessing whether the model can be stably integrated into existing systems and whether it offers observability and controllability.

5. Why Post-Training Systems Can Improve Long-Horizon Task Performance

5.1 From More Data to Verifiable Task Environments

Post-training for large models has traditionally emphasized supervised fine-tuning data and preference data. But long-horizon agent tasks require models to learn not only to produce human-like output, but also to achieve goals inside an environment. Recent post-training approaches have therefore shifted toward building verifiable task environments.

In practice, this means reconstructing real workflows as runnable environments. For example, a code maintenance task becomes an executable repository, and a data analysis task becomes a set of checkable results. The model generates actions in the environment, the environment returns state, and an automated verifier ultimately determines success or failure.

The value of this approach is that rewards no longer depend on human annotation. They come from environment outcomes. As long as task definitions are clear and verifiers are reliable, high-quality training signals can be generated continuously.

5.2 Task Factories: The Mechanism for Producing Long-Trajectory Data

The scarcest resource in long-horizon agent training is long-trajectory data. A long trajectory refers to the complete process of a model completing a task across multiple rounds of interaction, including both successful action sequences and post-failure corrections.

Some teams address this data scarcity through a task factory mechanism:

1. Extract task templates from real work.
2. Build runnable environments and automated verifiers.
3. Let the model attempt tasks in the environment and generate trajectories.
4. Mark trajectories as successful or failed based on final results.
5. Merge high-quality trajectories back into the training set.
6. Use unified training to improve general and domain-specific capabilities together.

The core of this mechanism is not writing answers manually, but producing processes automatically. It gives the model a chance to learn from failed trajectories, thereby improving error recovery in long-horizon tasks.

5.3 Reward Design: Single-Trajectory and Multi-Trajectory Approaches

Reinforcement learning for agent training has not converged on a single answer. Different teams have chosen different paths, and both have produced results.

One approach samples only one trajectory per prompt and trains a critic to evaluate the state value at each step. The advantage is manageable computational cost and more stable training. The disadvantage is lower trajectory diversity, which may miss better solution paths.

Another approach keeps multiple trajectories for the same prompt and learns by comparing their quality. This can use trajectory differences to provide richer learning signals. However, training on long trajectories is slow, so partial rollback mechanisms have emerged. These allow intermediate steps to be judged and learned from without waiting for the full trajectory to finish.

Both approaches illustrate that the key to agent post-training is not any single algorithm, but the ability to continuously produce high-quality trajectories and extract reliable reward signals from them.

5.4 Unified Models: Merging Domain Expertise with General Capability

A task factory generates large amounts of domain-specific and environment-specific data. If a team trains only a specialized model, it may improve a single class of tasks while damaging general capability. The more common approach today is to merge domain-specialist data back into a unified model.

This requires trade-offs in data mixing, training objectives, and model architecture. For example, the model needs to retain general instruction-following ability while gaining stronger environment interaction capability. It needs to improve math and coding without becoming overly mechanical in open-ended dialogue.

The core principle behind this merging is straightforward: use general data to preserve generalization, and use task factory data to strengthen long-horizon execution.

6. Implications for Engineering Practice

6.1 Do Not Focus Only on Leaderboards; Look at the System Loop

When selecting or training an agent model, enterprises should build their own evaluation sets and runtime environments. Strong public benchmark performance does not guarantee the same stability on internal systems, internal tools, and internal data.

A more practical approach is to:

• Extract real business tasks and construct verifiable environments.
• Compare multiple models under the same harness, tools, and budget.
• Track process metrics such as success rate, average step count, error recovery rate, and timeout rate.
• Record failed trajectories and analyze whether the problem lies in planning, tooling, or verification.

6.2 Verifiable Rewards Are a Long-Term Moat

For enterprises that want to improve model agent capability, the most valuable investment is not simply accumulating more data, but building verifiable task environments. Only an environment that can reliably determine success or failure can continuously produce high-quality training signals.

For code tasks, this may mean running test cases. For data processing tasks, it may mean checking output formats and key metrics. For operational tasks, it may mean comparing final states. As long as the verifier is reliable, training data can keep growing.

6.3 Long-Trajectory Training Requires Supporting Engineering Capabilities

Long-trajectory training places high demands on infrastructure. Multi-turn interaction generates large amounts of intermediate state, which must be stored, truncated, and sampled during training. Parallel sampling introduces resource overhead, and varying trajectory lengths create training efficiency problems.

Teams therefore need:

• A trajectory data pipeline.
• An automated verification service.
• Scheduling that decouples training from sampling.
• Sequence segmentation and packing strategies for long contexts.
• Quality filtering and deduplication during training.

These capabilities are no less important than the model algorithms themselves.

7. Conclusion

The recent improvements in long-horizon agent tasks do not reflect a single breakthrough. They reflect progress in the overall post-training system. By building runnable environments, automated verifiers, long-trajectory data pipelines, and reliable reward signals, models can continuously learn from tasks that more closely resemble real work.

For developers and enterprises, the most worthwhile investment follows the same logic: build your own task and verification loop, track process metrics, and do not fixate on leaderboard numbers alone. Competition in agent capability is shifting from model weights toward systems engineering. Whoever can continuously generate high-quality training signals and reliably execute long-horizon tasks will gain a practical advantage in the next stage.