Long-Horizon Agent Post-Training: How Coding and Security Capabilities Emerge
Long-horizon agent post-training changes how coding and security capabilities are trained and evaluated. This article explains why smaller open-source models can perform well on repository-level coding, terminal interaction, and long-horizon security tasks, and discusses task formulation, reward design, credit assignment, counterfactual branch valuation, sandbox scaling, security boundaries, and a layered training and evaluation framework.
Recent improvements in open-source models on coding and cybersecurity tasks raise a question worth discussing in depth: why can models without a parameter advantage achieve results close to or even better than larger models on repository-level coding, terminal interaction, and long-horizon security tasks? If we look only at single-turn code generation or single-point vulnerability identification, it is easy to attribute the reason to “the model is stronger.” But from the perspective of training and evaluating frontier agent systems, the real changes often come from task formulation, reward design, and training infrastructure. In particular, once tasks shift from one-shot question answering to interactive closed loops of dozens or even hundreds of rounds, the “emergence” of coding and cybersecurity capabilities is no longer the result of a single model’s parameters. It becomes a systems engineering outcome shaped jointly by long-horizon decision-making, credit assignment, sandbox scaling, and security constraints.
1. Evaluating Coding Capability: From Single-Turn Generation to Long-Horizon Closed Loops
Traditional coding capability evaluations mostly focus on function-level completion, single-file repair, or code generation with given context. Such tasks provide short feedback and dense rewards, and a single model output can be judged as correct or incorrect. Frontier coding tasks, however, are closer to real software engineering: understanding repository structure, locating defects, modifying multiple files, running tests, and rolling back or continuing repairs based on failure logs. A task usually requires 30 to 80 interaction rounds. The model must not only write code but also use tools, read logs, manage context, and control risk.
Therefore, evaluating frontier coding capability should not rely only on final pass rate. The following dimensions also matter:
- Task success rate: closed-loop pass rate on hidden test sets and real repository tasks;
- Interaction efficiency: number of rounds, token consumption, and tool calls required to complete the same task;
- Long-horizon stability: whether the model can self-check, roll back, and recover after errors in intermediate steps;
- Generalization: whether the model only memorizes public tests or can transfer to new repositories, new dependencies, and new failure modes;
- Security boundaries: whether the model complies with permission, auditing, and rollback requirements.
Cybersecurity capability follows similar logic. A real security task is not single-point pattern identification but multi-step reconnaissance, evidence-chain reasoning, vulnerability localization, fix verification, and false-positive control. Evaluation should use authorized sandboxes and defensive scenarios, and focus on reproducibility, fix closure, false-positive rate, audit completeness, and risk control rather than merely pursuing attack success rate. Otherwise, so-called “emergent capabilities” can easily be amplified by task leakage or overfitting.
2. The Credit Assignment Challenge of Outcome Rewards
The core difficulty of long-horizon agent training is credit assignment. Suppose a task requires 80 steps. The model’s exploration, localization, and understanding in the first 78 steps are close to optimal, but at step 79 it writes an incorrect variable name, causing the test at step 80 to fail, and the environment returns a final reward of 0. Traditional PPO, GRPO, and similar methods will uniformly penalize all tokens from step 1 to step 80 because the entire trajectory receives a final reward of 0. This is similar to a chess game: as long as the game is lost, the entire trajectory is treated as a failure regardless of whether brilliant moves were made in between.
This outcome-only reward approach creates several problems:
- Starting difficulty: the model struggles to judge which early explorations are valuable, and behavior becomes conservative;
- High token consumption: to avoid risk, the model may repeatedly confirm and reread, reducing efficiency;
- Training degrades: valuable intermediate steps are penalized along with the failure, and the policy may lose effective exploration patterns;
- High variance: under sparse rewards, gradient signals are noisy and training is unstable.
The ideal solution is to score each step, but manual process annotation is extremely costly and requires software engineering and security experts. More problematic is a circular problem: without understanding software engineering, one cannot score accurately; without scoring, the model struggles to improve; without improvement, the model cannot automatically provide more accurate feedback. To break this loop, an automatic method is needed that does not rely on dense human annotation but can still attribute key steps.
3. Adaptive Optimization and Counterfactual Branch Valuation
The idea of adaptive optimization can draw on Monte Carlo tree search and causal inference: instead of requiring an absolute score for each step, estimate how much advantage a step brings relative to a counterfactual baseline. Specifically, keep the context of the first k steps unchanged, clone multiple branches for key steps, and let the same model continue exploring under equal intelligence and equal tool conditions. If subsequent task success from step k is significantly higher than the baseline, the step has a positive contribution; otherwise, it may have a negative contribution.
The pseudocode is as follows:
def counterfactual_step_advantage(trajectory, k, branches=8):prefix = snapshot(trajectory[:k])base_success = estimate_success(trajectory, prefix)success = 0for _ in range(branches):env = restore(prefix)roll = policy.rollout(env)success += roll.successbranch_success = success / branchesreturn branch_success - base_success
The focus of this method is not to assign an absolute label to every step but to compute local counterfactual advantage. During training, the final outcome reward can be mixed with local advantage, weighting key steps and downweighting irrelevant steps. Engineering requirements include prefix caching, environment snapshots, parallel sandboxes, branch budget control, and variance suppression. More branches are not always better; branches should be prioritized for high-uncertainty, high-impact key steps such as defect localization, interface modification, and security policy decisions.
4. Sandbox Scaling: From Parameter Scaling to Environment Scaling
Long-horizon agent capability improvement also depends on sandbox scaling. Unlike merely increasing parameters, sandbox scaling emphasizes expansion of environment count, task diversity, and interaction depth. Dozens of times more environment parallelism can significantly improve exploration coverage, provide enough samples for counterfactual branch valuation, and let the model learn from more real failure modes. This can be understood as environment-side scaling: not just making the model larger, but making the training environment richer, more reproducible, and closer to real engineering.
Engineering implementation should focus on:
- Environment images and dependency caching: ensure repository-level tasks can start quickly;
- Snapshots and restore: support branch exploration and failure rollback;
- Network and permission isolation: security tasks must run in authorized sandboxes;
- Resource quotas and scheduling: control parallel sandbox costs;
- Logging, monitoring, and auditing: record each interaction round and support reproducibility and attribution;
- Hidden tests and leakage prevention: prevent the model from memorizing evaluation sets.
In a generic cloud environment, container platforms, object storage, logging services, monitoring and alerting, and message queues can be used to build sandbox orchestration and observability. The key is not a specific product but training infrastructure that is snapshotable, isolatable, parallelizable, and auditable.
5. Parameter Efficiency, Post-Training, and Security Capability Evaluation
Architectural improvements plus post-training can indeed achieve strong long-horizon task capability at smaller parameter scales. This suggests that data quality, environment scale, reward design, and credit assignment methods may be more important than simply scaling parameters. However, evaluation should not rely only on a single leaderboard result; it should consider unit-parameter efficiency, success rate per token, long-horizon stability, and security boundaries. Cybersecurity capabilities are especially double-edged: a model that can find vulnerabilities can also be misused. Therefore, red-team testing, permission isolation, operation auditing, policy constraints, and rollback mechanisms must accompany such capabilities. When evaluating security capabilities, assess vulnerability localization, fix verification, false-positive control, and compliance together, rather than pursuing only attack success rate.
6. Engineering Implementation Recommendations
A layered training and evaluation system is recommended:
- Outcome reward: keep final pass rate as a hard constraint;
- Counterfactual step advantage: perform branch valuation for key steps to reduce variance under sparse rewards;
- Rule-based and security penalties: immediately penalize unauthorized, destructive, and non-compliant actions;
- Curriculum learning: gradually expand interaction rounds from short tasks to long tasks;
- Budget control: concentrate branch budget on high-impact steps;
- Offline evaluation: cross-validate using hidden tests, real repositories, and security test ranges.
In summary, frontier coding capabilities and emerging cybersecurity capabilities should not be simply understood as a byproduct of parameter expansion. A more reasonable explanation is that long-horizon agent post-training obtains rich environments through sandbox scaling, alleviates credit assignment under sparse rewards through adaptive optimization and counterfactual branch valuation, and ensures controllable capabilities through security constraints. When evaluating such capabilities, look at both outcomes and process efficiency, reproducibility, and security boundaries; focus on both the model itself and the training environment and engineering infrastructure.
