Long-Horizon Coding Agent Selection: Architecture, Evaluation, and Cost

    Choosing a coding agent for long-horizon work requires more than a single model ranking. This article explains how to compare sparse and compressed attention architectures against routine and long-horizon task profiles, evaluate them with fixed budgets and external acceptance checks, and calculate cost per successful task so that routing, escalation, and production rollout decisions reflect real delivery outcomes.

    1. Start with the Conclusion: Do Not Answer “Which Is Stronger?” with a Single Score

    A common mistake when deploying coding agents is to ask only “which model is stronger.” Engineering decisions are affected by at least five variables: cost per token, tokens required to complete a task, first-pass acceptance rate, repair cost after failure, and human takeover cost. Sparse attention or compressed attention architectures address marginal computation and cache pressure under long-context conditions. Planning stability, correct tool use, test closure, and state management determine whether a long-horizon task can ultimately be delivered. Conclusions are meaningful only when these factors are compared under the same budget constraint.

    The problem can be abstracted into two technical routes and two task spectra:

    • Technical route A: Use an indexing mechanism to select important tokens and apply sparse attention only to local windows, global tokens, and key segments, thereby reducing computation over long sequences.
    • Technical route B: Compress the key-value cache in blocks and interleave attention computation with different compression strengths and levels, reducing cache and computation overhead under very long contexts.
    • Task type A: Routine tasks with clear input and output boundaries, fixed toolchains, and deterministic acceptance rules, such as multi-source information summarization, structured extraction, fixed script generation, and batch repair.
    • Task type B: Long-horizon tasks with high-level goals and insufficient initial information, requiring autonomous research, planning, implementation, testing, repair, and delivery, such as delivering a runnable small interactive 3D prototype based on a requirements description.

    From an engineering perspective, focus on “cost per successful task” and “task success rate under a fixed budget,” not just a public leaderboard or single-turn chat performance.

    2. Task Spectrum Determines Evaluation Method

    2.1 Routine Tasks: After Capability Saturates, Stability and Cost Matter More

    Routine tool use usually has the following characteristics: fixed input and output formats, a small number of steps, clear tool permissions, detectable failures, and retryable tasks. For example, collecting data from multiple public information pages and outputting a structured summary, or generating a maintenance script according to an established template. Such tasks do not necessarily require the strongest planning capability, but they place strong emphasis on the following:

    • Whether tool-call parameters are correct and whether limited retries occur after failures.
    • Whether output conforms to the agreed structure and whether erroneous fields can be detected by validators.
    • Whether p95 latency is stable under high concurrency and whether cost per task is controllable.
    • Whether the system can degrade to a deterministic process or escalate to a stronger model when encountering rate limits, empty results, or format anomalies.

    In real projects, routine tasks often reach “capability saturation”: multiple technical options can complete the basic task, and differences appear more in stability, failure recovery, and cost. At this point, prioritize options with low cost per successful task, good observability, and clear fault isolation, and set human confirmation or stronger-model fallback for critical business processes.

    2.2 Long-Horizon Tasks: Planning and Test Closure Matter More Than Context Window Length

    The difficulty of long-horizon tasks is usually not “writing the first version of the code,” but whether the agent can continuously maintain the goal, use external materials, keep architecture consistency, repeatedly test, and fix defects. A typical evaluation task can be designed as follows: based on a requirements description, independently research public technical materials, plan and implement a runnable small 3D racing prototype, including basic movement control, collision, lap timing, boost zones, ramps, items or power-ups, UI state, restart, and packaging instructions.

    This task appears to be “writing code,” but it actually tests multiple capabilities:

    • Requirements decomposition: Can the agent break “a playable racing prototype” into scene, input, physics, rules, interface, and deliverables?
    • External research: Can it judge which materials are useful and capture conclusions in an evidence table?
    • Engineering implementation: Can it build a skeleton first and then implement features step by step, avoiding a single unmaintainable large file?
    • Testing and repair: Can it proactively construct smoke tests, run acceptance cases, locate failure causes, and perform limited repairs?
    • Delivery completeness: Can it provide run instructions, a self-test report, dependency declarations, and known limitations?

    Common failure modes include announcing completion too early, skipping acceptance tests, repeatedly modifying the same error, forgetting requirements constraints, unreproducible dependencies, lack of basic code modularization, and showing only a “it runs” screenshot without automated verification. Therefore, long-horizon task evaluation must write gates into the execution framework rather than trusting the model’s claim that it has “completed” the work.

    3. How Attention Architectures Affect Cost and Experience

    Dense attention computation grows rapidly with sequence length, and the key-value cache also consumes substantial GPU memory. Sparse attention and compressed attention reduce cost per token by “computing fewer tokens” or “compressing historical information.” Their engineering significance can be summarized in three points:

    1. Reduce marginal cost of long contexts. In scenarios such as code repository reading, long log analysis, and multi-turn tool calls, the input cost of long prompts is high, and sparsification can reduce computation and cache pressure.
    2. Improve concurrency and deployment elasticity. A smaller key-value cache helps increase concurrency and improve throughput for long-context tasks.
    3. May introduce information-selection error. Index selection or block compression may miss key details, especially cross-file symbols, implicit constraints, and long-distance dependencies. Therefore, recall capability must be tested with real codebases, not judged only by context-length metrics.

    It is especially important to emphasize: cheaper per token does not mean lower total task cost. An option may have lower cost per token, but due to weak planning, premature termination, insufficient testing, or repeated rework, it may consume more tokens to complete the same task and may even require human repair. The following general formula can be used for cost accounting:

    1. Cost per task =
    2. input tokens × input unit price
    3. + output tokens × output unit price
    4. + tool and sandbox compute cost
    5. + retry cost
    6. + human repair cost
    7. + latency and opportunity cost
    8. Cost per successful task = average cost per task / task success rate

    If an option has 30% lower cost per token, but long tasks consume 80% more tokens on average and acceptance success rate drops by 10 percentage points, then its cost per successful task may not be advantageous. Selection should use measured token consumption, repair rounds, and final success rate as inputs, not just compare unit prices.

    4. How to Design Long-Horizon Evaluation

    4.1 Fixed Task, Fixed Environment, Fixed Budget

    A credible long-horizon evaluation should at least fix the following variables: the same requirements description, the same tool permissions, the same network snapshot or cache, the same token and time budget, the same runtime sandbox, and the same acceptance scripts. The task should require the model to complete “research—planning—acceptance cases—coding—self-testing—repair—packaging—final acceptance” in sequence, with inspectable artifacts produced at each stage.

    It is recommended to set hard delivery gates:

    • The research stage outputs source summaries and a conclusion table, and is not allowed to jump directly to coding.
    • The planning stage lists module boundaries, risks, and acceptance items.
    • Automated acceptance cases are defined before coding.
    • The self-testing stage must run reproducible commands and save logs.
    • The repair stage must explain root causes and the scope of changes.
    • The delivery stage must include run instructions, dependency declarations, and known limitations.
    • The system accepts a “finish” request only after external acceptance scripts pass.

    4.2 Scoring Dimensions Should Cover Results and Process

    The following suggested weights can be adjusted according to business needs:

    Dimension Suggested Weight Focus
    Functionality and playability 35% Whether core interaction, rules, collision, and state transitions run
    Automated testing and reproducibility 20% Whether there are executable tests, logs, and stable reproduction steps
    Code structure and maintainability 15% Modularization, naming, error handling, and extension cost
    Long-horizon process quality 15% Planning, research evidence, repair root causes, and completeness of stage artifacts
    Security and dependency management 10% Least privilege, dependency sources, sensitive information, and resource limits
    Delivery completeness 5% Documentation, configuration, packaging, and known limitations

    Process metrics are equally important: token consumption at each stage, tool-call success rate, test pass rate, average repair rounds, premature termination count, human takeover minutes, and failure classification. Failure classification can include “premature wrap-up,” “tool misuse,” “context contamination,” “test avoidance,” “unavailable dependencies,” “state inconsistency,” and “security privilege escalation,” used to identify whether the problem comes from model capability, execution framework, or tool environment.

    4.3 Multiple Runs and Blind Review

    Long-horizon tasks have obvious randomness, and showing only the best single result is meaningless. Run repeatedly under the same budget and report mean, median, and confidence intervals. Code quality can use anonymous blind review, with submissions scored jointly by external acceptance scripts and human reviewers. Recording all failure cases is more valuable than keeping only successful cases.

    It is recommended to report two metrics together:

    1. Fixed-budget success rate: proportion of successful tasks within a given token and time budget
    2. Cost per successful task: total cost of successful tasks / number of successful tasks

    If two options have similar success rates, compare cost and latency. If one option has a higher success rate but a higher unit price, check whether cost per successful task remains acceptable and whether the business cost of failure is high enough.

    5. Routine Task Evaluation Should Focus on Stability and Cost-Effectiveness

    Routine tasks can be accepted in a more deterministic way. For example, require the agent to aggregate a set of structured results from multiple public information sources, and it must pass field validation, deduplication rules, timeout handling, empty-result handling, and output schema checks. The scoring focus is not “whether it writes a lot,” but:

    • First-pass structured validation rate.
    • Tool parameter accuracy and failure recovery rate.
    • p95 latency and cost per task.
    • Consistency across repeated executions.
    • Degradation capability for rate limits, timeouts, and abnormal responses.

    When basic capability already meets the business lower bound, prioritize lower-cost and more stable options, and ensure reliability through rule validators, caching, idempotent retries, and escalation mechanisms. For tasks that truly require more planning, long-term research, or cross-module modification, escalate to stronger models or more complex execution frameworks.

    6. Execution Framework Recommendations for Deployment

    6.1 Use a Stage State Machine to Prevent “Premature Completion”

    The execution framework should treat “completion” as a pending approval request, not a fact. Only when the current stage passes acceptance is the system allowed to enter the next stage. The following pseudocode expresses this:

    1. PHASES = [
    2. "requirements clarification", "research", "planning", "acceptance cases",
    3. "skeleton implementation", "feature development", "automated testing", "defect repair",
    4. "packaging and delivery", "external final acceptance"
    5. ]
    6. for phase in PHASES:
    7. while not gate_passed(phase, state):
    8. action = model.propose_next_action(
    9. context=state.context,
    10. tools=allowed_tools(phase),
    11. budget=phase.budget
    12. )
    13. if action.kind == "finish":
    14. state.feedback = gate_failure_reason(phase)
    15. continue
    16. state = execute(action)
    17. trace.record(action, state, budget)
    18. if state.budget.exhausted:
    19. return fail("budget exhausted", phase, trace)
    20. state = checkpoint(phase, state)

    This structure can significantly reduce the problem of “declaring delivery before the code has been tested.” Combined with stage budgets, maximum repair rounds, and critical-path checks, the controllability of long-horizon tasks improves noticeably.

    6.2 Manage Context Instead of Piling It Up

    Even if a model supports million-token context, do not stuff all historical logs into the prompt. A more effective approach is to maintain structured state:

    • The requirements description is the single source of truth, and changes must be explicitly recorded.
    • Research materials are compressed into evidence tables and conclusion lists.
    • Code context is obtained through symbol indexes and retrieval slices, avoiding loading the entire repository.
    • Test reports are archived by stage, retaining only current blockers.
    • Unresolved issues, risks, and TODOs are maintained separately.
    • Outdated tool outputs are summarized or discarded promptly.

    Context capability should serve “accurate recall and consistent editing,” not simply pursuing longer input. For tasks with dense long-distance dependencies, specifically test whether index selection or compression strategies miss key constraints.

    6.3 Sandbox, Permissions, and Observability

    The execution environment should use a least-privilege container sandbox, restricting network access, dependency sources, file write scope, and resource usage. Artifacts can be stored in object storage, runtime logs can be sent to a logging service, and key metrics can be connected to monitoring and alerting. Record a complete trace for each task: stage duration, token consumption, tool-call parameters and results, test output, failure reasons, and human takeover points. Without traces, it is impossible to distinguish insufficient model capability, prompt defects, and toolchain failures.

    6.4 Hybrid Routing and Escalation Strategy

    For teams with complex task portfolios, tiered routing can be used:

    • Routine tasks that are high-frequency, low-risk, and rule-clear use lower-cost options.
    • Tasks involving multi-file modification, external research, automated testing, and repair use options with stronger planning capability.
    • Automatically escalate when validators fail, tests fail consecutively, or humans mark risk.
    • Validate all routing rules through internal regression sets, avoiding coarse judgments based only on prompt length.

    7. Selection Decision Checklist

    Judge in the following order:

    1. Clarify the task portfolio. Count the proportion of routine tasks and long-horizon tasks, failure costs, human takeover costs, and concurrency requirements.
    2. Build a capability ladder. Cover at least five task types: single-function generation, multi-file modification, test-driven repair, long-horizon project delivery, and routine tool use.
    3. Run repeated evaluations under a fixed budget. Unify tool permissions, network snapshots, time and token budgets, run multiple times, and report confidence intervals.
    4. Use external acceptance as the standard. Use automated tests, runtime logs, and artifact checks; do not use model self-evaluation as the acceptance basis.
    5. Calculate cost per successful task. Include tokens, tools, retries, human effort, and latency in the calculation.
    6. Select “primary option” and “escalation option” separately. Routine tasks prioritize cost-effectiveness and stability; long-horizon tasks prioritize first-pass acceptance rate and repair efficiency.
    7. Continue regression after launch. Any change to the model, prompt, execution framework, or toolchain may change evaluation results, so maintain a repeatable internal benchmark.

    8. Conclusion

    “Which is stronger” is not a single-dimensional answer. For routine tasks, as long as basic tool-use capability meets the lower bound, cost, latency, and stability often become the main concerns. For long-horizon coding tasks, planning capability, test closure, state management, and repair efficiency determine final delivery, and low cost per token does not necessarily lead to lower total cost. Sparse attention and compressed attention can improve the economics of long contexts, but they must be judged together with actual task success rate, token consumption, and human repair cost.

    A more reliable approach is to build a budget-aware internal evaluation system: use real tasks, fixed environments, and external acceptance scripts to measure “cost per successful task,” and then perform hybrid routing and continuous regression on that basis. Selection conclusions obtained this way are closer to production reality than any single leaderboard or one-off demo.