How to Systematically Evaluate the Long-Horizon Engineering Capability of LLMs

    As LLM release cycles accelerate, rankings and isolated demos provide limited guidance for engineering teams. This article presents a systematic, task-oriented approach to evaluating long-horizon engineering capability: layered task design, four core metrics (first-pass success rate, average tool calls, verifiable result ratio, and total cost per successful task), reproducible and auditable execution, and governance of integration, replacement, and rollback. It also includes a compact pseudocode framework and practical rollout recommendations for turning subjective impressions into comparable, reproducible engineering decisions.

    Large model versions are iterating faster and faster, and discussion around “which one is the strongest” or “where it ranks on a leaderboard” keeps growing. For developers, these discussions are not without value, but they can easily drift away from the real engineering questions: whether a model can reliably complete tasks in real repositories, with complex dependencies, multi-file changes, and repeated testing; whether the total cost of completing a task is controllable; and whether, after integration, the model is easy to evaluate, replace, and govern. Evaluating the long-horizon engineering capability of a large model should shift from “can it generate a code snippet” to “can it deliver verifiable tasks.”

    1. Evaluation Focus: From Snippet Quality to Task Completion Rate

    Getting a model to write a piece of code that looks elegant is not difficult. The difficult part is getting it to understand the structure of an existing repository, locate the relevant files, form a modification plan, implement changes across multiple files, run tests, read failure logs, and continue correcting based on feedback. Long-horizon engineering tasks often involve multiple rounds of tool calls, context compression, error recovery, and side-effect control. If any link in this chain is unstable, it becomes human takeover cost.

    Therefore, when evaluating a model, you should not only look at the quality of a single-turn response. You should focus on the complete task loop: whether it understands the task boundary, whether it makes the necessary minimal changes, whether it follows project constraints, whether it can use tests and logs to verify results, and whether it converges after failure instead of repeatedly trying random fixes. For a team, the most valuable metric is not “a demo was impressive once,” but “within a given budget, how many tasks can be completed with less human takeover.”

    It is recommended to design evaluation tasks in layers, such as single-file fixes, cross-file refactoring, dependency upgrade adaptation, bug reproduction, test completion, documentation synchronization, and configuration migration. Each layer should have clear acceptance criteria and a resource budget, avoiding mixing simple completion tasks with long-horizon engineering tasks in the same statistics. Only stratified statistics can show whether model progress comes from improved generation ability or from improved planning, tool use, and error recovery.

    2. Four Core Metrics: Turning Subjective Impressions into Comparable Data

    2.1 First-Pass Success Rate

    First-pass success rate refers to the proportion of tasks that a model completes and passes acceptance in a single run without additional prompting or human correction. This metric best reflects the model’s task understanding and planning ability. When calculating it, the boundary of “first pass” should be clearly defined: whether automatic retries are allowed, whether the model may read failure logs and continue correcting, and whether humans may provide additional context. Different boundaries can significantly affect results, so they must be stated clearly in the evaluation report.

    First-pass success rate should also be combined with task difficulty layers. A high success rate on simple tasks does not mean strong long-horizon capability; a low success rate on complex tasks cannot simply be attributed to model capability, since it may be caused by unclear task descriptions, an unstable environment, or overly strict acceptance criteria. It is recommended to record failure types at the same time, such as localization errors, change scope too large, tests not passing, tool call limit exceeded, and context loss, so that these failures can be addressed later.

    2.2 Average Number of Tool Calls

    The number of tool calls reflects whether the model is repeatedly trying and failing, re-reading, or searching ineffectively. More tool calls are not necessarily worse, because complex tasks naturally require more exploration. However, if successful tasks also involve many repeated calls, this indicates problems with planning efficiency and state management. When calculating this metric, successful and failed tasks should be distinguished, and the proportion of invalid calls, the proportion of repeated reads, and the proportion of terminations due to exceeding budget should be recorded.

    This metric should also be observed together with token consumption, execution latency, and human takeover rate. One model may achieve a higher success rate through many calls, but the total cost may be unacceptable; another model may use few calls but often stops early or misses files. The goal of evaluation is not to minimize tool calls, but to find the balance among success rate, cost, and verifiability.

    2.3 Verifiable Result Ratio

    Verifiable result ratio refers to the proportion of task results that can be confirmed through automated tests, static checks, log assertions, structured output validation, or similar methods. Results that cannot be verified can only be considered candidate results and cannot be counted directly as stable delivery. Long-horizon engineering tasks especially need an evidence chain: which files were changed, why they were changed, whether tests passed, whether failure logs were handled, and whether additional side effects were introduced.

    During evaluation, automated acceptance should be preferred, with human review used only for ambiguity and boundary cases. At the same time, cheating behaviors such as “modifying acceptance scripts,” “bypassing tests,” and “hiding failures” should be prevented. Hidden tests, change scope limits, read-only acceptance environments, and command auditing can be set up to ensure results are reproducible and traceable.

    2.4 Total Cost per Successful Task

    The total cost per successful task should not look only at model pricing. It should include input and output tokens, tool execution resources, failed retries, human review, and waiting time. A more reasonable calculation is:

    Total cost / Number of successful tasks

    Total cost should cover consumption from failed tasks. Otherwise, a model may have a lower per-call price, but more failed retries and more human takeover may make the final cost higher. Long-tail task costs also matter: a small number of complex tasks may consume most of the budget. Evaluation reports should provide a cost distribution rather than only an average.

    3. Evaluation Design and Execution: Ensuring Reproducibility and Auditability

    The evaluation set should come as much as possible from real but sanitized engineering tasks, covering different levels of complexity and change scope. The environment needs to be isolated, using general-purpose container platforms or sandboxes to restrict network, resource, and file access. Dependencies should be cached and pinned to fixed versions. Initial code snapshots, prompt templates, tool configurations, and budget caps should all be recorded. Each run should retain logs, artifacts, and acceptance results so that different versions can be reproduced and compared.

    For scoring, automated acceptance should be preferred, with human review used only for parts that cannot be judged automatically. Task descriptions should clearly state acceptance conditions without revealing hidden tests. For long-horizon tasks, maximum rounds, maximum tool calls, and timeouts should be set. Once exceeded, execution should terminate and the reason should be recorded. Multiple repetitions should be run during analysis, and success rates and cost intervals should be reported to avoid misjudgment from a single run.

    4. Integration, Replacement, and Governance: Evaluation Must Extend into the Engineering System

    Model evaluation should not stop at offline testing. Whether a model is easy to replace, roll back, and govern after integration also determines total team cost. It is recommended to establish unified abstractions at the system layer: model invocation interfaces, tool registration, context management, budget control, audit logging, and result validation should be decoupled from one another. In this way, when changing model versions, there is no need to rewrite business logic. It is only necessary to replace the adapter layer and rerun the regression set.

    Version admission should include canary release, rollback strategy, and comparative evaluation. Core observability metrics include task success rate, human takeover rate, average cost, latency, and anomaly rate. For permissions, follow the principle of least privilege, mask sensitive data, and keep audit trails for tool calls. For cost governance, set budget caps, timeout policies, retry limits, and caching mechanisms. In cloud environments, general-purpose container platforms, logging services, and monitoring and alerting capabilities can be combined to form an observability loop, but evaluation standards should still focus on task completion and governance efficiency rather than a single feature checklist.

    5. Illustrative Evaluation Framework

    The following pseudocode only expresses the evaluation logic and does not depend on any specific product or interface:

    1. Define Task T:
    2. initial snapshot
    3. task description
    4. acceptance rules
    5. budget cap
    6. Run Task T:
    7. result = model.execute(T.description, T.initial_snapshot, T.budget_cap)
    8. acceptance = automated_acceptance(T.acceptance_rules, result)
    9. record success, tool call count, token consumption, human takeover, failure type
    10. Compute metrics:
    11. first-pass success rate = first-pass tasks / total tasks
    12. average tool calls = total tool calls / number of tasks
    13. verifiable result ratio = tasks passing automated acceptance / total tasks
    14. total cost per successful task = total resource cost / successful tasks

    6. Practical Recommendations

    Teams can begin by building a small baseline set covering typical long-horizon tasks, then gradually expand difficulty and language ecosystems. Every time a model version is updated, rerun the same regression set and record metric changes and failure cases. Release gates should include thresholds for success rate, cost, and verifiability, avoiding decisions based only on demo performance. After deployment, continuously monitor human takeover rate and long-tail cost, and feed real feedback back into the evaluation set.

    In summary, evaluating the long-horizon engineering capability of large models should not revolve only around rankings or “strongest” labels. A more reliable method is to answer three questions: Can it reliably complete real tasks at a controllable cost? Are the results verifiable and traceable? After integration, is it easy to evaluate, replace, and govern? Only by turning subjective experience into structured metrics can model selection become a reproducible engineering decision.