How to Evaluate Large Models on Long-Horizon Engineering Tasks

    A practical framework for evaluating whether large models can complete complex software tasks reliably and economically. The article introduces measurable criteria for first-pass success, tool usage, verifiable outcomes, and cost per successful task, then explains how to build reproducible evaluations and turn the results into decisions about integration, replacement, and governance.

    Discussions about large-model capability often get stuck on rankings and parameter counts. For developers and enterprises, a more important question is whether a model can reliably complete long-horizon engineering tasks at a controlled cost—and whether it can be evaluated, replaced, and governed after integration.

    This article presents a general evaluation framework rather than assessing any particular model or product. It focuses on three questions: How should task completion be defined? How can costs be quantified? How can evaluation results support engineering decisions?

    1. Redefine the Problem: Long-Horizon Engineering Is More Than One-Shot Code Generation

    Long-horizon engineering tasks typically involve understanding an existing codebase, tracing dependencies across files, creating a modification plan, performing multiple edits, running validation, interpreting failure messages, and continuing with corrective actions.

    This is fundamentally different from “writing a good piece of code.” The former tests persistent state management, context compression, error recovery, and result validation. The latter focuses primarily on local generation quality.

    Evaluation should therefore divide the task into observable stages:

    1. Understanding: Can the model identify relevant modules, dependencies, constraints, and established conventions?
    2. Planning: Can it produce executable steps rather than general recommendations?
    3. Modification: Can it maintain consistency across files and avoid changes that are locally correct but globally conflicting?
    4. Validation: Can it proactively run tests, inspect logs, and compare structured outputs?
    5. Repair: After a failure, can it use feedback to converge instead of repeating trial and error?
    6. Delivery: Can it provide a change summary, risk explanation, and reviewable evidence?

    Evaluating only the final code fragment can hide the cost of retries, human intervention, and validation. Meaningful improvements should appear in end-to-end task completion rates, fewer human takeovers, and more stable recovery after failures.

    2. Four Core Metrics: From “Appears Usable” to “Measurably Usable”

    2.1 First-Pass Success Rate: Measuring Completion Without Additional Guidance

    First-pass success rate is the percentage of tasks that pass acceptance criteria without additional human prompts or manual corrections. It is closer to real delivery efficiency than a task that succeeds only after an extended conversation.

    Results should be reported by difficulty level, such as:

    • Single-file changes
    • Cross-file changes
    • Multi-module refactoring
    • Bug fixing
    • Long validation workflows

    If simple tasks account for too large a share of the evaluation set, they can inflate the overall score and conceal instability on long-horizon tasks.

    It is also useful to record four outcomes separately:

    • Successful on the first pass
    • Successful after additional prompts
    • Successful after human takeover
    • Ultimately failed

    First-pass success rate is not the only metric, but it directly indicates how much explanation and correction a team must provide.

    2.2 Average Tool Calls: Measuring Trial-and-Error Efficiency and Stability

    Tool calls may include retrieving context, reading and writing files, running validation commands, inspecting logs, and generating structured reports.

    A very low number of tool calls may indicate insufficient validation. A very high number may indicate repeated trial and error, inefficient context usage, or unstable planning. Evaluation should therefore track:

    • Total tool calls
    • Invalid tool calls
    • Repeated tool calls
    • Recovery calls made after failures

    In engineering environments, tool-call counts should be normalized by task complexity. Report the mean, median, and upper percentiles for each task category. Upper-percentile results are particularly important because a small number of extreme tasks can consume a disproportionate amount of budget and slow overall delivery.

    2.3 Verifiable Outcome Ratio: Measuring Whether Results Are Supported by Evidence

    A model saying that a task is complete does not mean that it is complete.

    The verifiable outcome ratio can be defined as the percentage of task results supported by tests, log inspection, structured outputs, a change summary, or other reviewable evidence. Verification strength can be classified into levels:

    • Weak verification: A natural-language completion statement only
    • Moderate verification: Structured output is provided
    • Strong verification: Reproducible validation records are available

    This metric helps reduce the risk of delivering results that merely appear correct. In enterprise environments, verifiability is often more important than one-time generation quality because it determines whether results can enter review, audit, and regression workflows.

    2.4 Total Cost per Successful Task: Measuring Real Economic Value

    Cost should not be reduced to model pricing. The total cost of a successful task should include inference, tool calls, failed retries, environment execution, and human takeover, divided by the number of successful tasks.

    The formulas can be simplified as follows:

    1. Total task cost = inference cost + tool-call cost + retry cost + environment cost + human takeover cost
    2. Cost per successful task = total cost of all tasks / number of successful tasks

    This metric exposes two common patterns: a low-cost approach that fails repeatedly, and a highly capable approach whose validation process is too expensive.

    Only by including retry and human-intervention costs can an organization determine whether a solution is suitable for use at scale.

    3. Evaluation Sets and Execution: Making Results Reproducible and Comparable

    Evaluation sets should be derived from de-identified real-world engineering tasks and cover different difficulty levels and failure modes. Each task should specify:

    • Input
    • Permitted context
    • Acceptance criteria
    • Budget limits
    • Required tracking fields

    Execution environments should be isolated to prevent tasks from affecting one another. Each task should ideally be run multiple times, with the mean, median, and upper percentiles reported instead of only the best result.

    The following pseudocode illustrates how one task run can be recorded:

    1. task = {
    2. "id": "T-001",
    3. "layer": "cross-file modification",
    4. "prompt": "Implement the requested change in the existing module while preserving compatibility",
    5. "acceptance": ["validation passed", "no abnormal logs", "complete change summary"],
    6. "budget": {"tool_calls": 30, "time_seconds": 600}
    7. }
    8. record = {
    9. "task_id": "T-001",
    10. "first_pass": false,
    11. "accepted": true,
    12. "turns": 4,
    13. "tool_calls": 37,
    14. "invalid_calls": 6,
    15. "verified": true,
    16. "human_takeover_minutes": 12,
    17. "total_cost": 3.2
    18. }
    19. function summarize(records):
    20. total = length(records)
    21. successes = filter(records, r -> r.accepted)
    22. first_pass_rate = count(records, r -> r.first_pass and r.accepted) / total
    23. avg_tool_calls = average(records.tool_calls)
    24. verified_ratio = count(records, r -> r.verified) / total
    25. cost_per_success = sum(records.total_cost) / max(length(successes), 1)
    26. return first_pass_rate, avg_tool_calls, verified_ratio, cost_per_success

    A recommended evaluation workflow is:

    1. Build a baseline task set.
    2. Run the baseline version.
    3. Record execution traces and costs.
    4. Aggregate metrics by task category.
    5. Analyze the causes of failed samples.
    6. Create a regression set.

    Whenever a model is replaced or the integration strategy changes, run the regression set before deciding whether to expand deployment gradually. Evaluation should be treated as an ongoing governance process rather than a one-time model-selection exercise.

    4. Integration, Replacement, and Governance: Engineering Capabilities Beyond Evaluation

    Model assessment should consider not only performance but also how easily the model can be replaced and governed after integration.

    A unified integration layer can decouple task inputs, context construction, tool permissions, output protocols, and validators. This allows teams to replace a model without making extensive changes to business code or evaluation sets, while preserving the comparability of historical metrics.

    At a minimum, the governance layer should include:

    • Least-privilege access: Prevent tasks from accessing unrelated data.
    • Redacted logging and auditing: Record important calls and validation evidence while protecting sensitive information.
    • Cost budgets and alerts: Intercept abnormal call patterns and high-cost tasks in a timely manner.
    • Version baselines and regression sets: Ensure every change is traceable.
    • Gradual rollout and rollback: Validate changes with limited traffic before expanding deployment.

    General cloud capabilities such as object storage, logging services, monitoring and alerting, and container platforms can support evaluation artifacts, execution records, and isolated environments. However, technology choices should be based on general categories and actual requirements rather than on tightly coupled dependencies that make replacement difficult.

    5. Implementation Recommendations: Start Small and Establish a Team Baseline

    There is no need to build a comprehensive evaluation system immediately. A practical rollout can follow these steps:

    1. Select 10 to 20 real long-horizon tasks covering understanding, modification, validation, and repair.
    2. Define acceptance criteria and cost accounting rules, ensuring that retries and human takeovers are recorded.
    3. Run the tasks repeatedly and calculate the four core metrics.
    4. Analyze failed samples and distinguish among insufficient context, incorrect planning, missing validation, and improper tool usage.
    5. Add stable tasks to a regression set and establish an internal team baseline.

    The final decision should not focus only on whether a model is “the strongest.” It should ask whether the model is more stable, verifiable, and governable for the organization’s target task distribution.

    When first-pass success, tool-call efficiency, verifiability, and total cost per successful task can all be measured continuously, a large model’s long-horizon engineering capability can move beyond demo performance and become a quantifiable source of productivity.