Engineering Trade-offs in Evaluating AI Coding Agents on Long-Horizon Tasks
AI coding benchmarks are shifting from single-shot code generation to complete, real-world engineering tasks such as building app skeletons, implementing features, migrating legacy code, and fixing tests. This article explains why pass rate alone is a misleading metric, how to balance completion, pass rate, runtime, and cost in long-horizon task evaluation, and how to build a reproducible evaluation pipeline. It includes a practical four-dimension decision framework, a simplified cost model, pseudocode for metric aggregation and budget control, and actionable recommendations for development teams selecting AI coding agents.
Engineering Trade-offs in Evaluating AI Coding Agents on Long-Horizon Tasks
AI coding evaluation for mobile app development is moving from “complete a code snippet” toward “deliver a real engineering task.” A task may involve creating an application skeleton, implementing feature modules, migrating legacy code, converting application structure, and completing configurations and tests. Compared with single-point code generation, this kind of evaluation is far closer to the complex delivery work developers face daily, and it does a better job of exposing differences between agents in planning, tool calling, context management, error recovery, and cost control.
In public benchmarks, different agent approaches often diverge sharply on pass rate, average completion, average runtime, and per-run cost. Some approaches achieve high completion but at a significant cost in time and money; others have a mediocre pass rate but are more attractive in responsiveness and budget friendliness. Understanding what these metrics mean in engineering terms matters more than memorizing a leaderboard.
1. Pass Rate Alone Is Not Enough
Mobile app development benchmarks typically cover real-world tasks such as app creation, feature development, code migration, and application transformation. Their core metrics usually include:
- Pass rate: whether the task meets predefined acceptance criteria, such as compiling successfully, passing tests, and producing expected behavior.
- Average completion: the proportion of task steps, file changes, and feature points the agent completes. High completion does not guarantee a final pass, but it reflects the agent’s ability to push a task forward.
- Average runtime: the time from task start to final output, including planning, code generation, tool calls, and test fixes.
- Per-run cost: the cost of one full benchmark run, calculated from model calls, tool execution, and infrastructure resources.
Pass rate is the most intuitive metric, but it is a “result metric” and cannot by itself explain process quality. One agent may have a high pass rate while relying on longer attempts, more calls, and more human fallback; another may have a slightly lower pass rate yet deliver high completion within an acceptable time, making it a better fit for budget-sensitive teams or rapid iteration cycles. Reading evaluations therefore requires combining result metrics, process metrics, and cost metrics.
2. The Core Tension of Long-Horizon Tasks: Completion, Pass Rate, and Time Cost
Long-horizon tasks are hard because they cannot be finished in a single generation. They require multiple rounds of “observe—plan—execute—verify—fix.” The agent must maintain task state, understand the codebase structure, invoke build, test, and logging tools, and diagnose failures. The longer the task chain, the more uncertainty accumulates, and runtime and cost tend to rise non-linearly.
High completion usually means the agent can keep pushing the task forward and cover more files and feature points, but high completion does not guarantee a high pass rate. A common pattern: the agent modifies a large amount of code yet fails a critical test or introduces new configuration problems. In this case, high completion actually increases review overhead. Conversely, a solution with a high pass rate but very long runtime may suit offline batch jobs or overnight tasks, but not interactive development that demands fast feedback.
From an engineering perspective, long-horizon tasks should be broken into observable stages, such as environment setup, code localization, modification, build verification, test fixing, and delivery checks. Record the duration, call counts, failure reasons, and human intervention points for each stage. This makes it possible to both explain pass-rate differences and identify which stages consume the budget.
3. Cost Is More Than Unit Price: Per-Run Cost and Marginal Cost
Per-run cost is an important reference, but it cannot be reduced to “price per hour.” In long-horizon tasks, cost is composed of model calls, tool execution, sandbox resources, log storage, and failed retries. A low-cost solution with many retries and long human review times may not have a lower total cost of ownership, while an expensive solution that completes tasks in one pass with little rework may be more economical for critical workloads.
Teams can build a simplified cost model:
Per-task cost =model invocation cost+ tool execution cost+ sandbox resource cost+ failed-retry cost+ human review and rework cost
Human review and rework cost is frequently underestimated. For tasks like code migration and application transformation, agent-produced changes must be reviewed, tested, and merged. If completion is high but the change surface is large and poorly explained, review costs climb. If the pass rate is slightly lower but the output is well structured and failures are easy to locate, engineering teams can actually take over more easily.
4. A Four-Dimension Framework and Scenario-Based Selection
Evaluation results can be organized into a four-dimensional matrix of “pass rate—completion—runtime—cost,” with weights set by scenario:
- Fast-feedback scenarios: such as daily feature development assistance, code explanation, and local refactoring. Prioritize response time and per-task cost; the pass rate only needs to reach an acceptable threshold.
- Offline batch scenarios: such as bulk code migration and application structure conversion. Longer runtimes are acceptable; prioritize pass rate, completion, and reproducibility.
- High-value critical tasks: such as migrating core modules. Set quality gates and allow higher costs, but require test coverage, change documentation, and a rollback path for failures.
- Budget-sensitive teams: adopt a tiered strategy—use a low-cost configuration for simple tasks, escalate to a high-completion configuration for complex ones, and cap per-run cost with a budget limit.
Avoid making tool selections from a single leaderboard. A better approach is to build a small, team-specific task set covering your typical tech stack, codebase scale, and acceptance criteria, then cross-validate against public benchmarks.
5. Pipeline Practices: Sandbox, Observability, and Reproducibility
For evaluation conclusions to be trustworthy, the pipeline must guarantee isolation, observability, and reproducibility. Tasks should run in an isolated sandbox to avoid polluting the host environment; builds, tests, and tool calls should produce complete logs; each evaluation should pin code versions, dependency versions, task descriptions, and acceptance scripts. For long-horizon tasks, intermediate artifacts and retry trajectories should also be recorded.
At the infrastructure layer, teams can use general-purpose cloud offerings—such as those from Baidu AI Cloud, including object storage, container platforms, logging services, and monitoring and alerting—to host task artifacts, isolated execution environments, and observability data. This is a general infrastructure choice and does not change the evaluation methodology itself. The key is to correlate task artifacts, run logs, cost data, and acceptance results into a traceable evaluation archive.
6. Code Example: Metric Aggregation and Budget Control
The following product-agnostic pseudocode illustrates how to aggregate metrics and enforce a budget cap:
def evaluate_agent(tasks, budget_per_task):results = []for task in tasks:run = agent.run(task, timeout=task.timeout)cost = run.model_cost + run.tool_cost + run.sandbox_costif cost > budget_per_task:run.stop(reason="budget_exceeded")results.append({"task_id": task.id,"passed": run.tests_passed,"completion": run.completed_steps / run.total_steps,"duration": run.duration,"cost": cost,"retry_count": run.retry_count,"human_review_minutes": run.review_minutes})return aggregate(results)
When aggregating, output pass rate, average completion, average runtime, cost percentiles, and the human intervention rate together. Averages alone tend to hide long-tail tasks; percentiles help teams gauge the share of extreme-latency and high-cost runs.
7. Recommendations for Development Teams
First, treat public benchmarks as directional guidance, not as a procurement verdict. Second, build an internal task set that reflects your real codebase and acceptance workflow. Third, set budget, timeout, and human-takeover thresholds for long-horizon tasks. Fourth, review completion together with pass rate to avoid “lots of changes, but unusable output.” Fifth, pay attention to reproducibility and log quality—otherwise evaluation results cannot be explained. Sixth, adopt tiered routing and progressive automation: let agents handle low-risk, verifiable tasks first, then gradually extend to migration and transformation scenarios.
The value of long-horizon task evaluation lies not in producing a simple ranking, but in helping teams understand the engineering trade-offs among quality, speed, cost, and controllability. Only by translating evaluation metrics into enforceable engineering gates and budget strategies can AI coding agents genuinely enter a stable delivery process.
