Evaluating LLM Coding Agents: Benchmarks, Harnesses, and Cost-Performance Assessment

1. First, correct a premise: leaderboard scores cannot replace in-pipeline testing

When a new coding model is released, a common mistake is to infer how it will perform in your own engineering system from public benchmark scores. Those scores are obtained under specific prompt templates, tool sets, and context-management strategies. Production environments differ at almost every step: codebase size, retrieval method, test-suite quality, retry policy, task granularity, and context-compaction threshold. The same model weights can show pass-rate differences of more than ten percentage points under different harnesses.

The first principle of evaluation is therefore: any meaningful comparison must be conducted at the harness level. Comparing two models across different harnesses produces a mixed result of “model × engineering implementation” and cannot be attributed to the model alone.

At a minimum, the following harness variables must be fixed or explicitly recorded:

• Context construction: whole-repo injection, repo map plus retrieval, or directory-level slicing.
• Tools and permissions: read/write files, run tests, run builds, search code, query logs.
• Loop control: maximum turns, retries after tool errors, and timeout policy.
• Context compaction: whether to summarize when approaching the context-window limit, and the compaction trigger threshold.
• Sampling parameters: temperature, top_p, and thinking budget mode (off / adaptive / high).
• Evaluation criteria: unit tests, integration tests, human review, or automated evaluation.

The following harness description can serve as a minimal reproducible unit. Archive it together with results during review so that conclusions are comparable:

harness:
  name: repo-agent-v3
  context:
    strategy: repo-map + retrieval   # or whole-repo / dir-slice
    max_context_tokens: 200000
    compaction: { trigger_ratio: 0.75, keep_recent_turns: 6 }
  tools: [read_file, edit_file, run_test, grep, list_dir]
  loop:
    max_turns: 30
    retry_on_tool_error: 2
    temperature: 0.2
  thinking:
    mode: adaptive        # off | adaptive | high
    budget_tokens: null   # null means the harness allocates the budget adaptively

2. Use the same yardstick for cost and benefit

The debate over whether a new model’s capability justifies its price after a price increase is essentially a disagreement about cost measurement. At least three layers should be separated.

1. Direct cost per task

The full billing scope for one task, not just the output of the final turn:

Task cost = (input tokens × input price + cached tokens × cached price
           + output tokens × output price + reasoning tokens × corresponding price)
           × (1 + number of retries)

For multi-agent coding tasks, most token consumption is often on the input side: repeatedly reading files, repeatedly feeding test output back, and retrying with the full context. Looking only at the output price can severely underestimate real spending.

2. Effective cost: cost per successful task

This is the only metric suitable for procurement and model-selection decisions:

Effective cost = total spend on the same batch of tasks / number of tasks that pass on the first attempt

A solution that doubles the pass rate but also doubles the unit price does not necessarily reduce effective cost.

3. Marginal benefit curve

With the harness and evaluation set fixed, scan thinking budgets or reasoning tiers from low to high, and record token consumption, pass rate, and effective cost for each level. Plot the curve and identify the inflection point. For most tasks, benefits saturate at medium tiers; increasing the thinking budget further mainly leads to linear token growth and higher latency.

The following illustrative data shows why effective cost must be calculated. The values are examples used only to explain the method and do not represent any specific model:

Configuration	Thinking level	Average tokens/task	Relative task cost	First-pass rate	Relative effective cost
Baseline	Low	118k	1.00	41%	2.44
With thinking	High	274k	2.30	52%	4.42

Task cost increases to 2.3×, the first-pass rate rises from 41% to 52%, and effective cost increases by about 80%. This does not mean that stronger thinking is useless. It means that globally increasing the thinking level without task-level routing usually produces a negative cost-performance trade-off.

3. How to build an evaluation set that measures accurately

• Source from real work. Sample from internal tickets, code-review comments, and production defects rather than toy problems. Each task should retain the original repository snapshot and a definition of expected behavior.
• Stratify by difficulty. Separate tasks into categories such as single-file changes, cross-file changes, tasks requiring new tests, tasks requiring understanding of existing abstractions, and tasks requiring debugging external dependencies. Report each layer separately to avoid averages being dominated by easy tasks.
• Make criteria automatable. Prefer existing repository tests and build checks. For cases that cannot be judged automatically, use a fixed scoring rubric with blind review, position-bias correction, and double-review sampling.
• Ensure scale and stability. Use at least dozens of tasks per layer. Run the same configuration multiple times and report means and confidence intervals, so single-run fluctuations are not mistaken for version differences.
• Freeze versions. Tag and store the evaluation set. Reuse the same snapshot for regression testing of new versions to prevent data contamination and prompt drift.

4. Skeleton of an evaluation pipeline

The key is to do accounting inside the pipeline rather than estimating it afterward. Use a unified record of usage and latency, then aggregate at the end:

PRICE = {  # Fill in according to the commercial contract; unit: currency units / million tokens
    "input": 0.00, "cached": 0.00, "output": 0.00, "reasoning": 0.00,
}
def task_cost(usage: dict, retries: int) -> float:
    total = sum(usage.get(k, 0) / 1_000_000 * PRICE[k]
                for k in PRICE)
    return total * (1 + retries)
def summarize(records):
    n = len(records)
    passed = [r for r in records if r.passed]
    spent = sum(r.cost for r in records)
    toks = sum(r.usage_total for r in records)
    return {
        "tasks": n,
        "pass_rate": len(passed) / n,
        "spend": spent,
        "tokens_per_task": toks / n,
        "unit_cost": spent / n,
        "effective_cost": spent / len(passed) if passed else None,
        "p50_latency": percentile([r.latency for r in records], 0.5),
        "regression_rate": sum(r.regressed for r in records) / n,
        "human_touch_rate": sum(r.needs_human for r in records) / n,
    }

For engineering implementation, store raw logs and traces of evaluation tasks in object storage, write structured metrics to a queryable logging and monitoring system, and generate cost dashboards and alerts from the same data. Manage credentials through a secret-management service; do not hard-code them in evaluation scripts. On general cloud platforms, model services, object storage, logging, and monitoring/alerting are common composable building blocks. The evaluation approach itself is cloud-agnostic.

5. Turning evaluation conclusions into cost-optimization actions

• Task-level routing. Use rules or a small model to predict task difficulty. Single-file fixes, formatting, and comment additions can use lower tiers or smaller models; cross-module refactors and concurrency defects should use higher tiers. This is the most direct way to reduce effective cost.
• Prioritize context engineering over model switching. Use stable prefixes to improve cache hit rates, truncate and summarize tool results, retrieve by directory slicing, and avoid whole-repo injection. Raising cache hit rate from 20% to 60% often yields more benefit than switching models.
• Control retry amplification. Tool-error retries and full-task retries both incur additional billed cost. Set a total budget cap and turn cap for each task; degrade or hand off to humans when limits are exceeded.
• Use parallelism and batching. Run evaluation and offline batch jobs through asynchronous queues to smooth peaks; use streaming responses for interactive scenarios to improve perceived latency.
• Prefer gradual rollout over full switching. Run a new version against shadow traffic and a fixed evaluation set first. Compare pass rate, regression rate, human-intervention rate, and effective cost. Increase traffic proportionally only after targets are met, and keep a one-click rollback.

6. Common pitfalls

1. Inferring your own pipeline’s performance from public leaderboards.
2. Comparing only unit prices rather than cost per successful task.
3. Globally increasing thinking levels without task-level routing.
4. Drawing conclusions from a single run and ignoring variance.
5. Looking only at pass rate without regression rate and human-intervention rate—high pass rate accompanied by high regression rate is negative value in real-world delivery.
6. Comparing different versions under different harnesses or prompts, and attributing engineering differences to the model.

In one sentence: whether a version number represents incremental progress or a capability leap should not be judged by naming or marketing. Instead, run controlled comparisons on your own harness with a frozen evaluation set, and cross-validate using effective cost, regression rate, and human-intervention rate. Once this process is in place, every new release only needs a regression run to produce a trustworthy answer for your own business.