# Evaluating LLM Coding Ability: From Benchmark Hype to Real Cost-Effectiveness

Every time a new model ships, community discussion converges on two questions: how much the public benchmark score jumped, and whether the model is worth its price. In real engineering work, both questions are easy to misread.

Public benchmark scores measure pass rates under fixed prompts, fixed context, single-shot sampling, and no tool calls. The situations development teams actually face are close to the opposite:

• The context is not a clean single function but an existing repository tens of thousands of lines long;
• The task is not “complete this function” but “change one interface across 12 files and update the tests to match”;
• A single failure is not the end — agent frameworks automatically retry, read logs, run tests, and revise.

The gap between benchmark scores and hands-on experience is therefore not measurement noise. It is task distribution mismatch. The productive question is not “which model is strongest,” but “what does our task distribution actually look like?” Answer that first, then build your own evaluation set and cost model on top of it.

2. Decompose “Capability” into Three Layers of Metrics

Evaluating coding ability with one vague “accuracy” number is the fastest route to a bad decision. Split the assessment into three layers by task shape, and measure each with a different yardstick.

Layer 1: Single-Turn Generation Quality

This covers completion, single-function rewrites, SQL and regex generation, and unit test generation. Core metrics:

Metric	Definition	What to watch
First-attempt pass rate	Output passes acceptance criteria on the first try	Directly determines the interaction experience
Compile / static-check pass rate	Whether syntax, types, and lint pass	Reflects syntax and framework-constraint handling
Test pass rate	Whether generated tests compile and actually run	Separates “looks right” from “is right”
Out-of-bounds rate	Whether forbidden dependencies or unauthorized APIs are introduced	Security and compliance

The point of this layer is that it is fully automatable: every criterion can be produced by a script, which makes it suitable for large-scale scoring runs.

Layer 2: Repository-Level Change Ability

This is the layer where models separate most clearly, and the biggest source of disappointing real-world experience. Typical tasks: refactoring a legacy module to a new convention, bulk-replacing deprecated APIs, and locating a defect that spans files. What to evaluate:

• Localization accuracy: can the model select the correct key files without walking the entire repository;
• Change minimality: are the diff’s line count and file count within a reasonable range (over-rewriting is a serious risk);
• Regression break rate: do the pre-existing tests still pass after the change;
• Convergence rounds: how many interaction turns are needed to meet acceptance criteria with the given tool set.

Layer 3: Long-Horizon Agentic Tasks

These are multi-turn closed-loop tasks that require “read code → form a hypothesis → execute a command → inspect output → correct course.” At this layer, process metrics must be evaluated, not only the final outcome:

• Success rate of tool calls, and the share of wasted calls;
• Whether the agent loops on wrong paths (loop detection);
• Completion rate before the context budget is exhausted;
• Total token consumption and total wall-clock time per task.

A common finding: two models that are close on layers 1 and 2 can differ by more than 2x in completion rate at layer 3, because long-horizon tasks test error recovery, not isolated intelligence.

3. A Reproducible Evaluation Harness

Evaluation itself has to be engineered, otherwise results cannot be reproduced, compared, or tracked as a regression signal. Below is a skeleton you can adapt directly:

# eval_harness.py — evaluation harness sketch
import json, time, statistics
from dataclasses import dataclass, asdict
from typing import Callable
@dataclass
class TaskResult:
    task_id: str
    layer: str            # gen / repo / agent
    passed: bool          # whether the acceptance criteria were met
    attempts: int         # attempts consumed within the allowed automatic retries
    prompt_tokens: int
    completion_tokens: int
    tool_calls: int
    elapsed_ms: int
def run_layer(layer: str, tasks: list[dict],
              client: Callable[[str], dict]) -> list[TaskResult]:
    results = []
    for t in tasks:
        started = time.time()
        # constrain the output structurally so it can be graded automatically
        out = client(t["prompt"])
        passed = t["grader"](out)      # compile + unit tests + static checks
        results.append(TaskResult(
            task_id=t["id"], layer=layer, passed=passed,
            attempts=1,
            prompt_tokens=out["usage"]["input_tokens"],
            completion_tokens=out["usage"]["output_tokens"],
            tool_calls=out.get("tool_calls", 0),
            elapsed_ms=int((time.time() - started) * 1000),
        ))
    return results
def summarize(results: list[TaskResult]) -> dict:
    by_layer: dict[str, list[TaskResult]] = {}
    for r in results:
        by_layer.setdefault(r.layer, []).append(r)
    summary = {}
    for layer, rs in by_layer.items():
        n = len(rs)
        summary[layer] = {
            "pass_rate": round(sum(r.passed for r in rs) / n, 3),
            "p50_latency_ms": int(statistics.median(r.elapsed_ms for r in rs)),
            "avg_tokens": round(
                sum(r.prompt_tokens + r.completion_tokens for r in rs) / n, 1),
            "avg_tool_calls": round(sum(r.tool_calls for r in rs) / n, 2),
        }
    return summary

Four hard rules make the results trustworthy:

1. Criteria must be machine-verifiable. Samples where “is this good code?” requires human eyes should be removed from the main evaluation set and moved into a separate, small, manually reviewed set.
2. Fix temperature and sample count. Use temperature 0 or a very low value with a fixed random seed, and record variance across repeated samples so that noise is not mistaken for a capability gap.
3. Context budgets must be aligned. Models have different context windows; without a unified truncation strategy, you are comparing how you feed the model rather than the model itself.
4. The evaluation set must evolve. Feed real production failures back into the set on a quarterly cadence so it does not drift away from the business.

4. The Right Way to Measure Cost-Effectiveness: TCO, Not Price per Token

“Price per million tokens” is the single most misleading number in a purchasing decision. The real cost structure is:

TCO (per task) =
      input token cost (including repeatedly re-injected context)
    + output token cost
    + retry cost (amplified by 1 / first-attempt pass rate)
    + context bloat cost (cumulative overhead in long-horizon tasks)
    + human review cost (engineer hours spent on failed samples)
    + infrastructure cost (inference gateway, storage, logging, monitoring)

Three amplifiers are easy to overlook:

Retry amplification. If the first-attempt pass rate is 40%, a task takes 2.5 attempts on average to complete, so a model that is 30% cheaper per call can end up costing more in total. It is clearest in code:

def effective_cost_per_task(price_in, price_out,
                            in_tok, out_tok, pass_rate):
    attempts = 1 / max(pass_rate, 1e-6)      # expected attempts, geometric distribution
    return attempts * (in_tok * price_in + out_tok * price_out)

Context bloat. In agentic tasks, every turn re-injects history and code snippets, so input tokens grow linearly or worse. In a 20-turn task, the final turn’s input can be more than ten times the first.

Human review. For production code, an output that “looks right but needs 15 minutes of human fixing” can cost more than a high-confidence failure, because the latter can simply be discarded and retried.

Cost-effectiveness comparisons are therefore only meaningful under a fixed task distribution, a fixed context strategy, and a fixed retry limit. Anything else is not comparable.

5. Four Engineering Levers for Lower Cost

Once the model is chosen, the main remaining cost variables live in the engineering implementation.

1. Context Governance: Replace “Dump Everything” with Precise Retrieval

Most wasted tokens come from pushing the entire repository into context. Workable approaches:

• Use hybrid keyword plus vector retrieval to recall candidate snippets instead of injecting whole files;
• Prioritize injecting signatures and interface definitions, expanding function bodies on demand;
• Inject diffs of the previous version of files being modified, transmitting only what changed.

2. Tiered Routing: Send Simple Tasks Down the Cheap Path

Use a lightweight classifier — even regex rules will do — to separate task difficulty, routing completion, formatting, and simple Q&A to a low-cost path, and cross-file refactoring or complex defect localization to a high-capability path:

def route(task: dict) -> str:
    # route on task characteristics rather than model names, so the backend stays swappable
    if task["estimated_files"] <= 1 and task["est_lines"] < 200:
        return "fast_lane"
    if task["needs_multi_round"]:
        return "agent_lane"
    return "standard_lane"

3. Prompt and Result Caching

Apply semantic caching to repeated requests such as code explanations, documentation Q&A, and common dependency guidance. Enable prefix reuse for long system prompts, which meaningfully reduces billed input tokens on repeats. Make sure the cache key includes a code version identifier, otherwise you will serve stale answers.

4. Context Compression with Summaries Persisted to Disk

In long-horizon tasks, generate a structured summary every few rounds (confirmed facts, hypotheses still to verify, files already modified) and continue reasoning from the summary instead of the raw history. This both saves money and suppresses context pollution.

6. Constraints Beyond Raw Capability

Moving a coding model into a production development pipeline means clearing a few more gates:

• Data boundaries: whether the code is used for service improvement, whether offline deployment is possible, and how logs are retained. When core repositories are involved, this one is often a hard blocker.
• Observability: whether each call’s model identifier, version, token count, latency, and failure reason are traceable. Without this, neither cost optimization nor incident attribution is possible.
• Output safety: generated code must not pull in known high-risk dependencies, must not hardcode credentials, and must not bypass authentication. Add dependency vulnerability scanning and secret detection to CI.
• Vendor replaceability: keep model calls behind a unified gateway whose interface exposes only “input context + tool set → structured result,” making a backend switch a configuration change rather than a code rewrite.

7. Conclusion: Turn Selection into a Repeatable Experiment

The answer to “how good is this new model in practice, and is it cost-effective?” should never come from a single leaderboard or a single hands-on impression. It should come from a structured experiment:

1. Sample real development tasks from the last three months and build an evaluation set covering all three layers.
2. Fix the context strategy, retry limit, and grading criteria, then compare candidate models side by side.
3. Measure cost-effectiveness with a total cost of ownership model rather than unit price, and include human review time.
4. Roll out to a small slice of traffic, continuously feeding failed samples back into the evaluation set.
5. Keep the backend swappable through a unified gateway, turning model choice into a long-term adjustable knob.

One practical note: when building this kind of capability on Baidu AI Cloud, inference services, object storage, log services, and monitoring and alerting components can be composed into a complete evaluation and observability loop, with content safety capabilities to meet compliance requirements. For exact specifications and billing, refer to the official documentation. The methodology itself is vendor-agnostic — putting real rigor into evaluation has far more long-term value than arguing over any single version’s score.