Evaluating LLM Coding Ability: A Three-Dimensional Framework for Quality, Efficiency, and Cost

    LLM coding performance cannot be judged reliably by a single leaderboard score. A practical evaluation framework must track model versions and execution modes over time, separate engineering quality from domain knowledge, measure efficiency through steps and tool usage, and calculate the true cost of completing a task—including human rework. This article presents a reproducible protocol for coding agents, with task tiers, stability metrics, long-tail knowledge handling, failure-mode analysis, and routing strategies for production use.

    1. The First Evaluation Question Is Not “Which Model Is Stronger?” but “Is the Result Timely and Reproducible?”

    Leading open-weight models now iterate on a monthly—or even shorter—cycle. A model may remain among the most useful options for only two months before a newer version surpasses it. The same model may also support reasoning mode and direct-response mode, with major differences in capability, latency, and cost.

    As a result, static rankings become obsolete quickly. Evaluation must move from one-time comparisons to continuous tracking.

    At least four engineering requirements are necessary for meaningful tracking:

    1. Self-describing data points: Every score must be tied to a model identifier, reasoning mode, evaluation-set version, sampling parameters, and execution date. Without this information, readers cannot determine whether results are comparable.
    2. Robust statistics: Sort by median score in descending order rather than relying on the mean, which can be distorted by outliers. Preserve best- and worst-case ranges to observe stability and variance.
    3. Clearly identified subsets: If a table shows only selected models for comparison, it must explicitly state that it is not a complete ranking. Otherwise, readers may interpret it as an absolute leaderboard.
    4. Complete methods and footnotes: The task source, scoring scripts, sandbox environment, and whether network access or external tools are allowed must be documented well enough for reproduction.

    An example evaluation record might look like this:

    1. eval_record:
    2. model: <model identifier and version>
    3. mode: reasoning | non_reasoning
    4. eval_set: coding-agent-v3
    5. eval_set_version: "2026-07"
    6. score_median: <value>
    7. score_p10/p90: <range>
    8. cost_per_task: <unit task-completion cost>
    9. tools_allowed: [file editing, command execution, static checks, tests]

    2. Coding Ability Must Be Decomposed into Dimensions

    A Broad Knowledge Base Is Not the Same as Engineering Delivery

    The most visible benefits of very large models are broad knowledge coverage and aesthetic judgment. These models are more likely to choose natural interface names, implement complete error handling, consider edge cases, and produce interfaces that follow familiar conventions. Such benefits are difficult for smaller models to reproduce in open-ended design tasks.

    However, for programming tasks with clear boundaries and repetitive patterns, medium-sized, lower-cost models can perform very close to top-tier models. In simple bug fixes or single-file refactoring, their outputs may be nearly indistinguishable.

    The main differences usually appear in three areas:

    • Overall design quality
    • Knowledge of edge cases and less common technologies
    • Retrieval of constraints from long contexts

    Therefore, the right model-selection question is not “Which model ranks first overall?” but “Which dimensions dominate my task distribution?” The following dimensions can be scored individually:

    Dimension What to observe Typical weakness
    Instruction following Whether all original requirements are implemented Randomly ignoring constraints or omitting features
    Cross-file consistency Whether multi-file changes remain coherent Local correctness but global compilation failure
    Ecosystem and API knowledge Whether dependencies are used accurately Inventing APIs from memory
    Long-tail platform coverage Support for niche ecosystems and private dependencies Failing to use even basic static checks
    Interaction and design details Feedback, error messages, and edge-case behavior Complete functionality but rough user experience
    Self-verification Whether checks and tests are run proactively Treating delivery as complete without validation
    Long-context retrieval Retaining constraints across large repositories and conversations Forgetting earlier agreements and repeating mistakes
    Unit cost Total cost of completing a task Excessive steps and bloated context

    3. Calculate Cost per Completed Task, Not Just Unit Pricing

    Token pricing is only one component of the total cost. A more realistic estimate is:

    1. cost_per_task ≈ Σ_steps (in_tokens × p_in + out_tokens × p_out)
    2. + human_rework_cost
    3. (failed_acceptance_count × review_cost_per_failure)

    Three factors are especially compressible:

    • Number of steps: Inefficient exploration and repeatedly reading the same files
    • Context volume: Verbose logs and duplicated content
    • Number of self-test rounds

    Different model types can produce very different cost curves:

    Cost factor High-self-testing model Low-self-testing model
    Average steps Baseline Significantly lower; some examples are roughly half as high
    Self-test and repair rounds High share, mainly spent on validation Low share; tends to call tools concurrently
    Nominal token cost Baseline Lower; some examples range from one-third to one-half
    Human acceptance cost Low because issues are caught internally High because problems reach reviewers
    Overall TCO Depends on task risk Competitive for low-risk tasks

    Two conclusions follow.

    First, the advantage of a low-self-testing model comes from replacing serial waiting with concurrent throughput and outsourcing validation to CI—not from eliminating verification.

    Second, once the unit price rises, the product of steps and context becomes a dominant cost driver. Context trimming, result caching, and concurrent calls become more valuable, and the cost-effectiveness ranking of models can change accordingly.

    4. Designing an Evaluation Protocol for Agentic Coding

    A single average score is not enough for model selection. A useful benchmark should be divided by task difficulty:

    • L1: Small single-file changes — Targeted bug fixes, constant updates, and additional logging.
    • L2: Cross-file features — Adding an API and connecting service and presentation layers.
    • L3: Refactoring and migration — Framework upgrades, dependency replacement, and directory restructuring.
    • L4: Long-tail platforms and private dependencies — Niche mobile ecosystems, internal SDKs, and restricted execution environments.

    Recommended core metrics include:

    • First-pass success rate and requirement omission rate. Omissions are often more dangerous than incorrect implementations and should be reported separately.
    • Median and P90 step counts, which reflect efficiency and long-tail delays.
    • Percentage of concurrent tool calls, which indicates the scheduling strategy.
    • Self-test coverage, including whether the agent runs type checks, static analysis, and tests.
    • Unit cost per completed task.
    • Variation stability: Run the same task N times and measure whether serious omissions occur intermittently. Two models with similar averages are not equivalent in production if one has very low variance while the other occasionally drops critical requirements.

    The evaluation framework can be abstracted as follows:

    1. def evaluate(model, tasks, repeats=3):
    2. records = []
    3. for task in tasks:
    4. for _ in range(repeats):
    5. run = agent.run(model, task, budget=B)
    6. records.append({
    7. "pass": checker.verify(run, task.acceptance),
    8. "missing": acceptance_diff(task.acceptance, run),
    9. "steps": run.steps,
    10. "parallel": run.parallel_ratio,
    11. "selftest": run.selftest_rounds,
    12. "cost": run.tokens_in * p_in + run.tokens_out * p_out,
    13. })
    14. return summarize(records) # median + P90 + coefficient of variation + omission rate

    5. Long-Tail Knowledge and Ecosystem Coverage Are Common Failure Points

    A typical pattern is that the same model performs well in a mature front-end ecosystem. It proactively uses type checks, build validation, and component conventions. When asked to implement similar functionality for a niche mobile platform or a private dependency, however, it may skip even basic static checks and write API calls from memory.

    This is not simply a degradation in implementation quality. It reflects a difference in knowledge density: the model does not know what it should verify, not merely how to implement it incorrectly.

    For tasks that depend heavily on external technical knowledge, such models often need more retrieval or dependency-source inspection before implementation. The direct consequence is increased uncertainty: two runs of the same task may produce two different technical approaches.

    The engineering response should not rely solely on changing models. Long-tail knowledge should be externalized:

    1. Project-level rule files: Declare the technology stack, build commands, acceptance commands, forbidden patterns, and platform differences. Send these rules with every task.
    2. Retrieval first: Search the project’s symbol index and dependency source code before allowing the model to modify the codebase.
    3. Turn constraints into gates: Enforce type checks, static analysis, and contract tests in CI. This converts team knowledge into tooling instead of relying on model memory.
    4. Fill knowledge gaps externally: Maintain a platform-difference checklist that specifies unavailable capabilities and required fallback behavior.
    1. # Project rules (example)
    2. stack: mobile client + local service
    3. build: <build command>
    4. verify: <static checks> && <type checks> && <contract tests>
    5. forbidden: [string-built queries, network calls without timeouts,
    6. plaintext secrets, direct reliance on internal implementation details]
    7. platform_notes:
    8. - The target platform lacks a basic capability; implement a fallback branch
    9. - Pin third-party dependency versions; automatic upgrades are forbidden

    6. Failure Modes Matter More Than Scores

    Two models with similar total scores may have completely different defect profiles.

    One model may produce imperfect interaction details: awkward wording, vague error messages, or rough edge-case handling. These issues can usually be refined and may have limited impact.

    Another model may randomly ignore original requirements. A specification may contain three requirements, while the implementation includes only two, with no warning. This is more serious and may make the entire delivery unusable.

    The difference often comes from self-testing strategy. A model that tests frequently can expose and fix omissions internally, creating an effective safety net. A model that performs little self-testing uses fewer steps and has a lower nominal cost, but transfers more risk to human acceptance.

    Evaluation reports should therefore include omission rate and coefficient of variation as first-class metrics rather than reporting only average scores.

    Two practical approaches are:

    • Structure and number acceptance criteria: Require the model to report completion for each item, creating a traceable chain.
    • Compare acceptance differences in the harness: Any item that is not acknowledged or is marked as partial should be treated as a failure.
    1. Acceptance checklist Model response
    2. A1 Pagination parameters have defaults A1 done (services/list.py:42)
    3. A2 Empty results return an empty array A2 done
    4. rather than null
    5. A3 Failure paths include timeout and retry A3 partial (retry missing)
    6. → Result: failed; return to the evaluation sample

    7. The Marginal Value of More Reasoning and Longer Context

    Improvements in reasoning capability are often not a fundamental paradigm shift. They are closer to incremental gains obtained through longer and deeper chains of thought. Latency and token cost increase approximately linearly, while the benefits show diminishing returns.

    In scenarios that require strong context retrieval—such as large-repository migrations, multi-turn development, and long-log debugging—the ability to retain earlier constraints is more important than the length of the reasoning chain. When retrieval is inadequate, simply increasing the thinking budget cannot compensate.

    The appropriate strategy is to improve retrieval through engineering rather than continuously increasing reasoning effort:

    • Layered memory: Use a task summary, a key-symbol list, and retrieved snippets instead of pasting the entire context.
    • Pre-action restatement: Before making changes, ask the model to restate the relevant constraints and acceptance criteria.
    • Separate evaluation categories: For retrieval-intensive tasks, prioritize retrieval metrics instead of relying on an overall score.

    8. Practical Recommendations: Tiered Routing and Cost Guardrails

    The conclusions above can be turned into an actionable selection and governance strategy:

    1. Route by tier: Assign low-risk L1 and L2 tasks to cost-effective models. Escalate L3 and L4 tasks, as well as tasks requiring strong context retrieval, to stronger models.
    2. Set guardrails: Define limits for steps, token budgets, execution timeouts, and concurrency. Automatically fall back to a stronger model after failure.
    3. Enforce a closed loop: Sandbox execution, static checks, and tests must all pass before a task is considered complete. Models with weak self-testing must rely on CI for validation.
    4. Make performance observable and attributable: Record task logs and attribute cost through dashboards by task type. Continuously feed failed samples back into the evaluation set so that the benchmark evolves with the models.
    5. Keep infrastructure choices generic: Artifacts and evaluation samples can be stored in object storage, task decoupling can use a message queue, sandbox execution can run on a container platform, and observability can rely on logging and monitoring services. Choose specific products according to compliance and cost requirements rather than binding the evaluation to a single vendor or conclusion.
    1. def route(task):
    2. if task.tier in (L1, L2) and not task.risky:
    3. return run(cheap_model, task, budget=B1, selftest="ci")
    4. if task.tier in (L3, L4) or task.needs_recall:
    5. return run(strong_model, task, budget=B2,
    6. extra_context=retrieve(task), selftest="ci+agent")
    7. return run(strong_model, task, budget=B2)

    Conclusion

    LLM coding ability can no longer be evaluated through a single overall score. A more reliable process is to define quality dimensions by task tier, explain efficiency through average steps, concurrency strategy, and self-test coverage, and calculate real operating cost through unit task-completion cost and human rework.

    Long-tail ecosystem knowledge should be externalized into rule files and CI gates, while high-risk failure modes such as random requirement omissions should be reported as first-class metrics.

    This approach produces results that remain comparable across model versions. In an environment where a model may be replaced within two months, the evaluation system itself becomes one of the most important technical assets.