Evaluating LLM Coding and Agent Capabilities: Efficiency, Cost, and Usability

    Leaderboard accuracy scores alone can't support model selection for real engineering work, which is more about delivering a task than answering a question. This article explains why coding and agent evaluations should focus on floor stability, iteration efficiency, and per-task cost, and how to put that into practice. It covers splitting coding ability into "floor" and "ceiling" curves, judging self-repair through defect-count convergence, assessing deliverable maintainability and interface completeness, managing agent-specific risks such as environment awareness and ineffective loops, computing cost per successful task, building reproducible evaluation sets, and landing results through tiered routing, deterministic verification loops, and observability.

    1. Why Leaderboard Scores Aren’t Enough for Model Selection

    Traditional evaluations center on single-turn question-answering accuracy, but real engineering work is closer to “delivering a task”: a model has to read the repository, plan steps, call tools, self-correct after failures, and finally produce code that runs or a result that’s actionable. Whether such tasks succeed doesn’t hinge on the accuracy of any single knowledge point; it hinges on three variables that leaderboards tend to downplay: floor stability, iteration efficiency, and per-task cost.

    Together these three form a decision triangle for engineering use. A model that shines on a few popular problems but frequently produces “half-finished” output on niche tech stacks or unconventional requirements will have its real productivity eaten up by rework. Conversely, a model whose ceiling isn’t dazzling but whose floor is solid and whose interactions take few rounds tends to make it into production pipelines more easily.

    2. Break Coding Ability into a “Floor” and a “Ceiling” Curve

    When evaluating code generation, it helps to separate the aspects you examine.

    Floor: baseline usability in unconventional scenarios. This covers uncommon frameworks and legacy tech stacks, obscure language features, cross-language calls, boundary-input handling, and error-branch coverage. In these scenarios, many models don’t “answer wrong” so much as “answer incompletely”: they omit necessary initialization config, skip exception handling, or return pseudocode that won’t compile. The right measure isn’t a pass rate but the directly deliverable ratio—the share of output that compiles and passes basic tests without any human completion.

    Ceiling: complex refactoring and long-chain reasoning. This includes multi-file refactoring, interface-contract changes, cross-module dependency adjustments, and performance-bottleneck localization. The gap on this axis usually shows up in “whether the model can produce a structurally correct plan in one shot,” not “whether it can fix one small bug.”

    Self-repair quality. A high-value but often overlooked observation: whether the model can correctly attribute bugs in its own output. Better behavior is localizing the true root cause after longer reasoning; worse behavior is repeatedly making equivalent edits at the same failing point. Fixing one thing while introducing new problems is common, so it’s worth quantifying with a convergence curve:

    1. Round 1 2 3 4 5
    2. Defects 6 5 4 4 3 # monotonically decreasing → worth iterating with the model
    3. Defects 5 6 5 6 5 # oscillating → needs human takeover

    If the defect count oscillates, the model’s edits are “trial-and-error” rather than “root-cause-driven.” At that point the marginal return of spending more rounds is low, and you should switch strategy as soon as possible.

    3. Deliverable Quality: Maintainability and Interface Completeness Beyond Functionality

    Correct functionality is only the passing grade. On the engineering side you also look at two groups of metrics.

    Structure and maintainability: naming consistency, module boundaries, proportion of duplicated code, whether config and logic are separated, and whether it leaves behind large amounts of meaningless comments and placeholder implementations.

    Interface and presentation completeness: these issues are extremely common in generative output—for example, a panel layout that’s only half done, broken proportions, missing control states, unhandled responsive breakpoints, and visible visual errors that the model judges as “as expected.” The last point is the most notable: the model lacks the ability to self-review its own output and won’t proactively doubt what it generated, which means manual visual walkthroughs can’t be skipped.

    4. Agent-Specific Risks: Environment Awareness and Ineffective Loops

    Agent task failures differ from single-turn generation, and the risk concentrates on “its interaction boundary with the environment.”

    The typical problem is misjudging capability boundaries. When a necessary tool is unavailable due to an isolation policy, the model may not understand the semantics of “unavailable” and instead repeatedly tries to work around the restriction, burning large amounts of time and call budget fighting the runtime environment rather than choosing an alternative path or clearly reporting that it’s blocked. This is especially common in isolated sandboxes and permission-restricted production environments.

    So the engineering constraints on the agent side should include:

    • Explicit tool-capability declarations: let the model know the tool list and their availability at the planning stage, rather than discovering it only after a failed call.
    • Structured failure feedback: unavailability should return a clear error with the reason and suggested actions, not a silent failure.
    • Ineffective-action detection: fingerprint by “tool name + normalized parameters” for deduplication, and interrupt on consecutive hits.
    • Hard budgets: triple caps on call count, wall-clock time, and token consumption; hitting any one triggers degradation and escalation.
    1. def run_agent(task, tools, budget):
    2. usage = {"calls": 0, "seconds": 0, "tokens": 0}
    3. seen = set()
    4. while True:
    5. usage["calls"] += 1
    6. if usage["calls"] > budget.max_calls or usage["seconds"] > budget.max_seconds:
    7. return escalate(task, reason="budget_exceeded", usage=usage)
    8. action = plan_next(task, history, tools) # model produces a structured action
    9. fingerprint = hash_action(action) # tool name + normalized parameters
    10. if fingerprint in seen:
    11. return escalate(task, reason="no_progress", usage=usage)
    12. seen.add(fingerprint)
    13. result = dispatch(action, tools)
    14. if result.blocked:
    15. # clearly state why it's unavailable and which alternatives exist, avoid fighting the sandbox
    16. result = dispatch(action, tools, fallback=True)
    17. history.append((action, result))

    escalate isn’t a failure—it’s handing a task that can’t converge automatically back to a human in time, which is itself a form of cost control.

    5. Measure Cost by “Effective Task Cost,” Not Unit Price

    Falling unit prices make it easy to overlook the real cost structure. A measure closer to engineering is cost per successful task:

    1. Effective task cost =
    2. (input tokens × input price + output tokens × output price + cache-hit tokens × cache price)
    3. / task first-pass success rate
    4. + human rework cost (rounds × average human time × hourly rate)
    1. def effective_cost(usage, price, first_pass_rate, human_rounds, hourly_rate):
    2. llm_cost = (
    3. usage["input"] * price["input"]
    4. + usage["output"] * price["output"]
    5. + usage["cached"] * price["cached"]
    6. )
    7. retry_factor = 1 / max(first_pass_rate, 1e-6)
    8. return llm_cost * retry_factor + human_rounds * hourly_rate

    This formula explains a counterintuitive conclusion: improvements in input-output efficiency often reduce total cost more than a lower unit price does. If one model needs four rounds on average to converge while another converges in two, then even if the former’s unit price is 30% lower, the latter may still cost less on an end-to-end basis—and it also saves wall-clock time and the human cost of waiting. So when you see a conclusion like “task cost dropped,” first confirm whether it’s measured on a token basis or an end-to-end basis.

    6. Build a Reproducible Evaluation Set

    For conclusions to be trustworthy, the evaluation set has to be trustworthy first. Organize it along these dimensions:

    Dimension Task type Key metrics
    Out-of-the-box usability Uncommon tech stacks, boundary inputs Compile pass rate, test pass rate, share of lines needing manual completion
    Complex implementation Multi-file features, cross-module changes First-pass rate, average interaction rounds
    Self-repair Codebases with injected defects Root-cause localization accuracy, defect-count convergence slope
    Aesthetics and completeness Frontend/interaction deliverables Layout completeness, visible defect count, self-review detection rate
    Agent execution Multi-step tool calls Task completion rate, ineffective-action share, budget-cap hit rate

    A few practical points:

    1. Keep hidden tasks. Public tasks are easy to optimize for, so a certain proportion of undisclosed problems is what reflects generalization ability.
    2. Fix the environment and the random seed, and record the model version, temperature, max context, and tool set; otherwise results across periods aren’t comparable.
    3. Sort after normalizing scores, and note that what’s shown is a comparable subset rather than the full ranking, to avoid readers misreading the standings.
    4. Record failure samples too. The distribution of failure modes is more instructive than the average score.

    7. Deployment Strategy: Tiered Routing + Verification Loop + Observability

    The conclusions of an evaluation ultimately have to land in the architecture. A three-layer design is recommended:

    Tiered routing. Classify tasks by complexity: route high-risk, high-complexity tasks to models with a stronger ceiling, and route high-volume, pattern-based tasks to low-cost models. Routing decisions can be based on heuristic rules (number of changed files, number of modules involved, whether external tools are needed) or a lightweight classification model.

    Verification loop. Don’t trust the model’s self-assessment; hand the output to deterministic checks: compilation, static analysis, unit tests, and interface screenshot comparison. Results that fail verification go back to the generation stage automatically, with structured error information attached as context for the next round.

    1. pipeline:
    2. - generate: { model: strong, max_rounds: 3 }
    3. - verify: { type: [compile, lint, unit_test] }
    4. - on_fail: { model: strong, feed: structured_error, max_rounds: 2 }
    5. - on_exceed: escalate_to_human

    Observability. Record each task’s call chain, token consumption, rounds, failure-reason classification, and number of human interventions; connect them to logging services, monitoring and alerting, and object storage for archiving and trend analysis. With this data, “cost reduction” can turn from marketing talk into metrics you can verify yourself. On the cloud-platform side, such capabilities are typically provided by general-purpose component categories (container platforms host the runtime sandbox, logging services and monitoring/alerting carry observability, and object storage archives evaluation artifacts), making it easy to engineer the evaluation pipeline itself.

    8. Key Takeaways

    • Break coding ability into two curves—floor stability and ceiling complexity—and use the defect-count convergence slope to judge whether it’s worth collaborating with.
    • Evaluate deliverable quality separately for maintainability and interface completeness; models lack self-review, so manual visual walkthroughs can’t be skipped.
    • The core on the agent side is environment awareness: explicit tool-capability declarations, structured failure feedback, ineffective-action detection, and hard budgets.
    • Measure cost as cost per successful task; efficiency often determines real spending more than unit price does.
    • Evaluations need a fixed environment, hidden tasks, a public normalized basis, and retained failure samples.
    • Production deployment depends on tiered routing, a deterministic verification loop, and complete observability data—not a single evaluation result.