Evaluating Coding LLMs: Real-World Capability and Cost-Effectiveness Methodology

1. Why “real capability” is hard to state in one sentence

Two common discussions dominate the community around new coding models: “How much have the benchmark scores improved?” and “The experience seems about the same.” Neither is an engineering conclusion.

Benchmarks suffer from distribution shift. Public evaluation sets usually contain cleaned problems: requirements are complete, context is self-consistent, and acceptance criteria are clear. Real development tasks are often different. A requirement may be a single sentence. Constraints may be scattered across meeting notes and historical commits. Acceptance criteria may need to be inferred. When a task has been cleaned up, differences between model tiers are compressed. It is like facing an empty goal: almost anyone can score, so shooting quality no longer distinguishes players.

Anecdotal experience suffers from sample size. Daily Q&A, single-file rewrites, and comment completion place low demands on reasoning depth. They mostly fall within the comfort zone of mainstream models. Using them to judge a model’s ceiling is like using an elementary school exam to distinguish graduate students.

The correct approach is not to ask “Which model is stronger?” Instead, answer three verifiable questions: on what types of tasks is it stronger, by how much, and how expensive is each unit of effective output?

2. Establish task tiers: differences appear only in certain ranges

In practice, capability gaps between models are not uniform. They concentrate in the medium-to-high range of task complexity. Tasks can be positioned using three orthogonal dimensions:

• Requirement ambiguity: Are the requirements complete? Are there implicit constraints that must be inferred?
• Chain length: How many files need to be changed? How many layers of calls are involved? How many tool calls are required?
• State dependency: Does a later step strongly depend on intermediate artifacts from earlier steps? If something goes wrong, can it be detected?

Based on these dimensions, tasks can be divided into three tiers:

Tier	Characteristics	Typical difference manifestation
L1 Deterministic tasks	Single file, clear requirements, existing tests	Models at different tiers are basically equivalent; prioritize unit price and latency
L2 Local reasoning tasks	Need to read a small amount of code and handle edge cases	Higher-capability models require fewer rework rounds and have higher first-pass success rates
L3 Long-chain exploration tasks	Cross-module refactoring, requirements need inference, proactive risk discovery	Differences are amplified: whether the model can point out “the pit on the third page of the drawing” appears at this tier

The key difference at L3 is not “writing more correctly,” but proactiveness and global consistency. Higher-capability models may complete your unfinished thought while also pointing out conflicts in adjacent design, missing migration paths, or ignored compatibility impacts. Lower-capability models tend to follow the literal instruction strictly: the task is completed, but risks remain.

The engineering implication is direct: the evaluation set must cover L3, otherwise conclusions are meaningless. Meanwhile, most cost-optimization opportunities in production lie in L1.

3. The evaluation environment must be fixed; otherwise conclusions are not comparable

Many model comparisons contradict each other because environment variables are not locked. A reproducible evaluation should at least fix the following:

1. Network policy: Disable the network throughout. Otherwise, the same external link may return different content at different times, directly contaminating results.
2. Tool set and permissions: The allowed scope for file reading and writing, command execution whitelist, and whether parallel calls are allowed must be consistent.
3. Context budget: The per-session context limit, history compression strategy, and number of retrieved snippets.
4. Termination conditions: Maximum number of turns, maximum number of tool calls, and timeout thresholds.
5. Source of runtime information: Model identifiers, context window, knowledge cutoff, and similar information should be read and recorded from runtime metadata, not asked of the model itself.

The last point is especially easy to overlook. A model’s statement such as “My knowledge cutoff is some month in some year” or “My context length is X” is not reliable. That is merely text generated from the system prompt and training corpus, not an observation of internal state. Any factual field entering a report should come from auditable runtime logs.

Task definitions should be structured for batch execution and horizontal comparison:

task_id: refactor-cache-layer-03
tier: L3                    # Determined jointly by ambiguity and chain length
network: disabled
max_turns: 24
context_budget: 128k
entry_checks:               # Objective acceptance items, machine-verifiable
  - unit_tests_pass
  - no_public_api_break
  - migration_path_documented
  - diff_under_400_lines
scoring:
  count_first_pass: true    # Whether to count first-pass success
  record_review_minutes: true

4. Metric design: turn “capability” into computable quantities

A single pass rate hides the cost structure. At least six metrics should be collected:

• First-pass success rate: The proportion of cases that pass all acceptance checks without human modification. This is the most direct expression of capability.
• Average rework rounds: The number of human-machine interaction rounds needed to reach acceptance. It reflects instruction understanding and self-correction ability.
• Human review time: The number of human review minutes from submission to merge. It is the key coefficient for converting “capability” into “money.”
• Context and output consumption: The number of input, output, and cache-hit tokens, as well as the proportion of invalid retries.
• End-to-end latency: On long-chain tasks, latency differences often affect experience more than unit price differences.
• Number of risk events: The number of occurrences of behaviors such as unauthorized access, modifying unrelated files without permission, or introducing undeclared dependencies.

The first two metrics measure capability. The latter four measure cost. They must be reported in pairs. A comparison that reports only capability is marketing. A comparison that reports only unit price is misleading.

5. The correct cost-effectiveness calculation: cost per accepted output

“A more expensive model has worse cost-effectiveness” is a common misjudgment because it records rework cost as zero. A more reasonable measure is the total cost consumed per accepted output:

def run_cost(r):
    return (r.input_tokens     * P_IN
            + r.output_tokens    * P_OUT
            + r.retry_tokens     * P_OUT      # Failed retries also cost money
            + r.review_minutes   * RATE_MIN)  # Opportunity cost of human review
def cost_per_accepted_patch(runs):
    accepted = [r for r in runs if r.accepted]
    if not accepted:
        return float("inf")                  # All failures: infinite cost
    return sum(run_cost(r) for r in accepted) / len(accepted)

This formula explains a counterintuitive phenomenon. On L3 tasks, a model with a lower unit price may have a higher cost per accepted output if its first-pass success rate is only half and human review time doubles. On L1 tasks, the extra capability of a higher-tier model has nowhere to be used, so its price disadvantage cannot be amortized, and the conclusion is reversed.

Further cost optimization does not depend on “choosing the cheapest model.” It depends on tiered routing:

def route(task):
    if task.ambiguity <= 1 and task.files_touched <= 2:
        return "standard_tier"        # L1: cost-first, can be fully automated
    if task.ambiguity >= 3 or task.files_touched >= 8:
        return "high_tier"            # L3: capability-first, allow longer chains
    return "standard_tier_then_escalate"  # L2: try the lower tier first, escalate automatically on failure

Related engineering practices also include: cache reuse for prompts and system templates to reduce duplicate input billing; context trimming and summarization for long sessions; batch scheduling for parallelizable tasks; and assigning stable deterministic steps to traditional programs rather than models. If something can be solved with regular expressions and static analysis, it does not need to consume tokens.

6. Observability: make evaluation conclusions defensible

The value of evaluation depends on whether conclusions can be audited and reproduced. In practice, observability across the entire chain should be completed:

• Call-chain tracing: Record the order and latency of each model request, tool call, and context trimming step. This helps locate whether failure occurred during “understanding requirements” or “executing operations.”
• Centralized logging: Structurally record input summaries, output summaries, exit reasons, and error codes to support aggregation analysis by task tier.
• Evaluation artifact archiving: Store complete conversations, diff patches, and test outputs in object storage, organized by task identifier and model version, supporting replay after the fact.
• Dashboards and alerts: Turn first-pass success rate, rework rounds, and unit cost into trend metrics and connect them to monitoring alerts. When a version upgrade causes a significant regression in any metric, it should be detected promptly.

On cloud platforms such as Baidu AI Cloud, these capabilities usually correspond to general-purpose product categories: logging services, monitoring and alerting, object storage, and container platforms. They can be combined directly into evaluation infrastructure without building it from scratch. It is worth emphasizing that observability construction is a one-time investment with long-term reuse. Without it, every model selection starts from scratch based on intuition.

7. About letting models evaluate themselves

A common practice in the community is to let a new model describe its differences from the previous generation, or to make a model imitate someone’s writing style and generate comparative answers. This kind of content reads smoothly, but its evidence level is low for three reasons:

1. It restates corpus, not facts. A model’s description of “its own capability” comes from expressions in training data and system prompts. It does not have introspective ability and cannot observe its parameters, routing strategy, or true thresholds.
2. Style imitation is not style mastery. Word choice and tone are easy to align, but sentence rhythm, discourse structure, and argumentation habits are difficult to cover with simple prompts. Evaluators may mistake “resembles” for “is.”
3. Player-as-referee structural bias. When the same model is both the subject under test and the judge, the conclusion is naturally biased toward itself.

The alternative is to turn self-evaluation into verifiable external evidence: the same batch of tasks, the same acceptance scripts, blind review, and cross-comparison. It is acceptable for models to help generate test cases, review patches, or summarize failure reasons, but the final judgment must be based on machine-executable checks and human spot checks.

8. Safety boundaries are also part of capability

Models differ in the strength of additional restrictions in sensitive areas such as biology and cybersecurity. This directly shows up as differences in task refusal rates. In scenarios with high security requirements, this is not a defect but a design choice. However, in lawful and compliant internal R&D scenarios, such as penetration testing preparation for one’s own systems or compliant bioinformatics data processing, excessive refusal also constitutes a loss of availability.

Therefore, the model-selection checklist should explicitly include a safety dimension:

• Does the refusal boundary in sensitive areas match your business scenario?
• Does it support fine-grained permission control? Can it limit the scope of repositories that can be read or written?
• Does it provide complete operation auditing? Can it meet internal compliance requirements?
• Is the data processing path clear? Does it support deployment inside your own network environment?

These questions are often more important than differences in benchmark scores, yet they are most often absent from design documents.

9. An actionable model-selection workflow

1. Collect real tasks: Extract 30 to 60 recent real tasks from your own codebase, tier them into L1/L2/L3, and make the ratio close to the actual work distribution.
2. Fix acceptance criteria: For each task, write down machine-verifiable checks. Avoid subjective descriptions such as “looks right.”
3. Lock environment variables: Fix and record the network, tool permissions, context budget, and termination conditions.
4. Run blind tests: Hide model identifiers. Run the same task multiple times to observe variance rather than drawing conclusions from a single run.
5. Report metrics in pairs: Provide capability and cost metrics together, and convert them into cost per accepted output.
6. Build routing and fallback: Send low-tier tasks to the low-cost tier, high-tier tasks to the high-capability tier, and set thresholds for automatic escalation and human intervention.
7. Maintain continuous regression: Incorporate the evaluation set into a regular pipeline, rerun it automatically after version updates, and judge benefit by trend rather than a single point.

10. Conclusion

The relationship between new coding models and previous generations is more like steps within the same tier than a cliff between “cannot” and “can.” On low-ambiguity, short-chain daily tasks, differences are close to zero. The gap concentrates on long-chain tasks that require requirement inference, cross-module coordination, and proactive risk discovery, and it appears as fewer rework rounds and higher first-pass success rates.

Therefore, the correct decision-making approach is: evaluate with your own business data by tier, include rework and human review in the cost calculation, and route tasks according to difficulty. Do not trust official positioning blindly. Do not rely on model self-reports or self-evaluation. And do not use one cheap model to fight every level of difficulty: using an oversized tool for a trivial task does not make the task better; but using a kitchen knife to take apart an engine will not even pay for the knife you saved.