LLM Coding Capability Evaluation and Cost-Effectiveness: A Practical Guide

1. Why “How good is it?” Is Hard to Answer

When a new large language model is released, two questions appear most often in community discussions: “How good is it?” and “Is it worth the price?” Both are real questions, but both lack verifiable criteria.

The difficulty with “How good is it?” is that most conclusions come from a few ad hoc conversations. Such experiences depend heavily on how the model is prompted, the quality of the prompts, whether tools are enabled, how much code is placed in the context, and even the server-side rate-limiting state on the day of testing. The performance gap caused by different host environments can be larger than the gap between different models.

The difficulty with “Is it worth the price?” is that cost is measured inconsistently. Some people only look at the price per token, some look at the monthly bill, and some also convert rework and human review time into cost. These three approaches can lead to completely opposite conclusions.

To solve these problems, model evaluation must be upgraded from “experience sharing” to “reproducible experimentation.” The following sections cover experiment design, test-set construction, scoring, cost accounting, and engineering implementation.

2. Designing Reproducible Evaluation Environments

A credible evaluation must be re-runnable by others. This means the record should include not only the model answers but also the testing conditions themselves.

2.1 Host Form and Capability Boundaries

Model performance in real products is jointly determined by “model capability + host capability.” At minimum, the host side should record four things:

• Runtime form: Is it a conversational API, or a coding assistant with file read/write, command execution, and code retrieval capabilities?
• Available toolset: Can it modify files, execute tests, or search the codebase? This directly determines the task ceiling.
• Context strategy: How large is the single-turn context window? Is history automatically compressed? How is overflow handled?
• System prompt: Who injects the role definition, output format constraints, and safety policies?

The performance of the same model in a “plain chat” setting and in a “coding assistant with a toolchain” is usually not directly comparable. Therefore, an evaluation report must first fix the host environment before discussing the model.

2.2 Freezing and Isolating Test Conditions

The recommended “test environment” record includes the following:

Condition	What to record	Purpose
Model identifier	Version number or model ID	Prerequisite for reproduction
Host form	Coding tool / chat API / SDK	Defines the capability ceiling
Time range	Start and end timestamps, including timezone	Helps investigate server-side changes
Network status	No network / limited network	Eliminates external information interference
Per-task time limit	For example, 30 minutes	Observes degradation behavior under pressure
Actual elapsed time	Wall-clock time per task	More informative than merely recording timeouts
Temperature and sampling	Parameters such as temperature and top_p	Affect result variance

Two easily overlooked details are especially important.

First, network isolation must be complete. If a task contains an external link and the model accesses it on its own, the task measures retrieval ability rather than reasoning ability. Mixing the two makes attribution impossible. Under a fully offline condition, the model should clearly say “I don’t know” or “I cannot confirm” when information is insufficient. This itself is an important evaluation point.

Second, actual elapsed time is more valuable than a simple timeout indicator. In an experiment with six tasks, each limited to 30 minutes, the total elapsed time may be only around 40 minutes. This shows that most tasks finish within 20% to 40% of the time limit. In this case, the binary “no timeout” signal has almost no discriminative power. The more informative signal is the elapsed-time distribution: which types of tasks are completed quickly and stably, and which tasks involve repeated probing, frequent file modifications, or being stuck on the same error for a long time. The latter is often the most sensitive probe for capability weaknesses.

2.3 Full Traceability

For each task, it is recommended to retain:

• The original prompt
• The complete model output
• The tool-call sequence
• The diff of generated files
• Execution logs
• The verdict and the reasoning behind the verdict

An evaluation without traceability cannot be reviewed later, nor can it support longitudinal comparison after model iterations.

3. Layered Test Set: From Baseline Probes to Complex Engineering

An effective test set should be layered by capability dimension, not simply arranged by increasing difficulty. Six layers are recommended.

3.1 Baseline Probes: Identity, Knowledge Boundaries, and Recency Sparsity

The most basic type of question asks: “What model are you? What version? What is your knowledge cutoff?” The value of this type of question lies not in the answer itself but in verifying three engineering facts:

• A model’s description of its own identity and version usually comes from the system prompt injected by the runtime environment, not from true self-inspection. Treating it as a factual source is unreliable. The real version information should be obtained from the model identifier returned by the service.
• A knowledge cutoff date is only a single point in time. Actual usability is a decay curve. Coverage becomes sparser near the cutoff date and denser toward earlier periods. Therefore, recency probe questions should select dates near the cutoff or recent events, rather than randomly choosing a fact from two years ago.
• Under a fully offline condition, the model should refuse to answer or declare uncertainty when facing time-sensitive questions, rather than fabricating an answer. This directly tests the tendency toward hallucination.

3.2 Constraint-Following Tasks

Provide a set of explicit format and behavior constraints and observe whether they are followed. Examples include:

• “Output JSON only, without any explanatory text.”
• “Do not modify existing function signatures.”
• “If an interface is uncertain, stop and ask first.”

This type of task measures instruction-following ability. It is highly weighted in engineering usability but often underestimated. When scoring, hard constraints (violation means failure) should be distinguished from soft constraints (partial credit may be given).

3.3 Code Generation and Self-Repair Tasks

Provide a piece of defective code or a vague requirement, and ask the model to implement it and self-repair after failure. Observation points include:

• The pass rate of the first implementation
• The number of repair rounds after failure
• Whether the model falls into oscillation, such as fixing one place while breaking another
• Whether the model directly deletes or modifies assertions just to make tests pass

The last point is especially important. “Fixing a bug” by deleting tests is a typical high-risk behavior and must be penalized separately.

3.4 Code Understanding and Refactoring Tasks

Provide several hundred lines of legacy code and ask the model to explain the key paths, point out potential defects, and refactor the code without changing external behavior. This task tests long-range understanding and engineering judgment, which are harder to fake than code generation. Scoring should focus on:

• Whether boundary conditions are correctly identified
• Whether real defects are discovered
• Whether behavior remains equivalent after refactoring

3.5 Long-Context and Document Understanding Tasks

Embed several key constraints in an extremely long input, such as “Do not use a certain third-party library” or “Must be compatible with a certain version,” and check whether these constraints are preserved in the final answer. Information loss in the middle sections is common. Position-dependent failure modes should be recorded separately.

3.6 End-to-End Engineering Tasks

Finally, include an end-to-end task: implement a minimal viable service from scratch, including interface definitions, input validation, unit tests, and dependency descriptions. This task assesses the ability to break requirements into executable steps, manage multi-file changes, and locate root causes when errors occur. It is also the closest to real work scenarios.

4. Scoring System: Metrics and Rules

Scoring rules should be fixed before running the evaluation to avoid adjusting standards afterward. A minimal metric system is recommended:

Dimension	Metric	How to judge
Correctness	Test pass rate	Automated execution
Stability	Whether tests still pass after multiple repair rounds	Automated execution
Constraint following	Number of hard-constraint violations	Rule checking
Safety	Whether tests are deleted or dangerous operations are executed	Manual review
Efficiency	Wall-clock time and interaction rounds	Automatically recorded
Resource usage	Input/output tokens and cache hits	Automatically recorded
Honesty	Whether uncertainty is declared when uncertain	Manual judgment

When judging results, hard constraints and safety issues should be treated as automatic disqualifiers to prevent high scores from masking fatal defects. An example scoring logic is as follows:

def score(result):
    if result.violated_hard_constraints:
        return {"pass": False, "reason": "Hard constraint violated"}
    if result.deleted_tests or result.ran_dangerous_ops:
        return {"pass": False, "reason": "Safety disqualification"}
    quality = (
        0.45 * result.test_pass_rate
        + 0.15 * result.constraint_score
        + 0.15 * result.honesty_score
        + 0.15 * result.refactor_quality
        + 0.10 * max(0.0, 1 - result.elapsed / BUDGET)
    )
    return {"pass": True, "quality": round(quality, 4)}

A key principle is that each model and each task should be repeated at least three times, with the median and dispersion reported. The result of a single run has almost no statistical significance because model outputs show clear variance under default sampling settings.

5. Cost-Effectiveness Assessment: Accounting Under a Unified Definition

5.1 Nominal Cost Is Not Effective Cost

The price per token only covers inference costs. The real cost per effective unit of output should include:

Effective cost = (inference cost + tool and retrieval cost + human review cost + rework cost) / number of passed tasks
Unit effective output cost = effective cost / (number of passed tasks × task complexity weight)

The most easily missed item is human review cost. If a cheap model only has a 60% pass rate, the remaining 40% may require engineers to locate issues, rewrite code, and verify results. The converted labor cost may exceed the inference cost saved by using the cheaper model.

5.2 Quality-Cost Matrix

Dividing tasks into four quadrants by “complexity” and “cost per call” can produce clear selection conclusions:

• Low complexity and low cost tasks, such as comments, naming, formatting, and unit-test completion: prefer lightweight models, especially for high-volume and latency-sensitive scenarios.
• Low complexity and high cost tasks: replace them with lightweight models where possible. This has the largest room for cost-effectiveness improvement.
• High complexity and low cost tasks: prioritize these. They are the sweet spot for cost-effectiveness.
• High complexity and high cost tasks, such as cross-module refactoring and complex defect localization: use strong models or human-led workflows. The cost may be acceptable, but timeout and budget limits should be set.

6. Engineering Implementation: Tiered Routing and Evaluation Loops

6.1 Tiered Routing

It is not recommended to route all tasks to the same model. Pre-routing by task complexity often brings more benefit than simply switching models:

Task enters
  └─ Complexity classification (rules / lightweight model judgment)
       ├─ Simple: formatting, comments, single-function generation --> lightweight model
       ├─ Medium: common bug fixes, local refactoring --> medium model
       └─ Complex: cross-module design, hard defects --> strong model + budget cap

6.2 Cost and Context Optimization

• Place stable, unchanging system prompts and long documents at the beginning of the context. Use server-side prompt caching to reduce duplicate billing and time to first token.
• For code retrieval, first recall summaries and then expand to full text only when needed. Avoid placing an entire repository into the context.
• For multi-file tasks, execute them step by step according to dependency order. Perform local verification at each step to reduce ineffective retries.
• For batch-oriented tasks, such as bulk comments or bulk unit tests, queue them offline. Trade throughput for cost.

6.3 Making Evaluation Pipelines Continuous

Evaluation should not be a one-time activity when a new model is released. It should be integrated into continuous integration as part of regression testing. A practical form can include:

• A fixed test set
• Automatic judgment
• A score and cost report generated for each run
• Alerts triggered when scores drop

On the cloud platform side, such a pipeline is usually assembled from several general capabilities: model inference services for unified calls and quota management, object storage for saving test sets and artifacts, message queues for handling peak shaving of batch tasks, and logging, monitoring, and alerting services for traceability and threshold alerts. Baidu AI Cloud and other public cloud platforms provide such general capability categories, but specific model versions, interfaces, and billing policies should be based on the official documentation of the corresponding services and actual API responses, rather than inferred from evaluation experience.

7. Common Pitfalls and Notes

1. Test-set contamination. Using original questions from public benchmarks can inflate results. A certain proportion of proprietary questions should be retained and rotated regularly.

2. Insufficient sample size. Running each task only once, or only presenting the best run, significantly amplifies bias. Use at least three runs and report the median.

3. Treating model self-reports as facts. Identity, version, and knowledge cutoff information should be obtained from the runtime environment or API metadata.

4. Confusing cost definitions. When comparing models, human review and rework must be converted into the same formula.

5. Ignoring server-side variability. Rate limiting, queuing, and gradual rollouts can change latency and success rates over short periods. Data spanning a long time range should be grouped by time period.

6. Ignoring safety boundaries. Before allowing a model to execute commands or modify files during evaluation, set up sandboxes and approval points to prevent the experiment itself from causing damage.

7. Looking only at averages and ignoring distributions. The difference between P50 and P95 latency often determines whether a product is usable under real concurrency.

8. Summary

The core of evaluating LLM coding capability is to turn subjective impressions into reproducible experiments: fix the host environment and test conditions, build a layered test set, define scoring rules in advance, and calculate effective cost under a unified definition. On this basis, cost-effectiveness is not a question of “expensive or not,” but a question of “unit effective output cost,” which includes inference cost, rework, human review time, and weight differences caused by task complexity.

Engineering benefits come from three places: using complexity-based routing to match models with tasks, using context and cache strategies to reduce invalid calls, and making evaluation pipelines continuous regression tests to maintain a quality baseline. With these three practices, teams no longer need to start from scratch and guess every time a new model is released. Instead, they can provide definite answers using their own test sets and cost definitions.