LLM Coding Evaluation: How to Measure Real Capability and Cost-Effectiveness

When a new model is released, developers usually care about two things: how strong its real capability is, and whether it is cost-effective. Both depend on whether the evaluation method is scientific.

Looking only at public leaderboards is easily biased by training-data contamination and overfitting. Relying only on personal trials lacks reproducible, horizontally comparable data. This article focuses on the engineering evaluation of code LLMs. It combines common practices such as multiple synthetic tasks, offline execution, and complete logging of elapsed time and token usage to break down a practical evaluation methodology.

1. Why Public Leaderboard Scores Should Not Be Used Directly for Model Selection

• Training-data contamination: Public task sets may have entered training corpora after long circulation. The model may show “memorization” rather than “reasoning”.
• Overreliance on a single metric: “Compilation passed” does not mean “logic is correct”. When unit-test coverage is incomplete, the code may run while boundary conditions and exception branches remain defective.
• Host effect: The same model can perform very differently in different agent frameworks. Retrieval accuracy, multi-turn repair strategy, and other framework-level factors can significantly affect results.
• Subjective bias: Conclusions based on impression are hard to reproduce and cannot be compared horizontally.

A reliable evaluation must satisfy three conditions:

1. Tasks are close to real workloads.
2. The process is reproducible, with objective judgment signals.
3. Quality, cost, and latency are measured under the same criteria.

2. Benchmark Task Design: Layered Capabilities with Automatic Signals

It is recommended to organize the evaluation set as a layered matrix. Each layer tests only one capability and uses automatically collectable signals to judge results, avoiding subjective “looks good” scoring.

Layer	Capability	Task Format	Judgment Signal
L1	Syntax and API correctness	Single-file completion; error fixing	Compilation passes; unit tests pass
L2	Cross-file refactoring	Modify multiple modules according to interface contracts	Regression tests, static checks, interface compatibility validation
L3	Understanding implicit requirements	Requirements are intentionally incomplete	Whether the model proactively asks questions or makes reasonable assumptions, and whether the patch and tests are consistent
L4	Long-horizon autonomous repair	Multi-round localization and repair based on failure logs	End-to-end success without introducing new failures
L5	Environment and tool awareness	Requires reading directory structure and dependency manifests before making decisions	Whether conclusions match repository facts and tools are invoked correctly

Each task should declare the acceptable patch scope, evaluation script, and network access permissions in advance.

Example task specification:

task_id: L4-014
capability: multi_turn_ci_repair
repo_snapshot: snapshot://baseline/2026-05-01
network: disabled          # Disable external network and evaluate behavior without external information
tools:
  - read_file
  - write_file
  - run_tests              # Allowed only inside an isolated sandbox
budget:
  max_turns: 30
  max_wall_clock_sec: 1800
  max_tokens: 2000000
acceptance:
  - ci_status == green
  - no_new_static_violation
  - diff_reviewer_score >= 4   # Average score from two blind reviewers; scale 1-5

After generating the case, perform a “seed check”:

1. Manually confirm that the task must fail before the fix.
2. Run it with another model of known capability to confirm that the task can be solved by a qualified model and is not likely to be guessed by an overly weak model.

Only then can invalid tasks such as “all models pass” or “all models fail” be avoided.

3. Ensuring Reproducibility: Lock the Environment and Host

The same problem can yield different answers in different hosts. A common approach is to run the model as an agent in a code agent, with the network fully disabled and external links inaccessible. The host injects system prompts, provides tools, decides how context is organized and compressed, and drives the model to continue working across turns.

Therefore, an evaluation report must record:

• Model identifier and service version
• Host framework and agent SDK version
• System prompt version
• Sampling parameters such as temperature, maximum output length, and context budget
• Start and end time, total elapsed time, network permissions, and tool allowlist

Only when these variables are locked can horizontal comparisons be meaningful. To assess host effects, run controlled experiments:

• Host A/B comparison: Fix the model and compare pass rates and costs across hosts to identify the framework’s contribution.
• Model A/B comparison: Fix the host and replace the model to evaluate model capability more fairly.

4. The Right Way to Calculate Cost-Effectiveness: Cost per Qualified Delivery Unit

“Price per million tokens” is a common comparison metric, but it does not equal actual spending. Real cost is determined by four factors:

• Token usage by type: input, cache read, cache write, output, and reasoning tokens
• Average number of turns per task
• First-pass success rate
• Rework cost after failure

Actual cost should be converted to cost per qualified delivery unit:

C_task ≈ average token cost per turn × average turns ÷ first-pass rate + human rework cost

Example implementation:

# Prices are illustrative; use the service provider's official pricing in practice
PRICE = {"input": 0.15, "cache_read": 0.02,
         "cache_write": 0.35, "output": 0.60}  # CNY / million tokens
def cost_per_qualified_task(usages, first_pass_rate,
                            human_min_per_fix=20, human_rate_per_hour=300):
    """
    usages: token usage list across multiple turns for one task; each element looks like
            {"input": 12000, "cache_read": 88000,
             "cache_write": 0, "output": 1500}
    first_pass_rate: first-pass success rate, in (0, 1]
    """
    token_cost = sum(
        (u["input"] * PRICE["input"]
         + u.get("cache_read", 0) * PRICE["cache_read"]
         + u.get("cache_write", 0) * PRICE["cache_write"]
         + u["output"] * PRICE["output"]) / 1e6
        for u in usages
    )
    rounds = max(len(usages), 1)
    expected_attempts = 1 / first_pass_rate
    rework = (expected_attempts - 1) * human_min_per_fix / 60 * human_rate_per_hour
    return {
        "token_cost": token_cost * expected_attempts,
        "rework_cost": rework,
        "total": token_cost * expected_attempts + rework,
        "avg_rounds": rounds,
    }

From this, three optimization lessons can be derived:

• Improve cache hit rate: Place stable content, such as system prompts and repository descriptions, near the beginning of the context to improve cache reuse. If cache-read pricing is significantly lower than normal input pricing, this is often the most effective lever for reducing token cost.
• Reduce invalid turns: Time and token consumption in multi-turn tasks usually grow linearly or even faster with the number of turns. Improving how tool output is organized is more effective than simply reducing unit price.
• Correct intuition with first-pass rate: Raising the first-pass rate from 50% to 75% reduces expected attempts from 2 to about 1.33, lowering cost by about one-third. Such quality gains can outweigh small price differences.

Latency must also be included. In one example, six synthetic tasks run in a code agent took about 43 minutes in total, or about 7 minutes per task on average, showing that elapsed time for long-horizon tasks must be included in capacity planning. With the following example settings, the throughput of the same batch can differ by nearly three times:

• With concurrency 10, about 85 tasks can be completed per hour.
• With concurrency 30, about 257 tasks can be completed.

5. Understanding Model Self-Reported Information and Knowledge Cutoff

In such evaluations, it is common to first ask the model to state its version and knowledge scope, then ask about recent events. The answer usually contains two parts:

• Identity and cutoff date: Comes from the system prompt injected by the runtime.
• Specific event details: Depends on parametric memory.

Two points must be noted:

1. It reflects information provided by the runtime, not reliable introspection. The model cannot truly “perceive” its weights or training details.
2. Parametric memory has a clear temporal gradient. Events long before the cutoff are usually remembered more strongly; memory near the cutoff is sparse, and details may be confused or hallucinated as the cutoff is approached.

Common mistakes include:

• Using whether a model can accurately recall news around the cutoff to judge model strength.
• Using a model’s self-reported time as evidence of data freshness.
• Directly trusting unverified time-sensitive information in business systems.

Correct practices include:

• Rely on tool results: For time-sensitive content, use retrieval augmentation or tool calls and retain verifiable sources.
• Use internal benchmarks: Build freshness benchmarks from recent internal changes; this is more aligned with business needs than tracking public news.
• Mark low confidence: Require the model to explicitly flag low-confidence content.

6. From Evaluation to Online Use: Turn the Method into a Pipeline

Evaluation becomes valuable only when run continuously. The process can be implemented as follows:

1. Fix a baseline set: Select tens to hundreds of tasks covering L1 to L5, and add new tasks every 1-2 quarters to prevent tasks from being “gamed”.
2. Automate execution: Use a container platform to host isolated sandboxes, generating independent repository snapshots and test environments for each run.
3. Collect data: Archive run logs, token usage, latency, and judgment results in a logging service, and set monitoring/alerting thresholds for pass rate and cost.
4. Run regression comparisons: After each model or host upgrade, rerun the baseline set and output four core metrics: pass rate, average turns, cost per task, and latency.
5. Canary rollout: Use desensitized samples of real online tasks for shadow evaluation, then switch production traffic after stable results.

Data assets can be stored in object storage, and task dispatch can rely on a message queue. For example, on Baidu AI Cloud, general capabilities such as object storage, container platforms, logging services, and monitoring/alerting can be combined to build this evaluation pipeline.

7. Common Pitfalls

• Benchmark tasks overlap with training corpora, inflating scores.
• Evaluation only checks compilation results and ignores boundary conditions and exception branches.
• Multi-turn tasks do not track total elapsed time and total token usage, systematically underestimating cost.
• Differences between cache-read and cache-write pricing are ignored.
• The same prompts are used for both tuning and evaluation, causing overfitting.
• Graders know the model identity, introducing preference bias.

Conclusion

Evaluating the “actual capability and cost-effectiveness” of code LLMs is essentially about answering three questions under the same reproducible criteria:

1. Which levels of tasks can it complete stably?
2. How much cost and time does it take per qualified delivery unit?
3. Do differences come from the model itself or from the agent framework hosting it?

By quantifying these three questions, model selection can move from “feeling good” to “data-driven decisions.”