How to Evaluate LLM Coding Ability: A Four-Scenario Testing Methodology

    Public coding benchmarks correlate poorly with real engineering outcomes. This article explains why, then shows how to build contamination-resistant evaluation sets from your own team's tickets rather than public problem banks. It covers four test scenarios—large-codebase understanding, cross-file refactoring, defect localization, and requirement-to-implementation—along with what to watch in each. You'll also get a two-stage scoring design (hard gates plus weighted dimensions), process metrics like first-pass rate and rework rounds, a reproducible execution skeleton, and the common pitfalls to avoid. The core idea: measure fit, not ranking.

    1. The Bottom Line First: Benchmarks Are an Entry Ticket, Not a Passport

    One repeatedly confirmed observation is that coding scores on public leaderboards correlate far less strongly with “how many rounds of rework it takes before the change can be merged” than most people assume. There are three reasons:

    • Data contamination and overfitting. Public problems and their solutions have circulated in training corpora for a long time. The model’s familiarity with problem patterns gets counted as ability, rather than genuine generalization.
    • Distorted problem shapes. Leaderboard problems are typically single-file, function-level, closed problems with clear inputs and outputs. Real engineering work is open-ended: it spans modules and languages and carries legacy baggage and implicit conventions.
    • A single scoring criterion. Most leaderboards only check “do the tests pass,” whereas an engineering view also cares whether the change is minimal, whether it fits the existing architecture and style, and whether it introduces new coupling or security risks.

    The more pragmatic approach is to keep a small set of benchmarks as a screen for the lower bound of capability, and invest most of your effort in a task set built from your own codebase. Such tasks are naturally contamination-resistant and align directly with business value.

    2. How to Build the Eval Set: Sample from Tickets, Not from Problem Banks

    The source of a high-quality eval set is not a practice website—it is the team’s own historical workflow:

    1. Task sampling. Pick 30–60 items from closed defect tickets, feature tickets, and code review comments. Cover four intents—“read code,” “modify code,” “write new code,” “add tests”—and make sure each item has clear, automatically judgeable acceptance criteria (existing tests, newly added hidden cases, or build-artifact comparison).
    2. Snapshot freezing. Each task maps to a fixed codebase snapshot (an immutable, commit-level copy), so upstream changes don’t cause results to drift.
    3. Prompt tiering. Prepare two tiers of input for the same task: one gives only the task description (testing autonomous localization ability), the other attaches relevant files and dependency notes (testing implementation ability given context). The gap between the two tiers is itself an important metric—it reflects how heavily the model depends on repo retrieval and context engineering.
    4. Multiple sampling. Run each task 3–5 times and record the pass rate rather than a single result. Coding tasks are extremely sensitive to sampling randomness, and “one success” is often not reproducible.

    3. Four Real Scenarios and What to Watch

    Scenario 1: Understanding large codebases and analyzing source code

    Give a real project codebase that is structurally complex and sparsely commented, and ask questions like “why is this abstraction layered this way,” “what are the boundary conditions of this cache invalidation strategy,” and “if the underlying communication implementation were replaced, where is the blast radius.”

    Differentiation shows up at two levels. Weaker performance is semantic-level reading: it can accurately restate what functions do and how types relate. Stronger performance is intent-level understanding: it can reconstruct the design motivation and point out implicit constraints and known compromises. This difference directly affects the success rate of every later modification task—models that don’t understand intent tend to “patch in place” rather than “work with the existing abstraction.”

    Watch points: whether it can cite specific files and symbols as evidence rather than speaking in generalities; whether it can distinguish “the code is written this way” from “the code must be written this way.”

    Scenario 2: Cross-file refactoring and dependency changes

    Typical tasks look like “converge the direct config reads scattered across multiple places into a unified config layer” or “introduce an interface abstraction for a shared capability while keeping the old implementation compatible.”

    Watch points include: whether the number of changed files stays within a reasonable range; whether all callers are updated in sync; whether backward compatibility is preserved; whether the build and existing tests pass on the first try; and whether the generated code style (naming, error handling, logging conventions) matches the repository. The problem most easily exposed here is local optima: changing only the few files in view while missing indirect callers, config files, scripts, or documentation.

    Scenario 3: Defect localization and fixing

    Pick cases from real defects where the root cause is well hidden: concurrency races, boundary conditions, state inconsistency after a swallowed exception, mismatched cross-language call conventions. Give only the symptom and the reproduction steps—never the root cause.

    Watch points: whether it can propose verifiable hypotheses and converge on them, rather than changing several places at once and “hoping to get lucky”; whether the fix actually hits the root cause; whether a regression test was added; and whether it slipped in unrelated refactoring along the way (such “noise changes” significantly raise review cost).

    Scenario 4: From requirement to implementation, including writing tests

    Give a complete requirement description that contains implicit non-functional requirements (performance, compatibility, observability).

    Watch points: whether requirement clarification is proactive (can it spot ambiguity in the description and ask); whether the implementation runs; whether unit tests and the necessary logging and monitoring instrumentation are produced alongside; and whether the change description is clear enough to review.

    4. Scoring: Turning “Good or Bad” into Comparable Numbers

    A two-stage design of “hard gates + weighted scoring” is recommended. If the hard gates fail, the task is judged negative immediately and never enters the weighted stage.

    1. {
    2. "hard_gates": ["Build passes", "All existing tests green", "No new high-severity static-analysis warnings"],
    3. "dimensions": {
    4. "Functional correctness": 0.35,
    5. "Minimality of changes": 0.15,
    6. "Style and architecture consistency": 0.15,
    7. "Test completeness": 0.15,
    8. "Instruction following": 0.10,
    9. "Readability and change description": 0.10
    10. },
    11. "process_metrics": ["First-pass rate", "Average rework rounds", "Manual review time", "End-to-end latency and call cost"]
    12. }

    Process metrics are often more valuable for decision-making than the result score. First-pass rate determines whether the model can enter the daily pipeline; average rework rounds determines whether human efficiency is actually improved or merely offset; and cost and latency determine which kinds of tasks the model is suited for.

    5. A Reproducible Eval Execution Skeleton

    The key to eval engineering is isolation and traceability. Each task runs in its own container with pinned dependency versions, and the model’s output patches and execution traces are fully retained for later attribution.

    1. from dataclasses import dataclass
    2. import json, subprocess, pathlib
    3. @dataclass
    4. class Case:
    5. id: str
    6. prompt: str # Task description (desensitized real ticket)
    7. repo_snapshot: str # Immutable codebase snapshot URL
    8. hidden_tests: list # Acceptance cases not included in the prompt
    9. rubric_path: str # Scoring weight config
    10. def run_case(case: Case, model_client, workdir: pathlib.Path) -> dict:
    11. workdir.mkdir(parents=True, exist_ok=True)
    12. subprocess.run(["git", "clone", "--depth", "1", case.repo_snapshot,
    13. str(workdir)], check=True)
    14. patch = model_client.solve(case.prompt, context=workdir) # Generate the change
    15. diff_file = workdir / "model.patch"
    16. diff_file.write_text(patch, encoding="utf-8")
    17. build = subprocess.run(["make", "build"], cwd=workdir,
    18. capture_output=True, text=True)
    19. if build.returncode != 0:
    20. return {"case": case.id, "verdict": "fail", "stage": "build"}
    21. results = []
    22. for t in case.hidden_tests:
    23. r = subprocess.run(["pytest", "-q", t], cwd=workdir,
    24. capture_output=True, text=True)
    25. results.append({"test": t, "passed": r.returncode == 0})
    26. return {
    27. "case": case.id,
    28. "verdict": "pass" if all(x["passed"] for x in results) else "fail",
    29. "hidden_test_pass_rate": sum(x["passed"] for x in results) / len(results),
    30. "changed_files": len(subprocess.run(
    31. ["git", "diff", "--name-only"], cwd=workdir,
    32. capture_output=True, text=True).stdout.strip().splitlines()),
    33. "artifacts": str(diff_file),
    34. }

    Combined with batch execution and result aggregation, this gives you the pass-rate distribution for each model across each scenario—not just a single mean.

    6. Eval Pitfalls You Must Avoid

    • No blind testing. When reviewers know which model produced a result, scores shift systematically. Task order should be randomized and outputs anonymized.
    • Comparing only means, not distributions. Two models with the same mean can differ wildly in stability. High variance means the model is hard to bring into an automated pipeline.
    • Ignoring the context budget. For the same task, results under different retrieval strategies and context lengths can differ by more than the difference between models. Fix the context-building strategy during evaluation, and separately assess the payoff of the strategy itself.
    • Ignoring security and compliance. The eval set must be desensitized before it is stored—remove keys, connection strings, and customer data—and any third-party dependencies introduced by generated code should go through license and provenance review.
    • One-time evaluation. Model capabilities keep iterating, so the eval set should be maintained long-term as a regression asset, with new task types added periodically to keep it from going stale.

    7. How a Team Rolls This Out

    The recommendation is three steps. First, build a minimal eval loop with 10–15 high-certainty tasks to verify that the scoring criteria are stable. Next, expand to 50+ tasks and wire it into continuous integration, so every model or prompt change produces regression data. Finally, route by task type: put models with a “high first-pass rate and convergent changes” on bulk refactoring and test-completion work, put models with “strong intent understanding” on solution design and code review, and keep changes to high-risk core paths human-led.

    At the infrastructure level, this eval setup needs no complex components. The task set and snapshots can live in general object storage or a code hosting system; eval tasks run on a container platform to guarantee environment isolation; execution logs and call traces are retained by a logging service; and latency and failure rates are observed through monitoring and alerting. Baidu AI Cloud can provide hosting options for these general capabilities (model services, a container platform, object storage, and logging and monitoring), with the specific selection and quota to be evaluated against actual business needs.

    8. Summary

    The core of evaluating a model’s coding ability is not finding the “strongest model,” but answering three engineering questions: what is its first-pass rate on my codebase, how many rounds of rework does it take, and are the per-task cost and latency acceptable. Moving evaluation from public leaderboards to your own ticket snapshots, constraining quality with hard gates plus weighted scoring, characterizing stability with multiple sampling, and measuring human efficiency with process metrics—this approach stays effective across any model iteration, because it measures fit, not ranking.