How to Evaluate LLM Coding Ability: A Reproducible Evaluation Method

    This article presents a reproducible approach to evaluating large language models' coding capabilities. It covers why unified evaluation matters, how to define evaluation dimensions (functional correctness, requirement adherence, code quality, bug-fixing, stability, cost and latency), how to design a realistic task set, control experimental variables, automate execution and scoring, and interpret statistical results. It also highlights common pitfalls and explains how to turn evaluation results into concrete team decisions.

    1. Why a Unified Evaluation Method Matters

    When choosing an AI coding assistant, many developers rely on demo videos or screenshots of a single conversation. These materials typically involve small sample sizes, inconsistent prompts, and results that are hard to reproduce. The same model can perform very differently under different prompts, so a single result cannot represent its overall capability.

    A reliable evaluation needs to answer three questions: Can the model complete tasks correctly? Are the results stable? And are the latency and cost within acceptable limits? The goal of evaluation is to support decisions for specific scenarios, not to produce a universal ranking.

    2. Define Evaluation Dimensions

    Before evaluating, clarify what you want to measure. Common dimensions include:

    1. Functional correctness: Does the generated code pass the predefined unit tests?
    2. Requirement adherence: Does it follow interface signatures, input/output formats, and boundary conditions?
    3. Code quality: Readability, complexity, and compliance with the team’s coding standards.
    4. Bug-fixing ability: Given buggy code and error messages, can the model locate the cause and produce a correct fix?
    5. Stability: Are results consistent when the same task is run multiple times?
    6. Cost and latency: Response time per task, input/output token counts, and the corresponding cost.

    Different teams weight these dimensions differently. A team focused on automated testing may care most about the first dimension, while a team maintaining a legacy project may prioritize the fourth.

    3. Design the Task Set

    The task set should come from real work scenarios rather than ad-hoc interview questions. It can be organized by category:

    Category Examples Suggested Count
    Algorithm implementation String processing, graph traversal, deduplication and sorting 30
    Business logic Order amount calculation, permission checks, state transitions 30
    Bug fixing Given broken code and error logs, fix it 20
    Test writing Add boundary test cases for existing functions 20

    Follow these principles when designing tasks:

    • Clear interfaces: Specify the function name, parameter types, and return value for each task to avoid failures caused by misinterpreting the interface.
    • Hidden tests: Unit tests are used only for scoring and should not be included in the prompt, preventing the model from superficially adapting to the tests.
    • Original tasks first: Prefer tasks written by your own team to reduce the chance that the questions already appear in public training corpora. This produces results closer to real-world capability.
    • Layered difficulty: Include a proportion of easy, medium, and hard tasks so you can observe how performance changes with difficulty.

    4. Control Experimental Variables

    Comparable results require keeping everything constant except the model. Unify the following:

    • System prompts and user prompt templates
    • Sampling parameters, such as fixing temperature at a low value to reduce randomness
    • Context content and length
    • Number of runs per task

    Manage these parameters in a configuration file and save them together with the results for later review:

    1. EVAL_CONFIG = {
    2. "temperature": 0.2,
    3. "max_output_tokens": 2048,
    4. "runs_per_task": 5,
    5. "prompt_template": "Implement the following function. Output code only, no explanation.\n{spec}",
    6. "timeout_seconds": 10,
    7. }

    Do not tune prompts for a specific model. If prompts are optimized for only one side, the comparison loses fairness.

    5. Automated Execution and Scoring

    Manually reviewing code one by one is inefficient, and judgment criteria tend to drift. A more reliable approach is to concatenate the generated code with the hidden tests, execute them with an automated script, and use the exit code to determine pass or fail.

    1. import subprocess
    2. import sys
    3. import tempfile
    4. from pathlib import Path
    5. def run_candidate(candidate_code: str, test_code: str, timeout_s: int = 10) -> dict:
    6. """Run the candidate code with tests in a temp directory and return the scoring result."""
    7. with tempfile.TemporaryDirectory() as tmp_dir:
    8. script = Path(tmp_dir) / "candidate_test.py"
    9. script.write_text(candidate_code + "\n\n" + test_code, encoding="utf-8")
    10. try:
    11. proc = subprocess.run(
    12. [sys.executable, str(script)],
    13. capture_output=True,
    14. text=True,
    15. timeout=timeout_s,
    16. cwd=tmp_dir,
    17. )
    18. except subprocess.TimeoutExpired:
    19. return {"passed": False, "reason": "timeout"}
    20. return {
    21. "passed": proc.returncode == 0,
    22. "reason": proc.stderr[-500:] if proc.returncode != 0 else "ok",
    23. }

    Pay special attention to security. Model-generated code is essentially unreviewed input, so it should be executed in an isolated environment such as a container or virtual machine, with restricted network access and no write permission to the host filesystem. The example above only demonstrates the scoring logic and must not be used directly in production.

    Failure reasons are also worth recording. Timeouts, syntax errors, assertion failures, and runtime exceptions each indicate different problems. Categorized statistics help the team determine whether a model’s weakness lies in understanding requirements or in implementation details.

    6. Statistics and Result Interpretation

    A single run of a task carries randomness. Run each task multiple times and compute the following metrics:

    • Single-run pass rate: The proportion of runs that pass the tests across all executions.
    • Task-level pass rate: The proportion of tasks that pass at least once across multiple runs, commonly known as pass@k, which represents the probability that the model succeeds within k attempts.
    • Standard deviation: Reflects variation across runs. A model with high variance is harder to predict in production.
    1. from statistics import mean, stdev
    2. def summarize(results_by_task: dict[str, list[bool]]) -> dict:
    3. """Aggregate results from multiple runs per task."""
    4. per_task_rates = [sum(runs) / len(runs) for runs in results_by_task.values()]
    5. overall = mean(per_task_rates)
    6. spread = stdev(per_task_rates) if len(per_task_rates) > 1 else 0.0
    7. any_pass = [any(runs) for runs in results_by_task.values()]
    8. return {
    9. "mean_pass_rate": round(overall, 4),
    10. "std_pass_rate": round(spread, 4),
    11. "tasks_with_any_pass": round(sum(any_pass) / len(any_pass), 4),
    12. }

    Interpret results together with the difficulty breakdown. A high pass rate on easy tasks does not mean the model can handle complex business logic. When the sample size is small, provide confidence intervals or state the margin of error explicitly, and avoid drawing conclusions from a difference of a few percentage points.

    7. Common Pitfalls

    • Only looking at the best run: Cherry-picking the best result overestimates the model’s capability. Use data from all runs.
    • Unfair prompts: Repeatedly tuning prompts for one model while others use the initial template makes the comparison unreliable.
    • Test leakage: Publicly available problems may already be in the training data, inflating scores and failing to reflect real capability.
    • Single-metric decisions: Choosing a model based only on overall pass rate ignores cost, latency, and compliance requirements.
    • Ignoring failure types: Counting only the number of failures hides whether the model systematically misunderstands requirements or occasionally makes small mistakes.

    8. Turning Results into Team Decisions

    Evaluation results should serve concrete scenarios. You can apply them as follows:

    1. First, break down requirements by scenario, such as test generation, code completion, or legacy code migration.
    2. For each scenario, review pass rate, stability, and cost separately.
    3. Determine the allowed scope of use based on data security requirements. For code involving sensitive business data, confirm the data processing policy and obtain approval before calling external services.
    4. Keep human code review. Passing automated tests does not mean the code can be merged directly.
    5. Re-test regularly after model version updates, since both capability and pricing may change.

    With a unified process, teams can turn the debate over “which model is better” into verifiable conclusions about “on which types of tasks, at what cost, and with what level of stability.”