Evaluating LLM Code Capabilities: A Practical Framework for Comparing Coding Models

1. Why “Which Model Is Best at Coding?” Is Hard to Answer Directly

The question “Which LLM is best at writing code?” is difficult to answer because “coding ability” is not a single scalar. The performance gap between models can be smaller than the gap between different task distributions.

• Task distribution matters: Writing a 30-line utility function, modifying one method in an existing project, and locating the root cause of a bug across 20 files require very different capabilities.
• Evaluation criteria can conflict: Passing tests, readable code, no obvious security issues, and a minimal diff are all desirable, but they often pull in different directions.
• Context conditions change results dramatically: Providing only a function signature is very different from providing the full repository structure, dependency versions, and coding conventions.
• Sampling introduces randomness: Pass rates across multiple generations of the same prompt can vary by more than ten percentage points, so a single test run has little statistical meaning.

A more rigorous question is therefore not “Which model is strongest?” but: Under my task distribution, tech stack, and acceptance criteria, which model offers the highest expected return?

2. Breaking “Coding Ability” into Measurable Dimensions

To make evaluation actionable, coding ability should be decomposed into dimensions that can be tested independently.

Dimension	Typical task form	Main metrics
Code generation	Convert natural language into functions or classes	Pass rate, compilation success rate
Code completion	Continue code from given context	Acceptance rate, retention after acceptance
Bug localization and repair	Given a failing test case and stack trace	Fix success rate, whether regressions are introduced
Refactoring and optimization	Improve code without changing behavior	Test consistency before and after, cyclomatic complexity change
Cross-file and repository-level understanding	Requires retrieving multiple files to answer correctly	File reference accuracy, reasonableness of change scope
Explanation and review	Explain logic and identify risks	Defect recall, factual consistency
Test generation	Add tests for existing code	Coverage improvement, assertion effectiveness

Importantly, pass rate is only an entry-level metric. Code that passes tests but reinvents wheels, silently swallows exceptions, or hardcodes secrets into source files can be a negative engineering asset.

3. The Reliability Boundaries of Common Evaluation Methods

3.1 Three Limitations of Public Benchmarks

1. Data contamination: Public problems and solutions are widely available online. A model may have seen similar answers during pretraining, which can inflate scores. One way to estimate this effect is to create equivalent variants by rewriting prompts, renaming variables, or replacing business semantics, then observe how much the score drops.
2. Unrealistic problem shapes: Public benchmarks are often self-contained single-function problems. They lack implicit constraints from real projects, such as dependency versions, internal utility functions, and coding standards. High benchmark scores and poor real-world usability are not contradictory.
3. Unreliable tests: Automatically generated validation tests can be too weak, checking only return values but not side effects. They can also be too strict, locking implementation details. Both cases introduce score noise.

3.2 Bias Sources in Human Review

Human scoring can be subjective. Evaluators may give higher scores simply because code “looks clean.” A practical mitigation is structured scorecards plus blind review:

• Hide model identity.
• Fix the review dimensions.
• Define observable criteria for each dimension.
• Measure agreement among reviewers.

If inter-rater agreement is low, the problem is usually an unclear scorecard rather than a meaningful model difference.

3.3 Pitfalls of Single-Session Chat Impressions

Chat-based impressions are highly sensitive to prompt quality. For example, changing “Write me a cache” into “Implement a local cache with TTL and capacity limits under the existing Cache interface, following the project’s logging and exception conventions” can produce dramatically different results.

Much of the observed experience difference comes from prompt engineering rather than model capability.

4. Building a Private Evaluation Set for Your Team

A private evaluation set is the only asset that can genuinely reflect business fit. Its cost is usually much lower than the cost of trial-and-error model selection.

Collection and desensitization: Source tasks from real development workflows, such as merged bug tickets, feature requests, and refactoring proposals. Remove or replace domains, secrets, internal identifiers, and customer information. Keep only the task description, relevant context, and acceptance criteria.

Task packaging: Represent each case as a declarative definition so it can be executed in batches and version-controlled.

# eval/cases/cache_ttl.yaml
id: cache-ttl-001
dimension: code_generation
language: python
context_files:            # Minimum sufficient context provided to the model
  - src/cache/base.py
  - src/logging_util.py
task: >
  Implement a TTL-based local cache class that inherits from base.Cache,
  supports a capacity limit and expiration cleanup, uses the project-wide
  logging utility to record eviction events, and rethrows exceptions
  instead of swallowing them.
acceptance:
  hidden_tests: tests/cache_ttl_hidden.py
  must_pass_ratio: 0.9
  checks:
    - no_hardcoded_secret
    - complexity_ceiling: 12
    - diff_files_max: 2

Scoring pipeline: Separate execution, scoring, and aggregation so that any stage can be replaced.

def evaluate(case, model_client, sandbox, k=5):
    results = []
    for _ in range(k):                      # Sample multiple times to reduce randomness
        patch = model_client.solve(case.task, case.context_files)
        report = sandbox.run(case, patch)   # Compile, run hidden tests, and apply static checks
        results.append({
            "passed": report.pass_ratio >= case.acceptance.must_pass_ratio,
            "diff_files": report.changed_files,
            "complexity": report.max_complexity,
            "regressed": report.regressed_tests,
            "unsafe_pattern": report.secret_leak,
        })
    return {
        "case_id": case.id,
        "pass_at_k": mean(r["passed"] for r in results),
        "variance": stdev(r["passed"] for r in results),
        "safety_ok": all(not r["unsafe_pattern"] for r in results),
    }

Execution must be isolated: containerized sandbox, no external network access, resource and timeout limits, and no access to real credentials. This is both a security requirement and a prerequisite for reproducible evaluation results.

5. Quality Metrics Beyond Pass Rate

Reports should include multiple metrics rather than a single ranking. A radar chart is often more useful than a linear leaderboard.

• Single-run pass rate and k-sample pass rate: The latter is closer to practical usability, while the former reflects stability.
• Reasonableness of change scope: For repair tasks, an ideal patch should be minimal. A much larger diff than necessary often means the model did not truly identify the root cause.
• Regression rate: The proportion of changes that cause existing tests to fail. This is one of the most direct quantifications of engineering cost.
• Static checks and complexity delta: Whether generated code increases duplicate code, cyclomatic complexity, or function length.
• Safety: Whether fixed unsafe patterns appear, such as hardcoded credentials, string-concatenated SQL, unsafe deserialization, or logging sensitive information. Rule-based scanning can detect these at low cost.
• Maintainability: Naming consistency, alignment with existing project abstractions, and whether comments describe facts rather than merely restating code.
• Response latency and invocation cost: Compare time and cost per task under similar quality levels to avoid paying a large premium for marginal quality gains.

6. Validating Long-Context and Open-Ended Generation Tasks

Beyond function-level tasks, another major category is long-form structured generation, such as producing long documents, technical proposals, or narrative chapters while maintaining consistency across long contexts.

For example, generating tens of thousands of words with consistent world-building, faction relationships, cultivation systems, and chapter pacing is fundamentally a long-context consistency problem.

These tasks cannot be validated by unit tests alone and require a different acceptance process:

1. Spot-check setting consistency: Extract key settings from the generated output, such as characters, place names, power systems, and important items, then compare them against the predefined setting sheet. Count conflicts.
2. Structural completeness: Check whether the number of chapters, chapter goals, and escalation of conflicts follow the planned outline. An outline-to-output mapping table can help verify this.
3. Local quality sampling: Randomly sample several passages and score them on four dimensions: detail density, logical coherence, naturalness of language, and avoidance of repetition or verbose filler.
4. Consistency decay curve across long outputs: Split the long output into sections by position and observe whether the conflict rate increases as later sections appear. This helps identify when long-context consistency starts to degrade.
5. Engineering-side compensation: Use setting sheets and outlines as structured context. Techniques such as chunked generation, summary replay, and key-entity checklist validation can reduce drift. In many cases, this kind of context orchestration is more valuable than simply switching to another model.

7. Practical Recommendations

• Build an evaluation set before making model decisions. With 30–50 real tasks that cover your own tech stack, you can obtain information that is more useful for decision-making than public leaderboards.
• Use tiered deployment. Function-level completion, single-file refactoring, and repository-level repair may require different configurations and different context strategies. Do not force one model to cover every scenario.
• Treat context engineering as first-class engineering. Retrieving relevant files, injecting coding conventions, and providing dependency versions and interface definitions usually deliver a better return than repeatedly comparing models.
• Keep checkpoints in human-AI workflows. Generated output should pass four gates: compilation, testing, static scanning, and code review. High-risk changes should require human confirmation before merging.
• Run regression evaluation continuously. Models evolve, and task distributions drift. Scheduling the evaluation pipeline on a regular basis helps detect quality regressions.
• Put cost and latency in the same table as quality. Quality, execution time, and cost together determine the final selection decision.

8. Conclusion

There is no universal answer to “Which coding LLM is strongest?” The answerable question is: Under my constraints, which configuration performs best?

Reliable judgment depends on three things: decomposing capability into measurable dimensions, building a private evaluation set aligned with your business, and ensuring reproducible results with isolated sandboxes and automated scoring. Once these three are in place, the benefits and risks introduced by model updates can be tracked quantitatively, turning model selection from a one-time bet into a sustainable engineering decision.