Why Are Large Language Models So Good at Coding? An Evaluation and Selection Guide

1. Start with the Right Question: Coding Ability Is Not a Single Score

In recent public evaluations, leading models have improved their pass rates on real-world repository bug-fixing tasks from roughly 60% in earlier tests to nearly 90%. Several models have also exceeded 90% on PhD-level scientific reasoning and competitive programming tasks.

These numbers naturally trigger debates about “which model is the strongest.” However, coding ability is not a single metric. It includes at least:

• Code generation
• Bug localization
• Cross-file modification
• Test generation
• Code review
• Dependency understanding
• Tool use
• Long-context repository understanding
• Command execution
• Result verification

Different tasks place very different demands on a model. Some emphasize memorization and pattern matching, while others require reasoning, planning, tool coordination, and feedback from execution.

Model comparisons should therefore define four conditions at the same time:

1. Task type
2. Available tools
3. Reasoning budget
4. Acceptance criteria

Without a defined context, claims about the “strongest” model are often unreliable. A more useful question is:

For a given codebase, programming language, defect type, and cost constraint, which type of model can consistently pass private acceptance tests?

2. Four Technical Sources of Strong Coding Ability

2.1 Data: High-Quality Code and Real Change Histories

Data is the foundation of coding ability. Large-scale code corpora include more than source files. They may also contain comments, documentation, test cases, commit histories, issue reports, and the relationships between issues and pull requests.

High-quality chains connecting an issue description, a code patch, and test results help models learn how to map observed symptoms to effective fixes. Data processing is equally important. Deduplication, low-quality content filtering, license and privacy protection, and stratified sampling by language and domain can all affect a model’s stability in real engineering environments.

2.2 Training: Continued Pretraining, Instruction Tuning, and Execution Feedback

General pretraining gives models knowledge of syntax and common programming patterns. Continued pretraining on code strengthens their understanding of repository-level structures. Instruction tuning teaches models to generate patches and explain changes according to requirements.

Preference optimization can further improve readability and safety. More importantly, execution feedback and unit-test rewards can move a model from merely being able to write code to producing code that passes tests.

One practical method is to generate multiple candidate patches, run the test suite, and retain the candidates that pass. Over time, this can help the model learn more reliable repair strategies.

2.3 Reasoning and Tools: Long Context, Retrieval, and Test Feedback

Real-world bug fixing often requires locating relevant code across multiple files. Long-context support allows a model to process more code, but it is not a complete solution.

A more effective approach combines repository indexes, symbol search, dependency graphs, and change-impact analysis to provide the model with the most relevant context. Tool calling can turn a model from a text generator into a repair agent capable of:

• Reading files
• Searching for symbols
• Running tests
• Inspecting logs
• Retrying based on failure messages

Test execution supplies objective feedback and makes multi-step repair possible.

2.4 Engineering: Sandboxes, Permissions, Caching, and Observability

To turn model capability into engineering productivity, the model must operate within a controlled workflow. Important safeguards include:

• Sandboxed execution
• Least-privilege access
• Dependency caching
• Log auditing
• Failure rollback
• Cost monitoring

Without this engineering support, even a high-scoring model can create unacceptable risks in production environments.

3. Why Evaluation Scores Cannot Be Compared Directly

First, task definitions vary. Real-world bug fixing, scientific reasoning, competitive programming, and Chinese-language knowledge question answering measure different capabilities. Their scores cannot simply be added together or compared as if they represented the same skill.

Second, test versions and reasoning settings differ. Whether tool calls are allowed, whether multiple retries are permitted, and how sampling temperature and reasoning budgets are configured can all significantly affect results.

Third, public leaderboards face risks from data contamination and overfitting. Strong performance on public datasets does not guarantee strong performance on a private repository.

Fourth, cost and latency are often excluded. A high-scoring solution that is expensive per task and slow to respond may not be suitable for high-frequency workloads.

Fifth, safety and compliance are usually not reflected in pass rates. Risks such as license violations, sensitive information leakage, and unauthorized access may not appear in standard coding benchmarks.

4. Building a Business-Oriented Evaluation Matrix

Organizations should not rely solely on public leaderboards. Instead, they should build an evaluation matrix based on their own engineering requirements.

Evaluation dimension	Core question	Suggested metrics	Data source
Real-world bug fixing	Can the model locate and fix historical defects?	Pass rate, first-pass rate, regression failure rate	Historical defects and tests
Code generation	Can it generate runnable code from requirements?	Compilation rate, test pass rate, amount of manual editing	Business requirement samples
Code review	Can it identify high-risk changes?	Defect recall, false-positive rate	Historical review records
Test generation	Can it produce useful missing tests?	Coverage improvement, mutation testing pass rate	Existing test suites
Cross-file refactoring	Can it preserve interfaces and behavior?	Regression pass rate, impact-analysis accuracy	Module dependency graphs
Chinese business-domain understanding	Can it understand Chinese requirements and comments?	Requirement reconstruction accuracy, terminology accuracy	Chinese business samples
Tool use	Can it use retrieval and execution tools correctly?	Tool success rate, retry count	Sandbox logs
Cost and latency	Does it meet interactive and batch-processing requirements?	Cost per task, P95 latency	Billing and monitoring data
Security and compliance	Does it leak sensitive information or introduce risk?	Blocking rate, audit coverage	Security scans and audits

A weighted formula can be used to calculate an overall score:

Overall score = Σ(metric score × weight)
                − cost penalty
                − latency penalty
                − risk penalty

Weights should be defined jointly by business and engineering teams. This helps prevent technical metrics from becoming disconnected from business value.

5. Why Real-World Bug Fixing Is More Informative

A real-world bug-fixing task usually includes six stages:

1. Understand the issue report
2. Locate the relevant code
3. Reproduce the problem
4. Generate a patch
5. Run tests
6. Check for regressions

This process evaluates semantic understanding, repository search, cross-file reasoning, tool use, and verification at the same time. As a result, it is generally closer to practical engineering value than isolated code-generation tasks.

A pass rate approaching 90% for leading models suggests that some low-risk, clearly bounded defects can be fixed independently by a model, or that the model can provide a high-quality patch proposal. However, important limitations remain. Ambiguous requirements, cross-service calls, dependencies on production data, permission restrictions, and architectural trade-offs still require human leadership.

A tiered collaboration model is more practical:

• Low-risk, repetitive defects: Allow automatic repair with human spot checks.
• Medium-risk tasks: Have the model generate patch proposals and test cases for engineer review.
• High-risk refactoring and core-path changes: Keep humans in control while using the model to analyze the impact.

6. Implementation Practice: Private Benchmarks and a Sandboxed Feedback Loop

Step 1: Build a Private Evaluation Set

Sample cases from historical defects, pull requests, and test changes. After removing sensitive information, organize the cases by language, module, and difficulty.

Freeze the test set during evaluation and refresh it periodically to reduce the risk of overfitting.

Step 2: Add Execution-Based Verification

Every candidate patch should run in a sandbox with tests, static checks, and security scans. If execution fails, record the reason and limit the retry budget.

Step 3: Improve Context Engineering

Combine repository indexing, symbol search, dependency graphs, and change-impact analysis. The goal is to reduce irrelevant context and provide the model with the information most relevant to the task.

Step 4: Control Cost and Latency

Route simple tasks to smaller models or rule-based systems, and escalate only complex tasks. Use caching and batch processing for high-frequency requests.

Step 5: Complete Security and Compliance Controls

The workflow should include least-privilege access, code exfiltration protection, audit logs, license checks, and clear documentation of ownership for generated code.

An illustrative evaluation loop is shown below:

cases = load_private_cases()
for case in cases:
    context = build_context(case.repo, case.issue, case.tests)
    patch = model.generate_patch(context, tools=[search, read, test])
    result = sandbox.apply_and_test(case.repo, patch, case.tests)
    record(case.id, result.passed, result.regression, latency, cost)
score = aggregate(records, weights)

The key is not one-time generation. It is the complete generate–execute–provide feedback–retry loop. Only when test results are used as an objective signal can model capability be converted into consistent engineering output.

7. Selection Conclusion: There Is No Single Best Model

Coding ability is the combined result of data, training, reasoning, tools, and engineering controls. Evaluation scores can provide useful signals, but they cannot replace private validation.

When selecting a model, organizations should first define task tiers, cost constraints, and risk boundaries. They should then run a small pilot using private evaluation sets and sandboxed execution before expanding the deployment scope.

Avoid both leaderboard-only decision-making and one-time full-scale replacement. The more reliable path is:

Define standards with real scenarios, use those standards to drive evaluation, use evaluation results to guide model selection, and rely on engineering controls to manage risk.