Why LLM Code Capabilities Are Improving: Technical Drivers and Evaluation Methods
This article examines the technical factors behind recent improvements in LLM code capabilities, including training data quality, reinforcement learning with verifiable outcomes, long-context repository understanding, tool use with execution feedback, and inference-time compute. It also explains why public benchmark rankings can be misleading and provides a practical framework for building internal evaluation processes tailored to your own engineering scenarios, covering scenario definition, evaluation set construction, metric selection, cost calculation, and risk control.
I. Starting with a Number
Over the past year, publicly reported results on realistic software bug-fixing benchmarks have improved significantly. These benchmarks present a model with an open-source project and an issue description, requiring it to locate relevant files, modify code, and pass the project’s own test suite. They assess multi-step engineering capability rather than single-question problem-solving.
A year ago, top models scored around 60% on these benchmarks. In recent public results, some leading models have approached 90%. But this number needs careful interpretation. Benchmark versions, scaffolding (the orchestration and tooling wrapped around the model), reasoning modes, and prompt design all affect results. The same model with a different scaffold can show dramatically different scores. Readers should therefore understand these figures as “results under specific conditions” rather than fixed capability values.
II. Five Technical Reasons for Improved Code Capabilities
1. Higher Quality Training Data
Data quality matters more than quantity for code. Training corpora typically go through deduplication, license filtering, syntax checking, and quality scoring. Teams also use executable verification to generate synthetic data: they generate problems and reference solutions first, then filter out incorrect samples using unit tests. This produces triplets of problem, test, and solution that are better suited to training repair capabilities than ordinary web code.
2. Reinforcement Learning with Verifiable Outcomes
Code is one of the few task domains where correctness can be automatically determined. Running a test suite produces a pass or fail result that can directly serve as a reward signal. Compared to open-ended writing tasks that require human grading, code tasks offer more stable feedback at lower cost, making reinforcement learning particularly effective in this domain.
3. Long Context and Repository-Level Understanding
Real bugs often span multiple files. Models need to understand call relationships, configuration items, and historical changes. With larger context windows, models can read more relevant code at once. Combined with retrieval and summarization, models can first locate relevant modules and then read details more deeply, reducing unnecessary modifications.
4. Tool Calling and Execution Feedback
Models no longer just output a block of code—they can read files, edit files, run tests, and iterate based on error messages. Below is a simplified illustration of an agent loop; function names are fictitious and intended only to demonstrate the flow:
def fix_issue(repo, issue, model, max_steps=30):context = build_context(repo, issue)for _ in range(max_steps):action = model.next_action(context)if action.type == "read_file":context.add(read_file(repo, action.path))elif action.type == "edit":apply_patch(repo, action.patch)elif action.type == "run_tests":result = run_tests(repo)context.add(result)if result.passed:return action.summaryreturn None # Exceeded step limit without passing; treat as failure
This closed loop of “act, observe, act again” is key to improved engineering capability. Models are shifting from one-shot generation to continuous correction.
5. Inference-Time Compute
Models can invest more thinking steps before responding. Another approach is to sample multiple times, generate several candidate patches, and select among them using test results. These methods increase latency and cost but typically improve success rates on difficult problems. Whether to enable them depends on the value of the task.
III. Why “Who’s the Best” Is a Misleading Question
Public leaderboards commonly contain four types of pitfalls.
- Different evaluation dimensions: Software engineering, graduate-level scientific reasoning, competitive programming, mathematics, and Chinese language understanding each test different capabilities. Within the same period, models from different vendors trade wins across different individual benchmarks.
- Different data sources: Some scores are self-reported by vendors, some come from third-party aggregations, and some are platform-collected values. When citing, specify the source and distinguish whether results have been independently reproduced.
- Different test conditions: Reasoning modes, context lengths, sampling counts, and scaffolding all change results. Comparing numbers obtained under different conditions directly has limited meaning.
- Rapid version iteration: Leaderboard data can become outdated within weeks. Numbers in articles should be treated as snapshots at a specific point in time.
Additionally, small differences in individual benchmark scores are usually not statistically significant. A one- or two-percentage-point difference may simply be random fluctuation or an incidental result of the test set.
The right question, therefore, should be: What scenario will you use it in, what metrics will you use to measure it, and how much cost are you prepared to pay?
IV. Building Your Own Model Selection and Evaluation Framework
Step 1: Define the Scenario
Common scenarios include code completion, bug fixing, code review, test generation, documentation writing, and Chinese-language requirements understanding. Different scenarios have very different requirements for accuracy, latency, and cost. For example, code completion requires low latency, bug fixing emphasizes correctness, and code review emphasizes recall.
Step 2: Construct an Internal Evaluation Set
The most reliable evaluation data comes from your own code repository. The specific approach is as follows:
- Extract already-fixed issues from historical bug tickets;
- Preserve the code snapshot at the time the bug occurred and the corresponding regression tests;
- Ensure that tests and fix logic are relatively independent to avoid the tests themselves leaking the answer;
- A sample size of at least several dozen is recommended, covering different modules and difficulty levels.
Below is a simplified evaluation flow; function names are again fictitious:
def evaluate(model, cases):results = []for case in cases:sandbox = prepare_sandbox(case.repo_snapshot)patch = model.generate_patch(case.issue, sandbox)passed = sandbox.apply(patch) and sandbox.run(case.tests)results.append({"id": case.id,"passed": passed,"cost": model.last_cost,})return summarize(results)
Step 3: Choose Metrics
We recommend recording the following metrics together:
- First-pass rate: Whether the model’s first patch passes the tests;
- pass@k: The proportion of cases where at least one of k generated candidate patches passes;
- Human modification cost: How many lines a reviewer needs to change before merging;
- Average time and cost: Measured by token count, number of calls, and reasoning mode;
- Regression rate: Whether the fix introduces new failing test cases.
Step 4: Calculate Costs
The actual cost of a model depends on input and output token counts, reasoning mode, retry counts, and scaffolding call strategy. Prices change with exchange rates and contracts; refer to the latest official pricing and review it periodically. The total cost of the same task should also include time for human review and rework.
V. Risk Control in Engineering Deployment
Minimal permissions. Agents should only have read and write access to sandbox environments, and should not directly access production branches or secrets. Human review is required before merging.
Prevent “test modification.” Models may make results green by deleting or weakening tests. Test files should be set to read-only, or test file hashes should be compared during evaluation.
Data compliance. Sensitive code and customer data should not be sent to unauthorized external services. Organizations should explicitly define which code can enter the model context.
Continuous evaluation. After model version updates, prompt adjustments, or scaffolding changes, the internal evaluation set should be re-run and compared against historical results.
VI. Conclusion
Improved code capabilities result from the combined effects of data, training methods, context management, and tool feedback loops. For users, it is more valuable to build an evaluation process tailored to their own scenarios than to ask “which model is the strongest.” Making decisions based on real tasks, clear metrics, and accountable costs will ensure that model selection conclusions remain valid as versions evolve.
