Beyond Benchmarks: How to Evaluate the Real Coding Capabilities of LLMs

Handing real-world daily development tasks—such as fixing bugs, adding new APIs, adjusting frontend pages, and performing cross-file refactoring—to different AI coding assistants is the best way to test a model’s true caliber. An interesting phenomenon is that several models with nearly identical scores on mainstream code benchmarks and only a few percentage points apart in general intelligence indexes can offer drastically different experiences in actual use. Some respond quickly and produce code that is ready to use on the first try; others require frequent human intervention, often stalling halfway through a task or going in circles.

Writing code is not an exam. Exams have standard answers, fixed time limits, and unified scoring rules, whereas real-world development is a long-chain collaboration in an open environment: how fast the model responds, whether it gets stuck midway, and how many rounds of prompts it takes to converge—none of these factors determining daily experience are captured by benchmarks. This article discusses two things: what leaderboards actually tell you, and beyond them, what factors truly determine whether a model is “good to use.”

1. Why Leaderboards Can Be Distorting: The Same Model Can Have a 10-Point Score Gap

When trying to compare models horizontally, there are very few common yardsticks available. Often, only a few terminal environment coding tasks have been tested across the board by everyone. However, even for the same test, numbers reported by different sources can vary wildly—the same model can have a score gap of nearly 10 percentage points across different channels. The reasons are not mysterious, mainly boiling down to four points:

1. Different reasoning tiers. Modern LLMs typically offer multiple reasoning configurations. Whether deep thinking is enabled and how much thinking budget is allocated will significantly impact results. The high-budget tier approaches the model’s capability ceiling, while the default tier represents the daily reality for most users. The two should not be confused.

2. Different agent scaffolding. Models in evaluations rarely “take the test naked”; they are usually wrapped in an execution framework: how tasks are decomposed, how tools are invoked, and how errors are read and retried. Putting the same “brain” into different qualities of “bodies” naturally results in different performances.

3. Different timeout and retry strategies. When encountering environment errors, whether the model immediately gives up, retries a fixed three times, or dynamically adjusts based on the error type will alter the final pass rate. Aggressive trial-and-error strategies can inflate scores but do not necessarily indicate that the model itself is stronger.

4. Different evaluation details. Sampling temperature, submission format validation, and hidden test rules—any difference in these will propagate to the final numbers.

From this, we can draw two important conclusions:

First, vendor-reported numbers usually represent the “upper limit under optimal conditions,” while third-party retests are closer to what ordinary users can achieve. Mixing the two for horizontal comparison is meaningless.

Second, the “signature scores” promoted by various vendors hardly overlap—one focuses on software engineering benchmarks, another on terminal task benchmarks, and a third on long-horizon maintenance benchmarks. With different questions and rules, the numbers cannot be directly converted. Therefore, looking purely at scores, several models often fall into the same tier, and “who is stronger” simply depends on which leaderboard you open.

2. What Leaderboards Can and Cannot Answer

Leaderboards are not useless; their value lies in at least two aspects:

• Coarse-grained tiering. The gap between the top and the bottom is real. A model consistently scoring at the bottom is unlikely to suddenly perform miraculously in real-world scenarios.
• Relative reference for specific sub-capabilities. A model that scores significantly higher on benchmarks heavily weighted toward test generation is highly likely to be proficient at writing unit tests.

However, leaderboards cannot answer the questions most relevant to daily development:

• End-to-end latency: How long do you have to wait from submitting a task to getting a usable result?
• Long-task autonomy: If you hand it a cross-file refactoring task, can it complete it all the way through, or will it stop midway and go in circles?
• Human intervention frequency: How many rounds of prompts are needed to converge? How much time is spent reviewing each rework?
• Cost and quota sustainability: If used intensively all day, is the quota sufficient? Will you be anxious about the quota while writing code?
• Fit: How well the model integrates with your existing codebase, tech stack, and toolchain cannot be reflected by a leaderboard.

These are precisely “experience-based metrics”: difficult to standardize and automatically score, so they almost never appear on any leaderboard, yet they dictate your actual daily efficiency.

3. Five Experience Dimensions That Truly Set Models Apart

1. First-pass success rate and rework cost. A good model’s output is close to “ready to merge,” whereas a weaker model might appear to finish writing but actually requires an equal or greater amount of time to fix edge cases. When evaluating, do not just look at “whether it can generate code”; calculate the total cost from generation to usability.

2. Autonomy in long tasks. Real-world tasks often require a continuous sequence of a dozen steps: locating the problem, making multiple modifications, running validations, and self-correcting based on errors. Whether it stops midway waiting for instructions or repeatedly goes in circles on the same error is the watershed of autonomy. When encountering a difficult problem truly beyond its capability, a reasonable approach is to promptly hand off to a higher-tier flagship model as a fallback, rather than getting stubbornly stuck.

3. Cross-file context management. Engineering-level changes have a butterfly effect: modifying a data structure requires synchronously updating serialization logic, API contracts, and test cases. Whether the model can proactively retrieve relevant files and maintain consistency across multiple changes directly determines its usability in real projects.

4. Interaction rhythm. Response latency dictates whether your “flow state” is interrupted. In exploratory tasks, fast but slightly rough feedback is often better than slow and precise answers, because developers need a conversational feel of rapid iteration.

5. Stability and quotas. Evaluations run once, but daily use runs for a year. Rate limiting during peak hours, truncation of long tasks, and quota anxiety—these operational factors also constitute part of the experience, yet are ignored by the vast majority of evaluations.

4. A Practical Self-Testing Methodology

Rather than agonizing over leaderboards, it is better to use real projects to establish your own yardstick. The core principle is controlling variables: keep the toolchain and scaffolding fixed and only swap the model; keep the task set fixed and only compare results.

Step 1: Build a personal task set. Extract one or two tasks from each of four categories based on recent real-world work: a typical bug fix, adding a new business API, a frontend page adjustment, and a cross-file refactoring. Write each task into a standardized “task card” recording the initial code state, acceptance criteria, and known constraints.

Step 2: Unify evaluation conditions. Use the same AI coding agent, the same reasoning tier, and the same timeout configuration. Run the full set of tasks for each model sequentially without interrupting or manually fixing things during the process, ensuring the results are comparable.

Step 3: Record quantitative metrics. It is recommended to record at least the following five:

• First-pass success rate: The percentage of tasks that pass acceptance without manual modification.
• Human intervention count: The number of rounds requiring supplementary prompts or manual corrections.
• End-to-end time consumption: The total time from task submission to acceptance.
• Rework rounds: The number of times code had to be regenerated due to unmet requirements.
• Subjective score: A 5-point scale evaluating the overall feel of the “model plus tool” combination.

Task cards and records can be maintained using a structured template, for example:

task: fix-login-timeout
type: bugfix
files_touched: 3
acceptance:
  - All relevant regression tests pass
  - No new compilation warnings introduced
metrics:
  first_pass: false
  interventions: 2
  total_minutes: 18
  rework_rounds: 1
  subjective_score: 4

Step 4: Regularly retest. Models iterate quickly. Running the same set of tasks again after a quarter will reveal the actual pace of improvement for each vendor, which is more valuable than chasing leaderboard updates.

5. Conclusion: Treat Leaderboards as Maps, Projects as Yardsticks

Leaderboards are like maps: they can tell you the general terrain and tiering, but they cannot tell you which road is easy to travel today. Real projects are the yardstick: your tech stack, your codebase, and your work rhythm determine that the same model might be two completely different things in the hands of different people.

A more pragmatic conclusion is: for models with similar scores, default them to the “same tier,” and then use your own task set for a small-scale trial run. Make decisions based on experience metrics like first-pass success rate, intervention count, and end-to-end time. Benchmarks tell you “who might be strong,” but real projects tell you “who works well for you”—and the latter is the answer that truly matters when it comes to writing code.