Beyond Benchmarks: Evaluating LLM Coding Capabilities in Real-World Development

I. The Phenomenon: Clustered Benchmark Scores, Divergent User Experiences

Over the past two years, public coding benchmarks for mainstream Large Language Models (LLMs) have become increasingly competitive. Models with similar positioning often claim the top spot on different leaderboards, with performance gaps of merely a few percentage points. When aggregated into various intelligence indices, their scores appear almost identical. Judging solely by these numbers, the conclusion is simple: they are in the same tier, pick any one.

However, real-world development tells a different story. Assign the same task to different models, and the outcomes vary drastically. Some respond quickly and produce usable code on the first try; others stall midway, requiring constant human intervention; some remain stable under high-intensity daily usage, while others make users constantly worry about quota consumption. None of these factors that dictate daily experience are captured by benchmarks.

The reason is straightforward: writing code is not an exam. Benchmarks measure short-haul capabilities based on static problems, fixed rules, and standard answers. Real-world development, conversely, is a collaborative process involving dynamic contexts, long-range tasks, and no single correct answer. These two fundamentally require different skill sets.

II. Why Benchmarks Distort Reality: Three Structural Reasons

1. Inconsistent Testing Conditions Cause Up to 10-Point Variances

For the same model on the same benchmark, scores reported by different sources can vary wildly. Taking terminal-based coding benchmarks as an example, public scores for a single model range from just over 60% to over 70%—a gap of nearly ten percentage points. The reasons are not mysterious:

• Different inference settings: Whether deep thinking is enabled and the allocated thinking budget directly impact scores.
• Different Agent scaffolding: Tool invocation methods, context management strategies, and file retrieval implementations vary.
• Timeout and retry strategies: Lenient retry policies can “grind out” higher pass rates.

Vendor-reported numbers typically represent the “upper limit under optimal conditions,” while third-party tests reflect what ordinary users can actually achieve. Mixing the two for comparison is inherently meaningless.

2. Vendor-Favored Benchmarks Rarely Overlap

Observing the “signature scores” highlighted in various technical reports, it becomes clear that they rarely appear on the same leaderboard. One model excels in software engineering repair benchmarks, another in end-to-end Agent task benchmarks, and yet another in long-range continuous development benchmarks. With different problems, rules, and scoring criteria, the numbers cannot be directly compared. As for aggregated indices that compress multiple dimensions into a single number, their discriminatory power is limited—a difference of a few points might mask entirely different capability structures.

3. Data Contamination and Overfitting Risks

Public benchmark problems are static. Over time, the probability of these problems being covered in training data increases. Models might have “memorized” the answers rather than genuinely mastering the underlying engineering capabilities. This is why many teams find that while benchmark scores rise quarter by quarter, performance in real projects does not necessarily improve synchronously.

The conclusion: Benchmarks are maps, not routes. They can be used for coarse filtering and identifying capability tiers, but they should not be the sole basis for model selection.

III. Four Dimensions That Truly Determine Daily Experience

Based on observations of real-world project development, factors that “benchmarks fail to measure but heavily impact experience” can be categorized into four dimensions.

1. Response Speed and Interaction Rhythm

• First-token latency and output throughput dictate the “Q&A rhythm.” In scenarios requiring rapid iteration or writing while debugging, waiting tens of seconds each time accumulates into significant hidden costs.
• More importantly, it affects developer flow. If the response is fast enough, developers are willing to delegate fine-grained problems to the model. If it’s too slow, developers tend to do it themselves, reducing the model’s utilization and overall value.

2. First-Try Pass Rate and Rework Cost

This is where user experience differs the most. Even if both generate “runnable code,” quality gaps hide in the details:

• Whether it understands implicit constraints: project conventions, dependency versions, existing code styles, and exception handling practices.
• Whether boundary conditions are fully handled: null inputs, concurrency, timeouts, and rollback paths.
• Whether the generated code requires major rework to integrate into the existing engineering project.

The cost of rework is not just rewriting; it includes reorganizing the context, re-reviewing changes, and rerunning tests. A model with a high first-try pass rate can be several times more efficient overall than a model with a “similar benchmark score.”

3. Autonomy in Long-Range Tasks

In tools like CLI-based coding Agents, models must execute multiple consecutive steps: locating files, modifying code, running tests, and continuing to fix based on errors. This tests:

• Continuous progression: Can it complete a task from start to acceptance, or does it stop halfway and wait for instructions?
• Avoiding loops: When facing failure, does it try a different approach, or does it repeat the same mistake?
• Self-verification: Does it proactively run tests and check outputs, or does it just “assume it’s fixed”?

This kind of long-range Agent behavior is exactly what existing benchmarks struggle to cover, yet it heavily impacts daily usage.

4. Stability and Quota Resilience

• Whether high-intensity usage triggers rate limiting or downgrades the model to a weaker inference tier.
• How fast the quota is consumed for the same workload.
• “Worrying about quotas while coding” tangibly alters usage patterns—developers stop asking questions they should ask and stop letting the model attempt things it should try, ultimately shrinking the output.

IV. Practical Guide: Building Your Own Evaluation System

Since public benchmarks are unreliable, the most pragmatic approach is to establish a private evaluation process based on your own projects. The core philosophy: use real tasks as test questions, use uniform conditions for comparison, and use quantifiable metrics to replace subjective impressions.

1. Curate a Fixed Set of Real Tasks

Select 10 to 20 representative tasks from your own projects, covering typical scenarios: bug fixes, small feature development, refactoring, test completion, and script/tool writing. Write each task as a structured “Task Card,” for example:

task_id: fix-session-leak
type: bugfix
context: |
  The session module occasionally fails to release connections under high concurrency, causing an error rate spike during stress testing.
inputs:
  - src/auth/session.py
  - logs/error_sample.log
acceptance:
  - Stress test script passes, error rate drops to zero
  - New regression test covers the connection release path
constraints:
  - Do not introduce new dependencies
  - Follow existing code style and exception handling practices

The value of a Task Card lies in clearly defining the “acceptance criteria,” preventing lenient, subjective evaluations.

2. Strictly Control Variables

• Use the same toolchain and scaffolding for candidate models: the same coding assistant or Agent configuration, identical file retrieval and test execution methods.
• Use the same prompt template, only swapping the model.
• Keep inference parameters (temperature, thinking budget, etc.) consistent.
• Record model versions and evaluation dates to avoid cross-version comparisons.

3. Replace Impressions with Quantifiable Metrics

Record the following fields for each evaluation, accumulating them into your own dataset:

import csv
FIELDS = ["task_id", "model", "first_try", "interventions",
          "rounds", "seconds", "tokens", "score"]
def log_result(**kw):
    with open("eval_log.csv", "a", newline="", encoding="utf-8") as f:
        csv.DictWriter(f, fieldnames=FIELDS).writerow(kw)
# Example: Log after a task is completed
log_result(task_id="fix-session-leak", model="model-A",
           first_try=1, interventions=0, rounds=3,
           seconds=412, tokens=58000, score=5)

Key metrics include:

• First-try pass rate: The percentage of tasks that pass acceptance without human modification.
• Human interventions: The number of times supplementary explanations, direction corrections, or manual pushes are needed midway.
• Round trips and end-to-end duration: Reflects interaction costs.
• Quota consumption: For the same task, token consumption differences are often larger than expected.

4. Supplement with Subjective Scoring to Form a Comparison Table

Beyond quantitative metrics, apply a 5-star subjective scoring system across the four dimensions to create mutual validation:

Dimension	Focus Points	5-Star Standard
Response Speed	First-token latency, throughput	Imperceptible wait throughout
First-Try Pass Rate	Rework and modification volume	Output is directly usable
Long-Range Autonomy	Stalling, looping, self-checking	Hand off and get it back completed
Quota Resilience	Rate limiting, downgrades, token usage	Stable under high-intensity all-day use

It must be emphasized that this scoring evaluates the actual experience of the “model + your preferred toolchain” combination, not a strict standalone benchmark. The conclusions are only valid for your own workflow, but precisely because they are tied to your workflow, they provide the most valuable reference.

5. Establish a Regression Mechanism

Model versions update frequently, and today’s conclusions might not apply next month. It is recommended to treat the task set as a regression test: rerun it when upgrading versions or switching candidate models to observe if metrics rise or fall. When the task volume is manageable, it can be scripted and executed periodically within a Continuous Integration (CI) pipeline.

V. Extension: Tiered Usage Strategy

The ultimate goal of evaluation is not to crown a single “king,” but to formulate a tiered usage strategy:

• Daily tasks (format conversions, simple functions, comments, and documentation) are assigned to tiers with fast responses and friendly quotas.
• Core tasks (complex bugs, architectural refactoring, cross-module modifications) are assigned to the primary tier with the highest first-try pass rate.
• Complex tasks (problems where the primary model repeatedly fails) are escalated to a stronger inference tier as a fallback.

Regardless of the cloud platform providing the model services, you can typically find general capabilities like multi-tier switching, usage statistics, and monitoring alerts to implement the above strategy as configurations rather than relying on manual switching. Taking Baidu AI Cloud’s model services as an example, this “multi-tier + usage monitoring” combination is a common practice for controlling costs and ensuring stability.

VI. Conclusion

Benchmarks answer “who can score the highest under optimal conditions,” while model selection should answer “who works best under my toolchain, my project, and my workload intensity.” The former is public and comparable but distorted; the latter is private and subjective but real.

Instead of jumping back and forth between dozens of leaderboards, spend an afternoon setting up your own task set, logging scripts, and scoring rubrics. Benchmarks are maps, but experience is the route—and the route must be forged by yourself.