Beyond Benchmarks: How to Evaluate the Real Coding Capabilities of LLMs

I. Introduction: Tied Scores, Polarized Experiences

Recently, I have offloaded a significant portion of my typical daily development work—fixing bugs, adding APIs, modifying frontend pages, and cross-file refactoring—to LLM-powered coding agents. A recurring phenomenon is that two models with nearly identical scores on benchmark reports, differing by only a few percentage points, can exhibit immediately obvious gaps in real-world projects. One model responds quickly, produces usable code on the first try, and completes tasks end-to-end; the other frequently stalls midway, goes in circles, and requires constant human pushing to move forward.

This is not an isolated case but a structural issue with current LLM evaluation systems: writing code is not an exam. Response speed, mid-task stalling, and the need for repeated human intervention—factors that truly dictate daily experience—are entirely absent from benchmark statistics. This article discusses two things: what benchmarks actually tell us, and what capabilities beyond benchmarks make the real difference, as well as how to turn them into measurable metrics.

II. What Benchmarks Can and Cannot Tell Us

2.1 The Legitimate Uses of Benchmarks

Benchmarks are not useless; their value lies in two areas:

• Coarse Filtering: They quickly eliminate models with obvious capability deficits, preventing wasted effort in the wrong direction.
• Trend Reference: Score changes across a model’s version iterations roughly reflect the direction of capability evolution.

However, when multiple models cluster into the same tier on a benchmark, its resolution is insufficient to support selection decisions.

2.2 Three Key Blind Spots

Blind Spot 1: Task Distribution Mismatch. Mainstream code evaluations are mostly “standalone tasks”: a clean repository, a clearly described bug report, and explicit acceptance tests. Real-world development, conversely, deals with legacy codebases: requirements scattered across fragmented discussions, tangled inter-module dependencies, and acceptance criteria defined as “fix the issue without breaking existing behavior.” The capability subsets assessed by these two task types do not overlap.

Blind Spot 2: Static Answering vs. Multi-Turn Agentic Interaction. Benchmarks mostly measure “one-shot generation” quality; real-world usage involves a multi-turn process where the model autonomously calls tools, reads/writes files, executes commands, and corrects itself based on feedback. Whether this loop is stable—whether it gets stuck, loops, or hallucinates erroneous intermediate states—is completely ignored by benchmarks.

Blind Spot 3: The Engineering Configuration Black Box. The same model can yield score differences of nearly ten percentage points depending on the source. The reasons are not mysterious:

• Different inference tiers: Varying thinking budgets and reasoning strength settings.
• Different agent scaffolding: Varying system prompts, toolsets, context assembly, and compression strategies.
• Different timeout/retry strategies and sampling parameters: Whether a failure leads to an immediate abort or multiple retries heavily impacts the final pass rate.
• Discrepancies between self-reported and third-party metrics: Vendor self-reports usually represent the “upper limit under optimal conditions,” while third-party tests approximate what an average user actually achieves. Comparing them directly is meaningless.

A simplified illustration shows the impact of scaffolding:

# Main loop of a coding agent (simplified): 
# Engineering choices at each step alter the final performance
while not task.finished:
    context = assemble_context(repo, task, history)  # How context is assembled and pruned
    action  = llm.decide(system_prompt, context)     # How system prompts and toolsets are designed
    ok, feedback = sandbox.execute(action)           # How timeout and retry strategies are set
    if not ok:
        history.remember(feedback)                   # Whether it can effectively self-correct after failure

The same model weights placed in different loops become different “products.” Furthermore, the evaluation sets promoted by different vendors often do not overlap: different questions, rules, and difficulty distributions mean the numbers cannot be directly converted. Thus, “who is better” depends entirely on which leaderboard you look at—which inherently proves benchmarks cannot be the sole basis for decision-making.

III. Beyond Benchmarks: Six Dimensions Determining Real Experience

Dimension	Covered by Benchmarks?	Impact on Daily Experience
Single-task code correctness	Mostly covered	Determines the capability floor
Long-range task autonomy	Barely covered	Determines if it needs “human pushing”
First-pass success rate / Rework rate	Partially covered	Determines review and debugging costs
Cross-file & repo understanding	Limited coverage	Determines capability for real projects
Response latency	Not covered	Determines interaction rhythm
Stability & quota endurance	Not covered	Determines viability for intensive use

1. Long-Range Task Autonomy. Once a task is assigned, can the model complete it end-to-end: decomposing steps, locating relevant files, modifying, self-testing, and reporting results? Stalling midway waiting for a user push, or oscillating between two approaches, is the most mentally draining failure mode in daily use. Behind this lies planning capability, self-repair ability after tool call errors, and cross-turn progress memory.

2. First-Pass Success Rate and Rework Cost. There is an order-of-magnitude cost difference between “generated code is mostly usable on the first try” and “it compiles but has logic pitfalls.” The former means short review times and low trust costs; the latter means you are doing the job the model should have done. Rework rate is a far more sensitive daily metric than a benchmark score.

3. Codebase Understanding and Context Management. Real-world project issues rarely confine themselves to a single file. Whether the model can track cross-file dependencies, adhere to existing code styles and architectural conventions, and retain earlier constraints in long sessions directly determines if it is just a “patch monkey” or a “collaborator.”

4. Response Latency and Interaction Rhythm. First-token latency and generation speed dictate the fluidity of conversational development. Coding requires a continuous train of thought; waiting thirty seconds versus three seconds represents entirely different working states.

5. Stability and Quota Endurance. Under high-intensity continuous use, will you encounter rate limiting, throttling, or quota exhaustion? A tool that makes you “worry about quotas while coding” will actually change your usage patterns—you will subconsciously use it less, reducing the sample size and making trust harder to establish.

6. Instruction Following and Communication Cost. Can it understand requirements after a single explanation? When encountering ambiguity, does it ask for clarification or make arbitrary assumptions? When corrected, does it genuinely fix the issue or just superficially comply? These “collaborative qualities” are untested by benchmarks but examined every single day.

IV. Implementation: Turning “Usability” into Measurable Metrics

4.1 Build a Personal Task Set

Select 10 to 20 representative tasks from your own real repository, covering four categories: bug fixes, small feature development, refactoring, and test additions. Write clear acceptance criteria for each task (runnable tests or explicit manual checklists). The task set need not be large, but it must be real—synthetic questions cannot test performance under legacy constraints.

4.2 Strictly Control Variables

• Same scaffolding and toolchain, same prompt templates;
• Same acceptance scripts and review standards;
• Vary only the model (and its tier), keeping everything else fixed.

It must be emphasized: the object of evaluation is the combination of “model + common tools,” not the isolated model itself.

4.3 Track Five Core Metrics

First-pass success rate, manual intervention count, end-to-end duration, rework rate, and quota cost per task. Use a simple record table to accumulate data continuously:

from dataclasses import dataclass, asdict
import json, statistics
@dataclass
class TaskRecord:
    task_id: str          # Task ID
    category: str         # bugfix / feature / refactor / test
    model: str            # Model and tier
    first_pass: bool      # Whether the output passes acceptance on the first try
    interventions: int    # Number of manual interventions (clarifications/corrections)
    duration_min: float   # End-to-end duration (including waiting and debugging)
    quota_cost: float     # Quota or monetary cost
    rework: bool          # Whether rework was needed after merging
def summarize(records):
    n = len(records)
    return {
        "First-pass success rate":   sum(r.first_pass for r in records) / n,
        "Average interventions":     statistics.mean(r.interventions for r in records),
        "Median duration (min)":     statistics.median(r.duration_min for r in records),
        "Rework rate":               sum(r.rework for r in records) / n,
    }
# Append a JSONL record after completing each task. 
# Once a dozen tasks are accumulated, stable differences will emerge.
# with open("eval_log.jsonl", "a") as f:
#     f.write(json.dumps(asdict(record)) + "\n")

4.4 Combine Subjective and Objective Scoring

On top of objective metrics, you can assign a subjective score (e.g., 5-star rating) to this combination based on the six dimensions outlined in Section III. Subjective scores explain the “feeling,” objective metrics explain the “why,” and the two cross-validate each other.

4.5 Tiered Fallback Strategy

Regardless of whether the model comes from a cloud provider—such as Baidu AI Cloud or other vendors offering LLM services—or from community-released open-weight models, most services offer multiple capability tiers. A universal approach is: assign routine tasks to cost-effective tiers, escalate a few tough challenges to flagship tiers, and accept the reality that “humans take over problems no model can solve.” The goal of evaluation is not to find a universal model, but to find the right tool and reasonable expectations for each task type.

V. Applying Evaluation Results to Daily Work

1. Route models by task type: Use faster tiers for mechanical modifications, stronger tiers for cross-file refactoring.
2. Standardize prompts and workflows: Reduce variables in each session to make results reproducible.
3. Conduct regular regression tests: Models iterate rapidly, and the validity of previous conclusions typically lasts only a few months. Re-running your task set offers a low-cost refresh.
4. Cross-reference data sources: Vendor self-reports, third-party evaluations, and personal task sets have increasing weights of credibility, respectively.

VI. Conclusion

Benchmarks answer “where is this model’s ceiling roughly,” while daily development must answer “can it reliably help me finish today’s work.” When multiple models cluster into the same tier on benchmarks, the decision-making basis should shift toward things benchmarks fail to track: autonomy, first-pass success rate, intervention cost, interaction rhythm, and endurance. By breaking them down into metrics and building your own task set, whenever a new round of models is released, you only need a day or two to get an answer much closer to your actual work scenario than any benchmark could provide.