How to Evaluate Large Language Models Scientifically: Coding, Agents, and Reasoning
Public leaderboard scores often fail to match real-world developer experience. This article explains why benchmark results and business goals are misaligned, then breaks model evaluation into six distinct capability dimensions—repository-level code repair, cross-language engineering, terminal tasks, interactive programming and front-end/document output quality, tool calling and retrieval, and complex reasoning. It shows how to read benchmark numbers correctly (baselines, saturation, variance, and scaffolding) and how to build a layered, in-house evaluation pipeline with reusable, replayable tooling, then closes with a checklist of common pitfalls. It is a practical methodology for translating "model capability" into success rate, unit cost, and human-intervention rate for your own business.
Where the Problem Starts: Strong Benchmarks, Yet “Not Usable” in Practice
Every time a new model version ships, the score improvements in public evaluations tend to dominate the conversation. But developers often feel a gap after integrating it: even when a leaderboard shows a double-digit gain in multilingual code repair, the assistant still edits the wrong file, misses a dependency, or repeatedly fumbles terminal commands when pointed at their own repository.
This isn’t benchmark fraud. It’s a misalignment between evaluation goals and business goals. Public benchmarks measure “average pass rate on a controlled task distribution,” while engineering teams care about “the probability of delivering a usable result in one shot under my codebase, my toolchain, and my acceptance criteria.” The systematic gap comes from four main sources:
- Task distribution bias: Benchmark tasks are cleaned and anonymized with clear boundary conditions; real repositories carry legacy baggage, private dependencies, implicit conventions, and a lot of “dirty” context.
- Scoring criteria bias: Benchmarks typically judge success by whether tests pass, but real delivery also weighs code maintainability, review cost, and rollback risk.
- Interaction pattern bias: Benchmarks mostly “provide full context and answer in a single turn,” while real usage is multi-turn collaboration of “reading the output and asking follow-ups.”
- Missing cost constraints: Benchmarks usually impose no token budget or latency ceiling, yet production environments are extremely sensitive to both.
So the right approach isn’t comparing “who scores higher,” but building an evaluation system that mirrors your own business. Below I first break down which capability dimensions today’s leading models are being tested on, then discuss how to read the numbers and how to build your own evaluation.
Six Capability Dimensions to Evaluate Separately
Treating “how strong is the model” as a single scalar is the most common mistake in model selection. Modern model capabilities are highly structured, and progress across dimensions is not synchronized. I recommend modeling and evaluating at least the following six dimensions separately.
2.1 Repository-Level Code Repair
This is the task category that best reflects real engineering value. The typical form: given a repository snapshot of an open-source project and a problem description, the model must locate the defect, modify the code, and a verdict on whether it is fixed comes from the project’s own tests.
The difficulty lies not in writing code but in locating and making minimal changes within a long context. When evaluating, pay special attention to three sub-metrics: localization hit rate (did it modify the correct file and function), first-pass rate (no manual rework needed), and regression rate (fixed the target problem but broke existing tests).
2.2 Cross-Language Code Engineering
Real systems are rarely single-language. Cross-language benchmarks test whether, when a repository contains multiple languages, multiple build tools, and multiple test frameworks, the model can correctly identify the project structure and invoke the appropriate build and test flows. This dimension depends far more on context retrieval strategy than on the model itself, so the retrieval approach should be fixed as a variable during evaluation—otherwise the data isn’t comparable.
2.3 Terminal Tasks
The terminal is the agent’s “hands and feet.” Such tasks provide no graphical interface; the model can only infer environment state by executing commands, reading logs, and checking exit codes. It tests state awareness and error recovery: when a command fails, does the model blindly retry, or can it locate the root cause from the error message? A common finding is that models are generally weaker on this dimension than on pure code generation, because terminal feedback is indirect, noisy, and often unstructured.
Suggested metrics include: average number of interaction turns, proportion of ineffective commands, whether destructive operations occur (e.g., accidentally deleting directories, force-overwriting), and whether the task is “abandoned and a conclusion fabricated” midway.
2.4 Interactive Programming and Front-End / Document Output Quality
An obvious trend over the past two years is that evaluation has expanded from “is the code correct” to “is the output polished and usable.” Concretely: whether generated front-end pages have precise layouts, reasonable responsive breakpoints, and restrained color and typography; whether generated slides have no text overflow, aligned hierarchy, and well-proportioned charts.
This capability is hard to judge with binary tests and usually requires a dual track of rule checks + human or model review: the rule layer validates structural correctness (whether it renders, whether there is overflow, whether controls are interactive), while the review layer scores visual quality and usability against a unified rubric. Be sure to fix the review prompts and scoring anchors, otherwise score variance will drown out real differences.
2.5 Tool Calling and Retrieval Tasks
The core capability of an agent framework is “selecting the right tool, passing the right arguments, and understanding the return.” When evaluating, distinguish three failure types: selection errors (should have retrieved but went to compute instead), argument errors (field name or type mismatch), and result misuse (got the return but drew the wrong conclusion). For multi-turn tool-calling tasks, also track average call count and redundant-call ratio—too many calls mean higher latency and cost.
Retrieval subtasks (e.g., information gathering and multi-hop verification) have their own challenges: the answer is scattered across multiple sources and needs cross-validation. The accuracy of such tasks is extremely sensitive to retrieval quality, so fix the retrieval backend and candidate-set size during evaluation.
2.6 Complex Reasoning and Long-Tail Hard Problems
High-difficulty subject Q&A, math competition problems, and logic problems requiring multi-step deduction are used to probe the model’s capability ceiling rather than everyday experience. The problem is high contamination risk: the questions and answers may already be in the training corpus. The credible approach is to build a private question set, or at least rewrite public sets, change the numbers, and rephrase the format to observe whether scores drop significantly.
Dimension Comparison Table
| Dimension | Typical Task Form | Main Bottleneck | Suggested Primary Metric |
|---|---|---|---|
| Repository-level repair | Defect localization and patching | Long-context localization | First-pass rate, regression rate |
| Cross-language engineering | Multi-language repo modification | Build/test identification | End-to-end success rate |
| Terminal tasks | GUI-less environment operation | State awareness and recovery | Interaction turns, destructive-operation rate |
| Output generation | Pages / slides | Layout precision | Rule pass rate + review score |
| Tool calling | Multi-turn tool orchestration | Argument and result usage | Call accuracy, redundancy rate |
| Complex reasoning | Hard subject problems | Data contamination | Private-set accuracy |
Reading the Numbers: Four Easily Overlooked Details
A report’s “improved by X percentage points,” if read out of context, easily leads to the wrong conclusion.
First, look at the baseline level for any gain. Improving several points in the high-score region is far harder than improving the same magnitude in the low-score region. For example, going from 60% to 72% versus from 20% to 32%—the latter looks like the same increment, but the former is already near that benchmark’s practical ceiling.
Second, watch for saturation. Some public benchmarks are near saturation, and score differences between models are compressed within statistical noise. At that point score differences have no discriminative power, so you should switch to a harder self-built task set, or turn to process metrics (interaction turns, token consumption, human-intervention rate).
Third, focus on variance, not a single point. LLM evaluation has significant randomness. Running the same configuration multiple times, the pass rate can fluctuate by several percentage points. Any difference smaller than that fluctuation should not be used as a basis for selection. An illustrative mean and confidence-interval calculation:
import math, statisticsdef summarize(runs: list[float]) -> dict:"""runs: pass rate from each repeated evaluation run (0~1)"""mean = statistics.fmean(runs)if len(runs) < 2:return {"mean": mean, "ci95": None}# 95% confidence interval half-width under normal approximationhalf = 1.96 * statistics.stdev(runs) / math.sqrt(len(runs))return {"mean": mean, "ci95": (mean - half, mean + half), "n": len(runs)}print(summarize([0.712, 0.735, 0.703, 0.728, 0.719]))
A rule-of-thumb practice: repeat each configuration at least 3–5 times, take the mean and report the interval; if two configurations’ intervals overlap, treat them as having no significant difference and instead compare cost and latency.
Fourth, distinguish “capability” from “scaffolding.” The same model, paired with different context-retrieval strategies and different retry and self-verification logic, can have very different end-to-end success rates. Evaluation conclusions must be declared together with the scaffolding, otherwise they aren’t reproducible. This also explains why many people get a better real-world feel outside the leaderboards—they switched to a more suitable agent framework, not a different model.
Building a Layered Evaluation Pipeline
Public benchmarks are for understanding the industry’s overall level horizontally; decisions must rely on your own evaluation set. I recommend a four-layer design, with cost and signal strength increasing per layer.
Layer 1: Unit-test level (seconds of feedback, hundreds of cases). Targets high-frequency, pattern-based tasks such as “give a fix suggestion for this error stack,” “complete unit tests for this code,” and “make this API idempotent.” Answers must be automatically judgeable, and are used for quick regression after each prompt or parameter change.
Layer 2: Repository-level tasks (minutes, dozens of cases). Mine historical issues and corresponding commits from your own codebase to form “problem–patch” pairs. Be careful to do a time-based split so the answer doesn’t appear in the model’s training corpus or the current repository snapshot. Judge with the project’s own tests and record whether regressions were introduced.
Layer 3: End-to-end agent tasks (tens of minutes, a dozen or so cases). Simulate real work tickets: without giving specific file paths, only describe the symptom, and require the model to autonomously retrieve, locate, modify, verify, and output a change summary. Humans only do final acceptance, while interaction turns, token consumption, and the number of human corrections are recorded.
Layer 4: Online canary (continuous). Instrument real traffic to track adoption rate, modification rate (the magnitude of developers’ changes to model output), abandonment rate, and average delivery time. This layer’s signal is closest to business value but needs sufficient sample size and a controlled design.
A skeleton of the evaluation runner is shown below; the focus is on fixing randomness, isolating concurrency, and being replayable:
import json, hashlibfrom concurrent.futures import ThreadPoolExecutordef run_case(case: dict, call_model, scorer, seed: int = 0) -> dict:"""Run a single case: call model -> score -> return structured result"""key = hashlib.sha256(f"{case['id']}|{seed}".encode()).hexdigest()[:12]try:output = call_model(case["prompt"], temperature=0.2, seed=seed)score, detail = scorer(output, case["expected"])return {"id": case["id"], "key": key, "ok": True,"score": score, "detail": detail}except Exception as exc: # record failure instead of aborting the whole runreturn {"id": case["id"], "key": key, "ok": False,"score": 0.0, "detail": repr(exc)}def evaluate(cases: list[dict], call_model, scorer,seed: int = 0, workers: int = 4) -> dict:with ThreadPoolExecutor(max_workers=workers) as pool:results = list(pool.map(lambda c: run_case(c, call_model, scorer, seed), cases))scored = [r["score"] for r in results if r["ok"]]return {"seed": seed,"pass_rate": sum(scored) / len(scored) if scored else 0.0,"error_rate": 1 - len(scored) / max(len(results), 1),"results": results,}
Accompanying engineering recommendations:
- Cases as code: Put the evaluation set under version control, route case changes through review, and ensure historical comparability.
- Replayable results: Save each run’s prompt, parameters, seed, and raw output for later root-cause analysis.
- Process traceability: Write call logs, token usage, and latency distributions uniformly to a logging service, and set up monitoring alerts for automatic notification when the pass rate drops below a threshold or the error rate rises; large-scale concurrent evaluation can scale elastically on a container platform, with cases and artifacts stored in object storage, and case distribution decoupled from result aggregation via a message queue.
- Layered archiving: Automatically route failed cases back as new regression cases, forming a self-reinforcing loop for the evaluation set.
A Checklist of Common Pitfalls
- Looking only at the overall score: An aggregate score masks huge differences across dimensions; always look dimension by dimension.
- Ignoring cost: A solution that gains two points but doubles token consumption and doubles latency is nearly unusable in an IDE-completion scenario.
- No blind testing: If human review doesn’t hide the model source, it will systematically overestimate familiar solutions.
- Contamination unchecked: Using question sets already in the training corpus mistakes memorization for capability.
- Drawing conclusions from a single run: Ignoring variance is the most common and most expensive mistake.
- Testing only the happy path: Differences in real value show up largely in exception branches and error recovery.
Conclusion
The capability gains from version iterations are real, but the role of public evaluations is to be an industry-level reference, not a basis for procurement decisions. What engineering teams truly need is to translate “model capability” into “the success probability, unit cost, and human-intervention rate within their own business.” Break it down across the six dimensions—repository-level repair, cross-language engineering, terminal tasks, output generation, tool calling, and complex reasoning—validate with repeated runs on a self-built layered evaluation set, and put cost, latency, and maintainability on the same scorecard. This methodology won’t become obsolete as models change generations, and each generation will make it more valuable. With the support of a model service platform, this evaluation pipeline can run long-term directly as part of continuous integration, turning selection from a one-time judgment into a continuously calibrated engineering capability.
