Agentic Coding Evaluation: From Benchmark Scores to Real Delivery
A practical methodology for evaluating agentic coding models. It clarifies what "agentic coding" actually means across three capability levels, explains what public benchmarks can and cannot tell you, and shows how to build an evaluation set drawn from your own repositories. It covers turning subjective impressions into measurable metrics, four scenario-based test cases (structured graphic generation, front-end components, terminal/scripting tasks, and long refactoring jobs), and the engineering scaffolding that turns high scores into stable delivery—context engineering, automated verification loops, sandboxing with least privilege, model routing, and observability. It closes with a reusable evaluation report template.
1. Aligning on the concept: what counts as agentic coding
Asking whether a model “can write code” and asking whether it “can get the job done” are two different questions. The first is a subset of text-generation ability; the second requires a full engineering loop. The field generally describes the evolution of coding ability in three levels:
- Level 1: Code completion. The input is a fragment of the current file and the output is the next few lines. It is judged on latency and acceptance rate, and the cost of a mistake is low.
- Level 2: Conversational coding. The input is a natural-language request plus partial context, and the output is a full piece of code or a change suggestion. The human still handles decomposition, locating the right spot, pasting, and verification.
- Level 3: Autonomous agents (agentic). The input is a goal-level task—for example, “add pagination to this service and align it with the existing error-code conventions.” The model decomposes it into subtasks on its own, retrieves relevant files, calls tools multiple times, runs commands, reads the results, and self-corrects based on errors before delivering a set of verifiable changes.
The third level demands a qualitative leap because its failure modes differ from the first two. When levels 1 and 2 go wrong, a human spots it at a glance. When level 3 goes wrong, it may show up as code that looks plausible but depends on something that does not exist, or as a silent over-edit. So the model capabilities that support autonomous agents should be broken into at least four separately measurable dimensions:
- Effective use of long context. The advertised context length only decides whether content fits—not whether it is used correctly. In a real repository, relevant clues are often scattered across a few fragments in many files, and the model must locate the key function among tens of thousands of lines while suppressing irrelevant noise. What engineering should measure is the effective context length, not the number on a spec sheet.
- Stability and structural consistency of long outputs. An autonomous task often needs to produce multi-file changes in one shot. The longer the output, the more likely you are to see structural drift in the middle and later sections, inconsistent naming, and premature wrapping up. Maximum output length is a hard constraint that determines whether a task is “finished in one pass” or “must be produced in chunks.”
- Controllability of thinking (reasoning) modes. Complex refactoring and defect localization benefit from point-to-point reasoning budgets, while turning on deep thinking for a simple change only adds latency and cost. Evaluation should separate performance with the mode on versus off, rather than reporting a single blended result.
- Format reliability of tool calls. This is the most easily overlooked weakness of autonomous agents. The model must reliably produce schema-compliant structured calls, fill in parameters correctly, read return values correctly, and fix errors instead of repeating the same action. One format error breaks the entire chain right there.
2. What public benchmarks can and cannot tell you
Mainstream evaluation systems fall into roughly seven categories, each measuring a different part of the pipeline:
| Evaluation category | What it measures | Typical blind spot |
|---|---|---|
| Function-level code generation | Correctness of a single function | Independent of repository context |
| Repository-level bug fixing | Cross-file localization and repair | Depends on test scaffolding quality |
| Multi-language coverage | Syntax and ecosystem fluency in non-mainstream languages | Uneven difficulty across languages |
| Terminal command execution | Command-line operations and troubleshooting | Environment differences have a large impact |
| Tool calling and dialogue state | Consistency of multi-turn structured calls | Metrics are sensitive to format details |
| Web/browser tasks | Visual and interaction goal completion | Page version drift |
| Hard general-knowledge Q&A | Problem understanding and domain knowledge | Weakly related to engineering ability |
What you need to be clear-eyed about is that the same model often ranks differently across leaderboards. This is not data fraud—it reflects the limits of the evaluation methods themselves:
- Data contamination. The probability of public problems appearing in pre-training corpora is uncontrollable. When training data overlaps with the evaluation set, the score reflects memorization rather than generalization.
- Scaffolding differences. Prompt templates, retrieval strategies, maximum turn counts, retry mechanisms, and sandbox image versions all significantly affect pass rates on repository-level tasks. The same model with a different scaffold can shift by a dozen percentage points or more.
- Different metric definitions. First-attempt pass rate (pass@1) and best-of-multiple-samples (pass@k) measure different abilities; unit-test pass rate and human-review pass rate are not the same thing either. Comparing scores across leaderboards often means comparing different definitions.
- Missing cost and latency. A task that passes after 80 tool-call turns and millions of tokens carries entirely different business value than one that passes in 12 turns.
- License and availability. Whether the weights are open and whether the license is permissive or copyleft directly determines whether it can be used commercially, deployed privately, or fine-tuned for a domain. This is a first-order filter during selection, not an afterthought.
Conclusion: use benchmarks to shortlist, use hands-on testing to choose. Public scores are good for narrowing the candidate pool from twenty to three, but the final decision must rest on reproducible testing on your own business corpus.
3. Building your own evaluation set: let the tasks grow from your own repositories
3.1 Sampling principles
The value of an evaluation set depends on how similar it is to your business. It is recommended to sample in layers along these dimensions, with no fewer than 20 tasks per category to ensure statistical significance:
- Language and framework coverage: Weight by share of lines of code; do not just pick the languages you are familiar with.
- Task types: New features, bug fixes, refactoring, writing tests, writing docs, dependency upgrades, and configuration changes.
- Difficulty: Local single-file changes, cross-module changes, and cross-service changes.
- Constraint intensity: Whether the task requires following existing naming conventions, error-code conventions, and compatibility requirements.
3.2 The four elements of a task definition
Every task must include: the context given to the model (not the entire repository), automatically determinable acceptance criteria, an isolated execution environment, and a clear scoring function. Tasks without acceptance criteria should not enter the evaluation set, or you will end up with subjective scoring.
from dataclasses import dataclassfrom typing import Callable@dataclassclass Task:task_id: strcategory: str # feature / bugfix / refactor / testprompt: str # Goal-level description shown to the modelcontext_files: list[str] # Whitelist of files allowed to be injectedsetup_cmd: str # Environment setup, e.g., dependency installationverify_cmd: str # Verification command; exit code 0 means passbudget: dict # {"max_turns": 30, "max_tokens": 400_000}rubric: dict # Structured scoring items, e.g., {"diff_lines_max": 200}
3.3 A layered evaluation structure
- L1 unit-level: Implement or fix a single function, with second-level feedback, used for fast regression.
- L2 module-level: Changes spanning 2–5 files, testing interface consistency.
- L3 repository-level: Requiring localization and modification across multiple directories, testing retrieval and planning ability.
- L4 end-to-end delivery: Including build, unit tests, static checks, and commit-message generation, testing the full loop.
3.4 Execution protocol
The credibility of an evaluation depends on whether execution is controlled. You must fix the following variables: temperature and sampling parameters, maximum turn count, timeout, token budget, sandbox image version, and dependency lockfile. Sample each task 3–5 times, so you get both the mean and the variance. A model with high variance carries greater risk in real business, even if the average score is the same. Also clean the workspace before each run to avoid context contamination and cache reuse across models.
4. Turning “feel” into metrics
The impression gained from running a few polished demos has no decision value. It is recommended to fix the following metrics:
- Task success rate (Pass@1): The share of tasks that pass the verification command, broken down by difficulty level.
- Average interaction turns and within-turn convergence: Too many turns means inefficient planning; repeatedly calling the same tool means being stuck in a loop.
- Tokens and cost per task: Including accumulated input, output, and tool return values, with cache-hit portions separated out.
- Change size and regression risk: Average lines changed, number of files touched, and whether files outside the task scope were modified.
- Manual rework rate: The gap between first-pass rate and final adoption rate directly reflects the “time saved.”
- Constraint-adherence rate: Whether explicit conventions and “do not change X” prohibitions were respected.
- Hallucinated-interface rate: The proportion of calls to functions, parameters, or config options that do not exist—the most dangerous failure type for autonomous agents.
- Dangerous-operation interception rate: Whether the model correctly refuses or asks for confirmation when facing requests to delete data, force-push a branch, or change permissions.
Failure modes should also be categorized and counted, rather than recorded as a single failure tally: incomplete structure, missing dependencies, ignored explicit constraints, out-of-scope edits, format errors, non-converging loops, and over-refusal. The categorization determines the direction of optimization—the first two point to context and retrieval, the fifth to schema constraints and retry mechanisms, and the last to an overly tight configuration of the safety policy.
5. Four scenario-based test cases
5.1 Structured graphic generation: check completeness, not just whether it looks good
Using vector-graphics (SVG) generation tasks to test a model’s coordinate reasoning and structural organization is very effective, because it is pure text output, can be automatically verified, and has a clear visual result. Evaluation should have three levels: the syntax is parseable, the element inventory is complete, and the visual result is reasonable.
import reimport xml.etree.ElementTree as ETREQUIRED_PARTS = ["body", "head", "beak", "wheel_left", "wheel_right", "crank"]def check_svg(src: str) -> dict:root = ET.fromstring(src) # Level 1: parseabilityids = {el.get("id") for el in root.iter() if el.get("id")}missing = [p for p in REQUIRED_PARTS if p not in ids]wheels = [el for el in root.iter() if (el.get("id") or "").startswith("wheel")]xs = [float(re.sub(r"[^\d.\-]", "", el.get("cx") or el.get("x") or "0")) for el in wheels]return {"parseable": True,"missing_parts": missing, # Level 2: element completeness"wheel_positions_sane": len(set(xs)) > 1, # Level 3: geometric sanity"element_count": sum(1 for _ in root.iter()),}
The value of this approach is that it turns subjective aesthetics into regression-checkable items. When tasks of this kind recur, “a missing part,” “overlapping element positions,” and “scattered parts” are the three most common defect types, and all of them can be captured by rules.
5.2 Front-end components and status cards
Generating a card component with four weather states tests state coverage and engineering discipline rather than drawing skill. The acceptance checklist should include: whether all four states are implemented; whether the animation degrades correctly under the prefers-reduced-motion setting; whether it overflows on narrow screens; whether text has associated labels to ensure accessibility; and whether styles are scoped to the component without polluting the global scope. To check this automatically: take screenshots in each of the four states and compare pixel differences to confirm the state switching actually takes effect, then run a style-isolation check to confirm no global selectors leak.
5.3 Terminal and scripting tasks
Command-line tasks most easily expose a model’s “happy-path hallucination.” Ask the model to write a deployment script, with these acceptance points: whether repeated runs are idempotent; whether a mid-way failure leaves a half-finished state; whether critical steps check exit codes; whether high-risk commands (recursive deletion, force push, permission changes) are guarded by confirmation or protection; and whether the output logs make problems locatable. Be sure to test negative cases too—deliberately create insufficient permissions or a missing dependency to see whether the model self-heals or masks the error.
5.4 Refactoring and migration long tasks
Ask the model to convert a module from callback style to async style, or to migrate config options from an old format to a new one. Success on tasks like this must be backstopped by the existing test suite: success only counts when all tests pass. Also track whether the diff size has spiraled out of control, whether unrelated formatting was changed along the way, and whether naming is consistent across files. Long tasks best reveal effective use of long context, because they require the model to keep remembering the original constraints over dozens of turns.
6. Engineering the rollout: turning high scores into stable gains
Model capability is only one variable; the scaffold determines the ceiling. The following five investments yield the highest returns in practice.
Context engineering matters more than switching models. Change “throw the whole repository at the model” to “retrieve + targeted injection”: first recall candidate files with keyword and symbol indexes, then trim along dependency relationships down to the relevant functions and interface definitions, while providing a directory-structure overview and key convention docs. The cleaner the context, the lower the hallucination rate. For tasks with very long outputs, enforce chunking: first produce a change plan and file list, then generate file by file, to avoid the mid-to-late degradation caused by a single long output.
Build an automatic verification loop. Let tests, static checks, type checks, and the build act as an automatic reviewer, feeding structured failure information back to the model to retry—but set an upper bound on turns (generally 3–5), or it easily falls into an unproductive loop and burns through the budget.
for attempt in range(MAX_RETRY):patch = agent.propose(plan, context)result = run(setup + verify + lint, sandbox=env) # execute in an isolated environmentif result.passed:return patch, attemptcontext = append_feedback(context, result.stderr) # feed back only the relevant error snippetsraise BudgetExceeded(attempt=MAX_RETRY)
Sandbox isolation and least privilege are must-haves, not nice-to-haves. An agent has the ability to run arbitrary commands, which means a single prompt injection or a single misjudgment can cause real damage. Baseline requirements: run all commands in a disposable container with no outbound network or only a whitelist of outbound traffic; inject no long-term credentials into the container, and route any required temporary credentials through short-lived tokens; snapshot the workspace so you can roll back; force human confirmation for high-risk actions like deletion, push, permission changes, and production-config edits; and keep full command-level audit logs. Without these constraints, you should not connect an agent to a real repository or production environment.
Use model-tiered routing to control cost and latency. Split tasks by complexity: formatting, test augmentation, and comment generation go to smaller models or templates; cross-file refactoring and defect localization go to flagship models; simple Q&A goes to cache. Combined with prompt and result caching, batching, and timeout and budget caps, this usually brings average cost down by an order of magnitude while preserving success rate. When evaluating, measure cost under the same routing strategy, not by comparing single models run bare.
Observability and canary rollout. Bring every agent run into your logging service as a traceable record, capturing task category, turns, tokens, latency, verification result, and whether human rework was needed; use monitoring alerts to set thresholds for drops in success rate, abnormal spikes in turns, and cost jumps. When a new model version launches, canary it on a small share of traffic first, and compare core metrics against the same batch of tasks before rolling out fully. These logging, container, storage, and monitoring-and-alerting capabilities fall under general cloud-product categories on Baidu AI Cloud and can be combined as needed to carry the pipeline above, without binding to a specific model.
7. A reusable evaluation report template
When reporting externally or deciding internally, it is recommended to fix seven items: the composition and sampling method of the evaluation set; the candidate models and inference configuration (including the thinking-mode toggle); the execution protocol and fixed parameters; a layered success-rate table (L1–L4); cost and latency percentiles (P50/P95); the failure-mode distribution; and license and deployment constraints. Add one more column—“manual rework rate”—and this report is enough to support a selection decision.
8. Conclusion
Marketing around coding agents often focuses on leaderboard rankings, but what actually determines business gains is three things: whether task definitions and acceptance criteria are clear, whether context and verification loops are engineered, and whether safety and cost boundaries are controllable. Treat evaluation as a one-off demo and all you get is an impression; build it into a sustainable pipeline and you get an efficiency curve that converges over time. Build the evaluation set first, then talk about selection; set up the sandbox first, then grant permissions—get the order wrong, and no matter how high the score, it will not convert into stable delivery capability.
