Coding Agent Evaluation: A Practical Guide to Capability and Cost-Effectiveness

Coding agents have moved from “completing a few lines of code” to “reading a repository, changing multiple files, running tests, and explaining trade-offs.” Demo videos often show only the happy path. What enterprises actually care about is whether an agent can reliably deliver reviewable changes under no-network, time-limited, and incomplete-requirement conditions, and whether the cost per unit of effective output is acceptable. The method below is not tied to any specific product, so R&D, platform, and procurement teams can use the same language.

Separate the problem from “does it work well?”

“Actual capability” and “cost-effectiveness” are not the same score. Capability answers: under agreed constraints, can it finish the job, and can it self-check when something goes wrong? Cost-effectiveness answers: for those effective results, are the time, invocation fees, human review, and rework worth it? Mixing the two leads to two common misjudgments: an expensive but stable option on a critical path is treated as uneconomical, while a cheap option that hallucinates frequently and requires a senior engineer to clean up line by line is treated as high-value.

It is better to fix four observation categories rather than only asking whether the final result “looks runnable”:

• Task understanding: Does it restate constraints, identify ambiguity, and refuse operations beyond its authority?
• Engineering actions: Does it locate the correct module, keep the change scope minimal, and add tests or provide verifiable steps?
• Factual boundaries: Does it cite sources for versions, interfaces, and knowledge cutoff, and does it clearly say “I don’t know” when it does not know?
• Cost structure: Wall-clock time, interaction rounds, context length, failed retries, and manual revision time.

The test environment must be defined before the candidate

Many comparisons are not comparable because the environment was not locked down. Write an evaluation as a “test environment specification” that includes at least the following fields.

1. Run form: chat completion, repository agent, or tool-enabled automated workflow. Different forms differ by an order of magnitude in capability ceiling and should not be compared directly.
2. Network and materials: no network access at all, internal documentation only, or retrieval allowed. No network access is closer to security-sensitive internal R&D and more likely to expose parameter memory and reasoning rather than retrieval-based stitching.
3. Time box: record per-task limits and overall session limits separately. If a task is not completed within the limit, it should be recorded as incomplete, not marked successful after the examiner gives an oral hint.
4. Permissions: whether it can execute commands, modify tests, or read secrets and production configuration. Evaluations should default to least privilege to avoid mistaking “willing to take risky actions” for capability.
5. Context: whether the repository snapshot, dependency lockfiles, and failure logs are all provided at once. Different information inputs lead to different conclusions.
6. Records: save prompts, tool traces, patches, and final explanations so others can rerun the evaluation.

Record wall-clock time separately. A model that “thinks fast” but goes through ten rounds of clarification is not fast for an on-call engineer. Conversely, spending one or two minutes confirming constraints before acting often shortens total time.

Tasks should cover real development, not stunts

A small sample of about six tasks is enough to expose style, but not enough to declare “comprehensive leadership.” Design tasks by risk layer rather than by adjectives for difficulty.

• Identity and boundary tasks: Ask for the model name, version, and knowledge cutoff. A passing answer should distinguish “information injected by the runtime environment” from “information the model inferred itself,” and acknowledge that memory near the cutoff is sparser. Reciting the system prompt as introspection or inventing a precise cutoff date should count against the agent.
• Local defect repair: A single file with a failing test or a clear error. Check whether the change is minimal and whether the root cause is explained.
• Cross-module change: An interface change affects callers and documentation. Check whether any call sites are missed and whether compatibility is preserved.
• Ambiguous requirement: Give only the business goal. Check whether it lists assumptions and acceptance criteria first instead of immediately piling up code.
• Negative-constraint task: Explicitly prohibit external links, major dependency upgrades, or public interface changes. Violating a constraint is a failure even if the code “runs better.”
• Regression and explanation task: Require an explanation of how to verify the change, possible side effects, and rollback points. A solution that cannot be verified should not count as complete.

Write the scoring rubric for each task in advance, for example: correctness, constraint compliance, change scope, verifiability, and explanation quality, each scored from 0 to 3. Write it in advance to avoid changing standards after seeing the trace. Sample tasks should be de-identified, use public or authorized teaching repositories, and never send customer code into an external evaluation.

How to read a single response

Do not look only at the final summary. Check the trace:

• Did it list constraints before acting? Under no-network conditions, still trying to visit external links in the task indicates unstable tool policy or instruction following.
• Did it present uncertainty as certainty? For interface fields, error codes, or performance numbers, giving specific values without evidence should be recorded as hallucination, not “confident style.”
• Are the changes reviewable? Large unrelated reordering, incidental formatting changes, or deleting tests to make commands pass are high-risk signals.
• Did it change strategy after failure? Repeating the same command meaninglessly usually indicates a lack of diagnosis, not “persistence.”
• Is the explanation handoff-ready? A passing explanation includes what changed, why, how to verify it, and what risks remain.

A knowledge cutoff itself is not a defect. The defect is describing ecosystem changes after the cutoff as if they were firsthand facts. A reliable agent should note that “current dependency documentation and locked versions should govern,” and leave verification to repeatable commands rather than verbal compatibility guarantees.

A simple record can use plain fields:

Task ID, constraints (no network/time limit/permissions), whether it clarified, number of patched files, whether agreed checks passed, minutes of manual revision, primary failure mode (misunderstood requirement, missed call site, invented interface, violated constraint, timeout).

Failure modes have more management value than average scores. If a task category fails consistently, do not put that agent behind the corresponding gate.

Cost-effectiveness should be calculated from effective output

Pricing is usually per token, per seat, or per task, but R&D cost is mainly human. A simple effective cost formula is:

Effective cost ≈ invocation fees + opportunity cost of human time spent waiting + review and rework time × reviewer hourly rate + cost of fixing escaped defects (if quantifiable).

When comparing, use the same task set and the same acceptance standard, and calculate “independently passed tasks / effective cost,” not “lines of generated code / cost.” More lines are often more expensive and harder to review. Also look at variance: an option that is fast and cheap on average but occasionally makes an irreversible mistake in permissions or dependency upgrades will have its expected cost pulled up by the long tail.

It is better to provide three usage tiers than a single overall score:

• Suitable for automation: low-risk, testable, and rollback-safe, such as internal scripts, boilerplate code, and test completion.
• Suitable as a copilot: medium-risk changes that require human patch review, such as small business-logic changes.
• Not suitable for unattended operation: privilege escalation, data migration, security policy, and external interface compatibility. Even if a demo succeeds, these tasks should not be decided on unit price alone.

Cost also grows with context. Dumping an entire monorepo into the conversation increases both expense and distraction. In practice, limit directories, let a human or retrieval step narrow the scope first, and then let the agent modify. This is process design, not model mysticism.

Integrate evaluation into rollout, not just rankings

A small sample of several dozen minutes is suitable for answering “is it worth entering a pilot?” It is not suitable for an absolute public ranking. During the pilot:

• Fix the question bank and golden patches, and rerun monthly to observe degradation rather than testing only on release day.
• Use separate accounts and repository permissions for production use and evaluation; evaluations must not touch real secrets.
• Gates should depend on tests, static checks, and human approval, not on the model’s self-report that it “completed” the task.
• Track adoption rate: the proportion of generated patches merged as-is, merged after small edits, and discarded. When the discard rate is high, even a low unit price is not worthwhile.
• For framework tasks sensitive to knowledge freshness, write expectations against locked dependency versions to avoid replacing “correct” with “new.”

During team retrospectives, discuss only failure modes and constraints, not brand preferences. If switching vendors, use the same test environment specification and question bank so conclusions remain comparable. Avoid citing irreproducible benchmark scores in procurement materials; keep trace hashes, task versions, and time boxes in internal documentation for audit.

Common pitfalls

Treating fluent answers to identity questions as strong capability. Identity and cutoff dates only show instruction following and honesty; they do not substitute for fixing defects.

Testing only online retrieval scenarios but applying the conclusions to air-gapped R&D. Successful retrieval masks weaknesses in parameter memory and reasoning.

Evaluating with a single total duration. Separate clarification, execution, and rework. Clarification often saves rework.

Rewarding “writing more.” In cost-effectiveness, long patches that cannot be merged are liabilities.

Using public leaderboards instead of tasks from your own repository. The same language and framework do not mean the same business constraints. Leaderboards can inspire task types but cannot replace your acceptance criteria.

Ignoring reviewer level. When junior engineers cannot detect invented interfaces, the real cost of a low-price option is underestimated.

Conclusion

To evaluate coding agents, first lock the environment and constraints, then score task understanding, minimal correct changes, verifiability, and honest boundaries, and finally compare cost-effectiveness using effective cost rather than list price. A small sample can decide whether to pilot; a sustained question bank, adoption rate, and long-tail incidents decide whether to expand. Only by narrowing “can write code” to “delivers reviewable changes under permission, time, and evidence constraints” will teams avoid being led by a single demo.