How to Evaluate AI Coding Agents for Real Workflow Capabilities

1. First, Distinguish Two Capabilities: Answering Questions and Completing Work

When evaluating a coding agent, the most common mistake is equating model capability with agent capability. Model capability measures the quality of understanding and generation in a single turn. Agent capability measures the overall ability to complete a task in a real environment.

The gap between the two is concentrated in four areas:

• State awareness: A model mainly processes contextual text, while an agent must read the actual state of external systems, such as installed components, granted permissions, and existing resources.
• Side-effect management: A response is idempotent, while every agent action may actually create, modify, or delete live data.
• Failure recovery: A response does not really “fail,” but an agent must diagnose the cause after a tool error and decide what to do next.
• Termination judgment: A model stops after producing an answer, while an agent must decide whether the task is complete or whether it must hand control back to a human.

Therefore, a model that performs well on exercises may still fail immediately inside an agent framework after a single authentication error. The key to judging an agent is not how polished a single output is, but how far the agent can go when things deviate from expectations.

2. Task Selection: Which Tasks Can Reveal Real Capability

Evaluation starts with task design. If the wrong task is chosen, all later conclusions are invalid. An effective agent evaluation task should satisfy four conditions.

Cross-system. The task must cross the boundary of systems outside the agent, such as reading and writing between a local environment and a remote service. Purely local code-generation tasks rarely expose defects in tool invocation.

External state. The task must depend on preparatory state, such as whether a component is installed, whether credentials are valid, and whether permissions are enabled. Only the presence of external state triggers the agent’s environment inspection and decision logic.

Automatically verifiable. Success or failure must be judged by a script after the task is completed, not by a human looking at whether the output seems reasonable. For example, the test can verify whether a new document record actually exists on the remote system and whether its title and body match expectations.

Failure allowed with a recovery path. The ideal task is one that almost certainly fails on the first attempt but has a clear troubleshooting path after failure. This kind of task is the real dividing line for agent capability.

A typical example is a command-line tool that connects to an external document system, reads a specified document, and then creates a new document. It naturally covers five stages: component installation guidance, account authorization, permission determination, reading, and writing. Any step that fails leaves an observable error message.

3. Capability Dimensions: Where the Gap Appears

In a real task observation, even an agent with a good capability rating can show obvious differences across layers. Breaking the observation into evaluable dimensions makes the conclusions more useful.

1. Guidance for Environment and Prerequisites

When an agent detects that a tool is missing, it should provide actionable installation or integration guidance instead of returning a vague error. Excellent behavior is to clearly explain which CLI tool needs to be integrated, which authentication action should be performed, and how to verify authentication after it is completed. Passing behavior is to ask the user to finish the missing setup manually and then continue. Failing behavior is to return an environment-unrelated error code and leave the user to guess what happened.

2. Multi-Layer Judgment in Authentication

This is where agents most easily lose points. A seemingly simple “insufficient permission” error may have multiple possible causes:

Missing local credentials → Expired token → Application not authorized for the target scope → Resource-level access policy not active → Cached old token not invalidated

Any layer above can return a similar error message. An agent that can distinguish between application-layer permission issues and resource-layer policy issues will actively query authorization status, check token validity, and prompt the user to confirm which layer is misconfigured. An agent without this capability collapses a multi-layer problem into a single “permission denied” message and stops there.

3. Fault Localization and Error Semantics

The key is not whether the agent can read an error, but whether it can translate the error into an action. The evaluation standard is what the agent chooses to do next when facing an error.

Error Type	Expected Action	Anti-Pattern Action
Authentication expired	Re-initiate authorization	Directly declare the task failed
Permission not effective	Explain the reason and request a retry	Terminate without any explanation
Invalid parameters	Fix the parameters and retry	Repeatedly retry with the same parameters
Resource already exists	Switch to an idempotent strategy or rename	Overwrite the resource and cause side effects

4. Autonomous Progress and Loop Termination

The value of an agent lies in completing a closed loop. The ideal behavior is an autonomous loop of “detect → judge → repair → retry → verify → finish.” In practice, a common failure mode is premature termination: the agent gives up after one failure and shifts diagnostic work that should be handled by the agent back to the user.

Autonomous progress does not mean infinite retries. A mature implementation needs a retry limit, a backoff strategy, and an escalation path. When a threshold is exceeded, the agent should proactively and clearly request human intervention and explain what it has already tried.

5. Human Intervention Cost

This is the metric closest to the real user experience and also the easiest to overlook. Even if the task eventually succeeds, that does not mean the agent is acceptable. What needs to be quantified is: how many times the user was pulled into the process, how long each intervention lasted, and how much specialized judgment the user had to provide.

If the user must log in to a console, understand the permission model, and manually return to the conversation to ask the agent to retry, then the task is actually being completed by the human. The agent only handles the typing.

4. Evaluation Design: Turning Subjective Experience into Data

1. Control Variables

The biggest risk in experience comparison is poor reproducibility. At least the following variables should be fixed and explicitly recorded: agent version and model version, operating system and runtime version, tool installation method, account type and role, permission configuration of the accessed resource, network egress, and task starting state, such as whether the environment is clean.

Any change to these variables requires rerunning the baseline. Otherwise, cross-version conclusions are not comparable. The recommended approach is to write the environment as a declarative configuration and archive it together with the evaluation results.

# eval-env.yaml
agent:
  version: "x.y.z"
  model: "model-id"
runtime:
  os: "linux"
  sandbox: true
toolchain:
  cli: { installed: false, authenticated: false }
account:
  role: "editor"
  scope: ["doc:read", "doc:write"]
network:
  egress: "restricted"
task:
  initial_state: "clean"

2. Metric Definitions

At least six metrics should be collected. The first four are hard metrics.

• First-attempt success rate: The proportion of tasks completed without human intervention.
• No-intervention completion rate / average number of interventions: The frequency of human prompts required for a successful task. This directly reflects the real cost.
• Recovery steps: The number of tool-call rounds from the first error to task success. This measures diagnostic efficiency.
• Side-effect correctness: Whether destructive operations occur, such as duplicate creation, unauthorized writes, or overwriting existing resources.
• End-to-end latency and token consumption: Cost and latency constraints.
• Escalation quality: When the agent decides to hand control back to a human, whether it provides enough information to locate the problem.

3. Minimal Evaluation Framework

A lightweight harness only needs three things: reset the environment, drive execution, and verify the result. A conceptual implementation is as follows:

from dataclasses import dataclass, field
@dataclass
class TaskResult:
    task_id: str
    success: bool
    human_interventions: int          # Number of human prompts
    recovery_steps: int               # Tool-call rounds after the first error
    unsafe_side_effects: list         # Records of destructive operations
    raw_trace: list = field(default_factory=list)
def run_task(env, agent, task) -> TaskResult:
    env.reset(task.initial_state)      # Important: start from a known state every time
    trace = []
    interventions = 0
    recovery = 0
    result = agent.execute(task.instruction, tools=env.tools)
    for step in result.steps:
        trace.append(step.to_dict())
        if step.kind == "human_prompt":
            interventions += 1
            result = agent.resume(result, answer=env.scripted_answer(step))
        if step.kind == "tool_error":
            recovery += 1
    ok = task.verify(env)              # Use objective assertions to determine success
    return TaskResult(
        task_id=task.id,
        success=ok,
        human_interventions=interventions,
        recovery_steps=recovery,
        unsafe_side_effects=env.detect_unsafe(result),
        raw_trace=trace,
    )

The evaluation task set should be organized into three categories: golden path, where everything is ready and the task should pass on the first attempt; exception path, where permissions are missing, tokens have expired, or the target already exists; and destructive scenarios, where write behavior is verified when resources already exist, testing idempotency and rollback. Exception paths should have higher weight than golden paths because they reveal the real boundary of agent capability.

4. Scoring Rubric

Dimension	Maximum Score	Key Criteria
Task completion	30	Whether objective assertions are met and whether the result is verifiable
Autonomy	25	Number of interventions, recovery steps, and whether the agent terminates prematurely
Diagnostic quality	20	Whether the agent identifies the correct layer of the root cause
Safety	15	Least privilege, idempotent writes, and no destructive side effects
Observability	10	Whether the trace is complete and whether escalation information is actionable

5. Engineering Implementation: Permissions, Safety, and Observability

Beyond evaluation, connecting an agent to a real workflow requires several hard constraints. Without them, a single mistake can cause irreversible damage.

Least-privilege authorization. Grant the narrowest scope required by the task. For example, grant read and write access only to the target directory rather than full account-level access. Permissions should be revocable at any time.

Temporary credentials and tiered authorization. Prefer short-lived credentials. High-risk operations, such as deletion, overwriting, bulk writes, or external publishing, should require explicit confirmation or a separate approval step.

Pre-write validation. For creation tasks, first check whether the resource exists and whether there are naming conflicts. Use an idempotent strategy such as “skip if it exists or rename,” and validate the content structure before submitting a write.

Sandbox and network egress control. Restrict the agent’s reachable scope so that it does not make unintended external accesses while troubleshooting.

Full-chain trace logging. Every tool call, parameter, return value, and error code should be recorded completely. Trace data can be archived to object storage, searched through a log service, and monitored with alerts for sudden spikes in error rates, rising average recovery steps, and other trend metrics. This way, when an agent’s performance degrades after a new version is released, you can discover it from data immediately instead of waiting for user complaints.

6. Conclusion: Treat “How Many Human Reminders Are Needed” as a Core Metric

Back to the original judgment: if an agent can complete a real task, that only means it has crossed the threshold of usability. Whether it can autonomously diagnose and recover when an exception occurs determines whether it truly reduces the total cost of the workflow.

The recommendations for practitioners can be summarized in three points:

1. Evaluate with real tasks rather than exercises. Tasks must cross systems, involve external state, and be automatically verifiable.
2. Include the number of human interventions and recovery steps as mandatory metrics. They reflect actual experience more accurately than “task success or failure” alone.
3. Control environmental variables and archive complete traces. Make evaluation conclusions reproducible, regression-testable, and traceable.

Model capability determines the upper limit. Engineering capabilities such as tool invocation, exception judgment, and autonomous progress determine the lower limit. When selecting tools for enterprise use, the lower limit is often more important than the upper limit, because what shapes the daily experience of a team is often those moments when someone has to stand by and click “Retry.”