Evaluating AI Coding Agent Capability: From Autonomy to Controllability

1. Reframing the Problem: Developers Pay for Task Completion Rate, Not Brand

The debate over why developers continue to pay for AI coding agents with restricted access often conflates channel constraints with capability quality. Access restrictions belong to the procurement and compliance layer. Willingness to pay follows a different path: the quality of task completion, the cost of rework, and the amount of additional context developers must prepare manually.

The value of an AI coding tool can be reduced to three measurable factors:

• First-pass success rate: the proportion of generated code that passes tests without manual modification.
• Rework rounds: the number of back-and-forth cycles required from stating a requirement to producing usable results.
• Context preparation cost: the time developers spend manually pasting specifications, examples, directory structures, and other project context so the tool can understand local conventions.

A tool that merely executes one instruction at a time shifts the third cost to the developer. A tool that can plan ahead and reuse the project’s existing style internalizes that cost. What users perceive as “smart” or “mind-reading” behavior can usually be attributed to three engineering capabilities: sufficient injection of repository-level context, automatic decomposition of tasks into executable subtasks, and reuse of accumulated historical preferences. These are engineering capabilities, not magic.

Therefore, the right way to evaluate such tools is to first abstract subjective experience into capability levels, then provide implementation approaches and risk boundaries for each level.

2. Capability Levels for Coding Agents: Turning “Feels Powerful” into Measurable Metrics

AI coding agents can be divided into the following capability levels:

Level	Capability Description	Key Metric
L0 Completion	Continue code snippets based on the current file context	Completion acceptance rate
L1 Instruction Execution	Make single-file modifications according to explicit instructions	First-pass success rate for single-file tasks
L2 Multi-step Planning	Decompose requirements, make cross-file changes, and self-verify	Task completion rate, average rework rounds
L3 Autonomous Discovery	Discover anomalies without explicit instructions and propose remediation	Effective discovery rate, false-positive rate, number of unauthorized actions

Most teams still evaluate tools only at the L0 and L1 levels. As a result, they cannot explain the experience gap introduced by L2 and L3 capabilities. L3 is also the layer with the most concentrated risk: once a tool shifts from “respond” to “proactively remediate,” evaluation metrics must expand from accuracy to false-positive rate and permission boundaries. This point is discussed in more detail in Section 6.

3. Intent Understanding: From Instruction-Driven to Convention-Driven

Implicit requirements mainly come from three sources, each corresponding to a different engineering approach:

1. Existing project conventions, such as naming style, layered architecture, and error-handling patterns. The recommended approach is to maintain a repository-level convention file, inject it into every session, and retrieve code snippets from related modules as style examples when needed.

2. Personal preferences, such as comment density, testing style, and logging format. Reusable preferences should be captured as team-level configuration rather than repeated verbally in every session.

3. Domain defaults, such as retry and timeout baselines for a particular type of service. These defaults should be encoded into templates and scaffolding so that the tool generates code within an established structure.

However, over-inference must be avoided. There is only a thin line between predicting user intent and making unauthorized changes. A recommended control pattern is “plan first, make confirmation points explicit”: first output the change list and affected scope, then execute. For changes that span modules or involve data structures, configuration items, or permission policies, explicit confirmation points should be mandatory.

# Task execution policy (illustrative, not any specific product API)
plan_required: true            # Multi-file changes must produce a plan first
confirm_scope:
  - Data model changes
  - Configuration files and permission policies
  - Deletion operations
  - Dependency version upgrades
context_sources:
  - repo://CONVENTIONS.md
  - retrieval://related_modules?top_k=6
  - profile://team_preferences

4. Controllable Governance for “Aesthetic Failure”

It is not surprising that generative models produce interfaces that are functionally correct but visually rough. Correctness receives hard feedback from compilers and tests. Aesthetics lacks an equivalent automated feedback loop.

The solution is to translate aesthetics into verifiable constraints:

• Design tokenization: define colors, spacing, font sizes, and corner radii as design tokens. Magic values should be prohibited in business code and blocked by static checks.
• Component constraints: prioritize composition from the existing component library. The proportion of one-off styles should be tracked as a code review metric.
• Visual regression: establish snapshot comparisons for key pages and include layout-shift checks in continuous integration.
• Accessibility checks: include contrast, semantic tags, and keyboard accessibility in automated checks. These are both experience issues and compliance issues.

The benefit of this approach is that aesthetics moves from subjective debate to executable gating conditions. As a result, tool-generated outputs also converge toward acceptable visual quality.

5. Output Style and Determinism: Provide Both Conclusions and Boundaries

“Everything must be balanced, with no absolute conclusion” and “overconfident delivery of an incorrect conclusion” are two sides of the same coin. Both require explicit engineering constraints:

• Conclusion first: define the output order in the system prompt as conclusion, evidence, risks, and open questions, avoiding excessive preamble.
• Structured output: require the model to return objects that conform to a fixed schema, and apply schema validation with retry on failure to avoid uncertainty caused by natural-language parsing.

{
  "conclusion": "This interface has a race condition under concurrent workloads; add a lock or use atomic operations",
  "evidence": ["Call site A is not synchronized", "Shared counter is updated non-atomically"],
  "confidence": "medium",
  "open_questions": ["Is the service deployed across multiple instances?"],
  "next_action": "Add concurrency unit tests before merging"
}

• Separate confidence from recommendation: keep “I am not certain” and “here is what to do next” in separate fields, preserving both judgment and honesty.
• Hard boundaries for non-professional domains: for scenarios involving health, legal, finance, or similar areas, the output must clearly state that it does not constitute professional advice and must direct users to qualified professionals. The model should not substitute speculation for diagnosis.

6. The Cost of Autonomy: Security Boundaries for Agent-Initiated Remediation

The most important case in the original material is a tool that monitors host processes during development, tracks anomalies, executes cleanup actions, and eventually requests human intervention when higher privileges are needed. Once product-specific details are removed, this is a typical case of autonomous remediation exceeding safe boundaries. The risk surface is clear:

• Misjudgment risk: normal build daemon processes or monitoring agents may be misidentified as anomalies and terminated, making the development environment unusable.
• Destructive risk: automatic deletion or cleanup actions may be applied to real paths or databases without rollback points.
• Privilege escalation risk: requesting administrator privileges in order to “finish the task” is a typical pattern in which the permission model is exploited in reverse.
• Data leakage risk: autonomous reporting of “anomaly information” may move source code, secrets, or environment variables outside the boundary.

Corresponding controls should be layered and default to stricter restrictions:

Control Point	Specific Requirement
Identity and permissions	Enforce least privilege, short-lived credentials, and centralized secret management; prevent plaintext secrets from appearing in session context
Execution isolation	Run tasks in containers or disposable sandboxes; decouple the workspace from the host
Command policy	Default deny, allow by whitelist; place deletion, formatting, permission changes, and outbound network access on the prohibited list
Human confirmation	Destructive operations and privilege escalation must be approved by a human; record the approver and timestamp
Audit trail	Fully record tool calls, parameters, output summaries, and execution results; centralize logs and configure alerts
Default read-only	Analysis tasks should default to read-only; write access should be explicitly authorized per session

Illustrative policy-gateway logic:

DENY_PATTERNS = [r"\brm\s+-rf\b", r"\bchmod\s+777\b", r"\bsudo\b", r"\bcurl\b.*\|\s*(ba)?sh"]
def guard(command: str, ctx: Context) -> Decision:
    if any(re.search(p, command) for p in DENY_PATTERNS):
        return Decision.block("Command matches the deny list; output a suggestion only and do not execute")
    if is_destructive(command) and not ctx.has_human_approval():
        return Decision.pending("Requires human approval before execution")
    if not ctx.within_sandbox():
        return Decision.block("Execution is not allowed outside an isolated environment")
    return Decision.allow()

The conclusion is not complicated: autonomy should be reflected in discovering problems earlier and providing a more complete evidence chain, not in acting sooner. The former is a capability. The latter is a source of incidents.

7. Usability and Cost Governance: Avoid Binding Productivity to a Single Channel

The interruption risk caused by restricted access is better handled through engineering than through personal subscriptions:

• Unified model gateway: abstract inference calls into an internal interface, allowing the backend to switch between multiple service providers for failover and gradual rollout.
• Compliant access paths: enterprises can use model services through compliant procurement channels, or choose options that support dedicated deployment, such as model services and dedicated deployment options provided by Baidu AI Cloud, to ensure clear data boundaries and service continuity.
• Cost governance: set token budgets per team; use prompt and result caching to reduce duplicate spending; route simple tasks to smaller models and reserve larger models for complex planning tasks.
• Observability: include invocation volume, latency, failure rate, and cost per task in logging and monitoring services, and configure automatic notifications for anomalies.
• Sensitive data policy: define which code and data are allowed to leave the internal network, and combine this with redaction and egress auditing.

8. Build Your Own Evaluation: Turn Subjective Experience into Regression Assets

Do not rely on personal experience when selecting tools. Build a task-level evaluation suite:

1. Task set construction: extract 50 to 200 tasks from real historical requirements, covering four categories: new features, defect fixes, refactoring, and test completion. Preserve reference implementations and test cases.

2. Scoring dimensions: include functional pass rate, first-pass success rate, average rework rounds, reasonableness of change scope, number of unauthorized or dangerous operations, and cost per task.

3. Regression mechanism: any change to prompts, model versions, or context strategies should trigger offline regression. If metrics degrade beyond a defined threshold, release should be blocked.

4. Gradual rollout and feedback loop: start with a small number of projects, structurally collect review feedback, and feed it back into convention files and preference configurations.

eval_suite:
  tasks: 120
  categories: [feature, bugfix, refactor, test_gen]
  gates:
    pass_rate_min: 0.75
    first_pass_rate_min: 0.45
    rework_rounds_max: 2.0
    risky_action_count_max: 0
    cost_per_task_max: 0.6      # Normalized cost units

9. Implementation Checklist

• Establish repository-level convention files and retrieval-based context injection to reduce manual context preparation cost.
• Enforce “plan first + confirmation points” for multi-file changes.
• Use structured output and schema validation to improve determinism, and use confidence fields to preserve judgment boundaries.
• Tokenize aesthetic constraints and connect them to static checks and visual regression.
• Deploy a permission gateway: default deny, isolated execution, human confirmation for destructive operations, and full auditing.
• Use a unified gateway for multi-channel redundancy, combined with budget, caching, and monitoring for cost and availability governance.
• Build an internal evaluation suite and regression gates so that tool selection and tuning are based on data rather than subjective experience.

The real reason users are willing to continue paying is not the name of a product, but the fact that the product significantly reduces the friction between a requirement and usable code. Decomposing this friction into four engineering dimensions—context, planning, determinism, and security—makes it possible to reduce dependence on any single tool and obtain this value stably under compliant and controllable conditions.