AI Coding Agents: A Practical Path to Selection, Evaluation, and Migration

1. Retrospective on the Migration Path: From “Picking One Tool” to “Orchestrating a Set of Capabilities”

For many teams, choosing an AI coding tool comes down to “try a bunch, then decide.” A more valuable approach is to abstract that migration history into a staged model, because the decision variables at each stage are completely different.

Stage 1: Trial period—establish a horizontal baseline. Run several AI coding tools in parallel over the same period, using the same set of real requirements (not the vendor’s sample tasks). Only two dimensions matter here: delivery efficiency and code quality. The most important output of this stage is not “the single best tool,” but a set of task samples that belong to your own repository—every later decision will rely on them to reproduce results.

Stage 2: Mainstream period—discover the depth of agent engineering. When the scenarios and the underlying model are the same, output quality can still differ noticeably between products. That is when you realize: prompts are only the surface. What really creates the gap is how context is assembled, how tools are orchestrated, and how the agent recovers after failure.

Stage 3: Migration period—cost and quota become the dominant variables. Switching billing from “per request” to “per token” is the direct reason many teams are forced to reselect tools: a budget that used to last a whole month is exhausted within days after the change. The engineering lesson is clear—any solution that depends on a single vendor’s quota must design an exit path in advance.

Stage 4: Multi-model period—from a single workhorse to hybrid orchestration. When complex tasks repeatedly fail on one tool but pass on the first try with another, the “pick one” mindset breaks down. A sound end state is to keep an abstract invocation layer, route requests to different models by task type, and continuously maintain a reproducible evaluation.

2. The Depth of Agent Engineering: Same Model, Different Output

One frequently overlooked fact: with identical model capability, delivered quality can differ by a full tier. The difference comes from four engineering areas.

1. Context assembly. This decides which files, which fragments, and what granularity of dependency relationships get packed into the context window. Stuffing everything in dilutes attention and drives up cost; packing only the current file loses cross-module contracts. An effective approach is layering: first give the repository structure and key interface signatures, then fill in details based on retrieval results.
2. Tool set and permission boundaries. The granularity design of the four tool categories—read, write, execute, and retrieve—directly determines whether the agent can discover errors on its own. Without the “execute and read the output” link, the agent can only guess.
3. Feedback loop and self-verification. Have the agent run tests, static checks, and type checks by itself before submitting, turning “looks correct” into “actually passes.”
4. Failure recovery and state management. During long-running tasks, context drifts. Explicitly writing the plan, completed items, and items pending verification into an external state file is far more reliable than letting the model rely on its own memory across a long context.

A minimal viable execution loop:

def run_task(goal, workspace, policy):
    state = State(plan=planner(goal, workspace.skeleton()))
    while not state.done and state.rounds < policy.max_rounds:
        action = model.next_action(goal, state)
        result = tools.execute(action, sandbox=workspace)   # read / write / execute / retrieve
        state.append(action, result)
        if policy.verify_each_round:
            report = tools.run("build && test && lint", workspace)
            if report.failed:
                state.append_fix_request(report)
                continue
        state.advance()
    return state.finalize()

3. Trade-Offs Between Two Technical Routes

There are roughly two ways to solve the problem in this space, and understanding them is more valuable in the long run than memorizing a product name.

Dimension	Route A: Purpose-built capability reinforced with vertical training	Route B: General model + engineering methods
Strength scenarios	Long-chain, cross-file complex refactoring and diagnosing stubborn defects	Interactive local edits and directional work that requires frequent probing
Delivery form	One-sentence description, then a long autonomous run to completion	Multi-turn dialogue, with a human able to step in at any time
Typical cost	A single run can take tens of minutes; feedback is slow	Complex problems must be split by hand; per-turn cost is high
Explainability	Process is a black box, hard to intervene midway	Process is visible, easy to steer and correct

Intuitively, you can sum it up as: one route is “smarter,” the other is “more flexible.” “Smarter” shows up in the ability to plan autonomously and see a long task through; “flexible” shows up in fast responsiveness to user intent, controllable change scope, and the pliability that comes from extension mechanisms (subtasks, skill packs, mode switching).

The engineering conclusion is not either/or, but routing by task characteristics: hand deterministic, well-scoped dirty work that can run for a long time to the former, and hand interactive changes that need repeated alignment and fast visible results to the latter.

4. Turning “Feel” into a Reproducible Evaluation

Selecting tools without an evaluation set is essentially gambling. Build a three-tier evaluation set; 20–50 tasks per tier is enough to start.

• L1, function level: Given a signature and a unit test, complete the implementation. Tests basic correctness.
• L2, file level: Fix an already-located defect, with reproduction steps attached. Tests localization and minimal-change ability.
• L3, repository level: Cross-module refactoring, bulk migration, dependency upgrades. Tests long-chain stability.

Record evaluation results in a structured way for easy side-by-side comparison and regression:

{
  "task_id": "L3-017",
  "level": "repo",
  "target": "Unify logging integration and replace scattered calls",
  "metrics": {
    "first_pass": false,
    "human_takeover_rounds": 3,
    "wall_clock_minutes": 42,
    "tokens_in": 1850000,
    "tokens_out": 96000,
    "cost_usd": 7.4
  },
  "verdict": "Pass, but required human wrap-up"
}

Fix five core metrics: first-pass rate, human takeover rounds, P50/P95 latency, token cost per task, and rollback rate. Run a regression every three months and the selection conclusion will be backed by data.

5. Task Routing: A Practical Routing Layer

Do not hardcode a specific tool in the application layer. Instead, abstract a unified interface and make the routing strategy configuration-driven.

routing:
  - name: quick-edit
    match: { diff_lines_max: 80, files_max: 2, needs_plan: false }
    provider: interactive_model      # low latency, strong interactivity
    max_rounds: 6
  - name: deep-refactor
    match: { diff_lines_min: 200, cross_module: true }
    provider: autonomous_agent       # long-running autonomy, allowed to run in background
    max_rounds: 40
    verify: [build, unit_test, lint]
  - name: frontend-polish
    match: { path_glob: "web/**", intent: [ui, style, layout] }
    provider: aesthetic_strong_model # model best suited for visual and layout aesthetics
    verify: [build, visual_regression, a11y_lint]
  fallback: interactive_model

Three key routing designs: first, capability tags rather than brand names, so a vendor can be swapped at any time; second, each branch is bound to an independent verification pipeline, and nothing may land without validation; third, timeout and fallback strategies, since long tasks must support checkpoint resume and human takeover.

6. Cost Governance Under Token-Based Billing

Once billing switches from per-call to per-token, cost moves from a “fixed subscription” to something that “grows linearly with context,” which must be governed as an engineering discipline:

• Context trimming: Retrieve fragments by relevance; forbid unconditional injection of whole files; summarize and compress long histories.
• Cache reuse: Put stable, unchanging project conventions and interface definitions in a cacheable prefix position.
• Model tiering: Offload format conversion, comment generation, and simple refactoring to small models, and reserve large models for planning and diagnosing stubborn defects.
• Parallelism and rate limiting: Fan out independent subtasks in parallel to multiple capabilities to shorten wall-clock time, while using a token bucket to avoid triggering the vendor’s rate limits.
• Quota observability: Track consumption by task, by person, and by branch so that abnormal consumption is surfaced immediately in monitoring and alerts.

7. Aesthetics: Turning “Ugly” into a Verifiable Constraint

Generative models vary in visual aesthetics and depend heavily on personal taste; a single “make it look nicer” instruction will not converge. The verifiable approach is to engineer it:

1. Fix design tokens: Colors, spacing, corner radii, and font sizes are all constrained by a token file; hardcoding is forbidden.
2. Component library first: Require the agent to compose only existing components, and to justify adding a new one.
3. Visual regression: Take screenshot comparisons of key pages and block anything whose pixel difference exceeds a threshold.
4. Accessibility and style checks: Wire accessibility rules and style specs into CI so that at least part of “aesthetics” becomes machine-checkable hard lines.

This way, even if the generative model is swapped, output will not regress to a “runs but ugly” state.

8. Supply Chain Risk and the Compliance Baseline

The most expensive lesson in the migration history is this: the tool is not your own asset. Risks to hedge in advance include changes to a vendor’s account policy, unstable gateways or proxy channels, prompt breakage caused by model version deprecation, and compliance problems from code and secrets leaving your environment.

The corresponding architecture requirements are straightforward: funnel all model calls through a unified gateway, and have business code depend only on the abstract interface; centralize secret management and forbid committing secrets to the repository; route sensitive code through an internal network or a private deployment, with the data residency location auditable. When hosting such a gateway, you can directly reuse mature general-purpose capabilities from a cloud platform—a container platform handles elastic scaling and canary rollouts, a log service handles call-chain traceability, monitoring and alerting handle abnormal-consumption and failure-rate alerts, and object storage handles archiving of evaluation sets and regression snapshots. On Baidu AI Cloud, these categories of capability are standard configuration and require no self-development.

9. Action Checklist

1. Within this week, solidify 30–50 task samples of your own, split into three tiers: function level, file level, and repository level.
2. Abstract a unified invocation layer, put vendor implementations behind adapters, and make switching configuration-driven.
3. Roll out a routing strategy based on task characteristics, separating long tasks from interactive tasks.
4. Bind each branch to a pipeline of build, test, static checks, and visual regression.
5. Build a token-consumption dashboard and quota alerts, and review per-task cost monthly.
6. Run a selection regression every quarter, and let data—not feel—decide the primary tool.

Conclusion: Do not bind your productivity to a single tool or a single vendor. The genuinely durable advantages come from three things—your own evaluation set, a replaceable invocation abstraction, and the engineering discipline built around verification pipelines. Tools will change, capabilities will drift, and billing will change, but these three always remain yours.