AI Coding Assistant Selection: Three Tool Categories, Four Evaluation Dimensions, and Workflow Decisions

1. Why “Which one is best?” Is the Wrong Question

When helping teams adopt AI-assisted development, the most common question is: “Which tool is the strongest?” However, real engineering conclusions are rarely about a single best tool. Ranking different tools in the same coordinate system often ignores three important differences: their capability levels, their runtime locations, and their permission boundaries.

More useful questions are:

• What type of task am I trying to solve: local completion, cross-file refactoring, defect localization, batch migration, or documentation/test generation?
• How much context radius does this task require: a single file, a single module, or the whole repository?
• How much autonomous execution am I willing to allow for irreversible operations such as writing files, running commands, changing configurations, or pushing to remotes?
• What verification mechanism will act as a safety net: compilation, unit tests, static checks, or human review gates?

Once these four questions are structured, tool selection becomes an engineering decision rather than a brand preference.

2. Tool Evolution: A Three-Layer Capability Model

From code completion to agents, the industry has produced three clearly distinguishable forms. Their differences are not mainly about model strength, but about where they enter the developer’s work loop.

Form	Runtime location	Main intervention point	Context radius	Typical strengths
Inline completion	Inline editor suggestions, asynchronous suggestions	During typing	Single file plus a few nearby symbols	Low friction, preserves train of thought
Editor-native assistant	In-editor chat and inline diffs	Conversational Q&A / explicit range editing	Open files + selected range + workspace index	Strong diff visualization, intuitive review
Terminal agent	Command-line process with shell and filesystem access	End-to-end task execution	Can autonomously retrieve the whole repository	Long-chain tasks, scriptability, orchestration

The three capability layers can be summarized as follows:

1. Prediction layer: Given cursor context, predict the next several tokens. The goal is to reduce keystroke cost.
2. Editing layer: Given a natural-language intent and a code range, produce a reviewable diff. The goal is to replace manual editing.
3. Execution layer: Given a task goal, autonomously complete the closed loop of “retrieve → plan → edit → run verification → fix.” The goal is to replace “completing a task.”

The key insight is this: Most team inefficiencies do not come from model capability. They come from using execution-layer tools for editing-layer tasks, or forcing editing-layer tools to handle execution-layer tasks. The former creates excessive changes and review burden; the latter creates a lot of manual copy-paste.

3. Four Decisive Dimensions

3.1 Context Construction and Repository Understanding

Context quality determines the upper bound of output quality. The three forms differ mainly in their retrieval strategies:

• Passive context: Uses only currently open files, selected ranges, and visible symbols. It requires zero configuration, but cross-file tasks often end up “seeing only half of the picture.”
• Indexed context: Builds vector or symbol indexes over the workspace and supports semantic retrieval and symbol-reference navigation. It is suitable for questions such as “Which call sites use this function?” However, indexes have lag. After switching branches or upgrading major versions, the index may need to be rebuilt.
• Active exploration context: The agent retrieves information on demand through shell and search tools, such as grep, directory traversal, reading build scripts, and inspecting test results. This allows it to expand the context window as needed.

Active exploration is clearly stronger for long-chain tasks, but two costs must be included in the evaluation: first, context noise, such as irrelevant build artifacts, lock files, and generated code being read into the context; second, token and cost predictability.

In engineering practice, regardless of which form is used, teams should maintain a repository-level task description file. It should explicitly document build commands, test commands, directory responsibilities, coding conventions, and directories that must not be modified. For example:

# .ai/repo-guide.md
## Build and Verification
- Install dependencies: see the `bootstrap` target in the Makefile
- Unit tests: `make test-unit` (must be all green; coverage must not fall below the baseline)
- Full build: `make build-all`
- Static checks: `make lint` (must pass before commit)
## Code Structure
- `src/core`: domain logic; web-layer dependencies are forbidden
- `src/adapter`: adapters for external systems and protocols
- `migrations`: database migrations; additions only; modifying published files is forbidden
## Hard Constraints
- Do not modify any files under `generated/` or `vendor/`
- Do not introduce new runtime dependencies unless the reason is explicitly stated in the plan
- All public functions must include type annotations and docstrings

The practical value of this file is often underestimated. It converts implicit team knowledge into preconditions that the agent can read for every task, directly reducing rework.

3.2 Edit Granularity and Change Visibility

Review cost is often higher than generation cost. The three forms differ significantly in visibility:

• Line-level ghost text: Suitable for very short changes. Accept or ignore can be done with one action, but it has almost no expressive power for cross-file changes.
• In-session diff view: Changes are aggregated and displayed as diffs. Developers can accept or reject them by file, making it easier to understand the impact before accepting.
• Terminal patch + external diff: The agent writes changes directly to disk, and the developer reviews them through version-control diffs. The advantage is that changes naturally enter the Git workflow, and can be precisely controlled with git diff and git restore. The tradeoff is that there is no visual preview before commit.

A practical rule of thumb is this: When a task’s changes span more than three files, the value of visual diffs rises quickly. When a task requires running commands to verify results, the value of the terminal form rises quickly. This also explains why the two are often used together.

3.3 Permission Model and Execution Boundaries

This is the most easily overlooked and most error-prone part of enterprise environments. Four types of permissions need to be distinguished:

1. Read permission: Which directories and files can be accessed, including whether secret files, production configurations, or customer data are included.
2. Write permission: What scope can be modified, including whether CI configurations, migration scripts, or infrastructure code may be changed.
3. Execution permission: Which commands can be run. Compilation and tests are usually safe; package-manager installations, curl outbound requests, database changes, and git push --force are high-risk.
4. Network and outbound permission: Whether code snippets are sent to external inference services, and whether full prompts are retained in logs.

A practical implementation is tiered approval plus an allowlist:

# Illustrative agent execution policy (pseudo-config; exact keys depend on implementation)
auto_approve:
  - "make test-unit"
  - "make lint"
  - "go build ./..."
require_confirmation:
  - "git commit"
  - "package-install"
deny_always:
  - "git push --force"
  - "db-migrate:prod"
  - "network-egress:*"
read_scope:
  include: ["src/**", "tests/**", "docs/**"]
  exclude: ["**/.env*", "**/secrets/**", "**/*.pem"]

The principle is simple: Default to least privilege, and keep confirmation authority for irreversible operations with humans. Terminal agents should run in containers or disposable workspaces as much as possible, and be destroyed after use to avoid polluting the local environment or credentials.

3.4 Verification Loop and Failure Recovery

The dividing line between agents and completion tools is whether they verify themselves. A reliable execution loop looks like this:

Read task → Retrieve context → Make plan → Edit → Run verification commands
   ↑                                              │
   └──────────── On failure: read error output and fix ←────────────┘
                                                 ↓
                                        After all checks pass, output change summary

Three details determine whether this loop succeeds:

• Verification commands must be deterministic and fast. If the unit test suite takes 20 minutes, the agent’s iteration rhythm will be damaged. Provide both a “fast subset” and a “full regression” mode.
• Error messages must be structured and parseable. Compilation errors and test failures must be readable reliably. Otherwise, the agent can only guess.
• An iteration limit must be set. Without a limit, the agent may repeatedly tweak the same error, consume budget, and still produce no result. A limit of 3 to 5 attempts is recommended. Beyond that, roll back and escalate to a human.

4. How to Conduct Credible Empirical Evaluation

Do not trust generic leaderboards. Build an evaluation set that belongs to your own repository. The following is a reusable method template.

4.1 Standardized Task Set

Select 12 to 20 tasks from real requirements, covering five categories, with 2 to 4 tasks per category:

Task category	Example	Context radius	Key difficulty
Local completion	Implement a parser function with boundary handling	Single file	Specification consistency
Cross-file refactoring	Extract a shared abstraction and update all call sites	Whole repository	Impact completeness
Test generation	Add unit tests and edge cases for an existing service layer	Module	Assertion validity
Defect localization	Locate the root cause from an error stack and fix it	Cross-layer	Root cause vs. symptom
Migration/upgrade	Major-version upgrade of a framework or dependency and adaptation	Whole repository	Coverage of breaking changes

Each task should have fixed inputs: task description + repository snapshot + repository-level guide. It should also have fixed verification commands and fixed acceptance criteria.

4.2 Metric Definitions

# Evaluation execution example (pseudo-code)
METRICS = {
    "wall_time":      "Wall-clock time (minutes) from human intervention to acceptance passing",
    "turns":          "Number of interaction turns required to reach acceptance",
    "accept_rate":    "Accepted changed lines / total generated changed lines",
    "review_flags":   "Number of times marked for rework during human review",
    "first_pass":     "Share of tasks that pass the first verification run",
    "cost":           "Token consumption and cost, calculated using the team's actual billing basis",
}

Two points are easily misread:

• Acceptance rate is not equal to quality. A high acceptance rate may simply mean the task is easy. It must be interpreted together with “first-pass rate” and “rework count.”
• Failure patterns must be recorded, not just success rates. Failure patterns such as out-of-scope changes, hallucinated APIs, missed call sites, and unauthorized dependency upgrades are more useful than average scores for improving conventions.

4.3 Evaluation Process Discipline

1. Use the same repository snapshot, the same guide file, and the same verification command to avoid variable contamination.
2. Repeat each task three times and take the median. Agent outputs are stochastic, so a single run is not trustworthy.
3. Have acceptance performed by colleagues who do not know which tool was used. This is blind evaluation and reduces confirmation bias.
4. Archive all results as a team baseline report, and retest quarterly as tools and models iterate.

5. Task-Feature-Driven Decision Matrix

Combining the dimensions above, we can build a selection matrix that does not point to any specific brand:

Task characteristics	Recommended form	Permission recommendation	Required mechanisms
Clear, small-scope change; line-level fix	Inline completion	Read-only + in-editor write	Lint gate before commit
Semantic editing in a single file or single module	Editor-native assistant	Write within workspace	File-by-file diff review
Refactoring across more than 3 files	Terminal agent	Workspace write + command allowlist	Plan first, then execute; force full test suite
Defect localization and root-cause analysis	Terminal agent, starting read-only	Read + execute diagnostic commands	Limit repair scope; forbid opportunistic refactoring
Batch generation of tests and documentation	Terminal agent, scripted	Write only to designated directories	Generated results must pass CI and coverage gates
Dependency upgrades, framework migration	Human-led + agent execution	Tiered approval	Batch commits, rollbackable branches, no changes to published migrations
Code involving secrets, customer data, or production configs	Carefully evaluate whether to introduce at all	Exclude read paths	Data non-export and desensitization audit

The first principle of decision-making is: The larger the context radius and the more irreversible the change, the more important it is to plan first and verify strongly, rather than rely on stronger automatic generation.

6. Combined Workflows Are the Endgame

Mature teams usually do not “choose one” tool. They divide responsibilities by workflow stage:

• When writing code: Inline completion handles boilerplate code and familiar APIs, preserving flow.
• When changing structure: Editor-native assistance handles small semantic edits and diff review.
• When handling long tasks: Terminal agents handle the complete loop of retrieval, planning, editing, and verification, and can process multiple repository tasks in batches or in parallel.
• During wrap-up: Scripts or CI handle static checks, coverage gates, and specification validation, downgrading “humans do review” to “humans make decisions.”

A practical daily loop can look like this:

# 1. Pull items from the task system, create an isolated work branch, and use a disposable container
# 2. Have the agent read the repository-level guide and output a plan first; do not modify code directly
# 3. After a human confirms the scope and risk points, allow execution
# 4. Agent self-verification: lint → unit-test subset → targeted tests
# 5. Output change summary and verification logs, then enter the code review gate
# 6. After full regression and human review pass, merge; on failure, roll back and archive the failure pattern

On the cloud side, this type of workflow requires connecting the agent’s execution environment and context sources to existing R&D infrastructure, usually through general-purpose capability categories: container platforms provide isolated execution environments, object storage and message queues carry task queues and artifacts, and logging services and monitoring/alerting handle audit and cost observability. Cloud platforms such as Baidu AI Cloud also provide this basic hosting through similar general-purpose capabilities. When selecting such platforms, the key checks should be permission granularity, auditability, and data residency, not the length of the feature list.

7. Common Anti-Patterns

1. Using agents as completion tools: Asking them to write familiar code line by line wastes planning and execution capability, and introduces unnecessary context overhead.
2. Starting long tasks without a guide file: Every task requires rediscovering how to build and test, and specification drift becomes severe.
3. Allowing automatic commit and push: One mistaken operation can pollute the main branch. Always keep confirmation authority for commits and pushes with humans.
4. Missing iteration limits: Budget is consumed by ineffective retries.
5. Using acceptance rate as the only KPI: This encourages teams to pick easy tasks and hides real value.
6. Ignoring data boundaries: Handing a repository containing secrets or customer data to an execution environment without isolation is dangerous.

8. Conclusion

Selection of AI coding assistants can be reduced to four conclusions:

• Classify tasks before discussing tools. Completion, editing, and execution correspond to completely different intervention points.
• Context and the verification loop determine the ceiling. The model itself is only one variable among many.
• Least privilege is a hard constraint for enterprise adoption. Irreversible operations must retain human confirmation.
• Replace generic reputation with a self-built evaluation set. Make the baseline report a quarterly retest asset owned by the team.

For most teams, the final form is a combined workflow: inline completion preserves flow, editor diffs preserve review, terminal agents ensure delivery, and CI gates ensure quality. Tools will keep iterating, but the four capabilities of task classification, context engineering, permission governance, and verification loops are stable. Once these capabilities are built well, any new tool can quickly be placed in the right position.