Choosing an LLM Coding Assistant: Capability Evaluation, Controlled Autonomy, and Permission Safety

From “Who Is Strongest” to “How Do We Manage Capability Differences?”

When bringing AI coding assistants into a team, many organizations decide on “which model is the strongest” and then keep running into problems in practice. The reality is closer to this: capability profiles are highly uneven across models. Some excel at backend defect localization and interface refactoring but are clearly weak at UI aesthetics and copywriting. Others are good at multi-step planning and filling in implicit requirements. Yet in multi-turn conversations they keep producing balanced, hedged explanations that leave conclusions hard to act on.

That points to three problems worth engineering:

• Measuring capability differences: turn subjective impressions into reproducible evaluation sets and metrics.
• Constraining initiative: let the model take a few extra steps to cover implicit requirements, without letting it change things it shouldn’t or make decisions on its own.
• Isolating autonomous execution: once the assistant starts reading files, running commands, and connecting to the network, permission boundaries and an audit trail must be in place first.

The rest of this article follows these three threads.

Build a Multi-Dimensional Evaluation Set Instead of Relying on “Feel”

“Feels good to use” doesn’t survive a technical review. Break evaluation down into scoreable dimensions and build test samples on your own codebase:

Dimension	What to look at	How to measure it
Functional correctness	Whether the generated code passes existing unit tests	First-pass rate, number of revision rounds
Engineering quality	Naming, layering, error handling, testability	Static-analysis warning count, manual score
UI quality	Layout, style consistency, accessibility	Design review pass rate
Context usage	Whether it follows the repository’s existing conventions and dependency versions	Number of changes that violate conventions
Multi-step planning	How well complex tasks are decomposed	Plan first-pass rate
Communication efficiency	Whether conclusions are actionable, how dense the information is	Manual adoption rate

Draw the evaluation set from real bug tickets and real requirements. It should cover three task types: single-point fixes, cross-file refactoring, and building a module from scratch. Use at least 20 samples per type, and fix the random seed and temperature parameters so results are comparable.

# Evaluation pipeline: load cases -> call the unified adapter layer -> run checks -> store metrics
for case in load_cases("evals/bugfix.yaml"):
    resp = llm.complete(case.prompt, context=case.repo_context)
    report = run_checks(resp, tests=case.tests, lint=case.lint_rules)
    metrics.append({
        "case_id": case.id,
        "tests_passed": report.tests_passed,
        "lint_issues": report.lint_issues,
        "revision_rounds": report.revision_rounds,
        "human_accepted": report.human_accepted,
    })
persist(metrics)  # Write to a logging service or object storage for cross-version comparison

The key conclusion: task-level routing usually beats a one-size-fits-all choice. Sending backend bug fixes and UI generation to different models, each suited to the task, usually pays off more than forcing everything onto a single vendor.

Context Sufficiency Sets the Ceiling on Output Quality

A pattern confirmed time and again: the more complete the context a model gets (framework background, dependency versions, coding standards), the closer its output is to something that can be merged as-is. When information is missing, it won’t guess and fill the gaps on its own. Instead, it leaves a lot of placeholders or improvises with generic patterns. This isn’t the model being “lazy.” With missing context, any guess is a gamble.

Split context into three layers and put them under version control:

1. Global conventions layer (a collaboration guide in the repository root): tech stack and versions, directory responsibilities, error-handling patterns, logging rules, dependencies that must not be added, and commit and review rules.
2. Module layer: interface contracts, data models, and state-flow descriptions for core modules.
3. Task layer: acceptance criteria, impact scope, and regression cases for the current change.

# Collaboration Guide (excerpt)
- Pin the runtime version and how the package manager is locked; no major-version dependency upgrades
- The service layer must not access the data access layer directly; cross-layer calls go through interface contracts
- All errors use the custom exception hierarchy; never throw a bare generic exception
- Every external interface must have authentication and rate limiting; a missing one counts as a defect
- Frontend components follow existing design tokens; no hard-coded color values

Pair this with on-demand retrieval and injection. Use vector search or a symbol index to pull out only the file fragments relevant to the current task. Keep the injected volume within the context budget instead of stuffing the whole repository into the prompt, which dilutes the key information. The context pruning strategy should itself be configurable and tuned to each model’s capabilities.

Output Contracts: Clear Conclusions That People Can Act On

On factual questions, models tend to give balanced, conditional answers. That’s an advantage when warning about risks, but in troubleshooting and conclusion-writing it becomes noise: readers finish and still don’t know what to do next. The fix isn’t to complain about the style. Define an explicit output contract in the prompt:

Output requirements:
1. Give the conclusion first, then the supporting evidence; no more than 3 conclusions.
2. Tag each conclusion with a confidence level: high/medium/low. A low-confidence conclusion must state what information is missing.
3. List assumptions explicitly; never present an assumption as a fact.
4. Don't repeat the information I've already provided, and don't summarize your reasoning.
5. For high-risk operations (deletion, migration, permission changes), output only the steps and verification methods, and do not execute them.
6. If information is insufficient, ask no more than 3 of the most critical follow-up questions.

Tie the contract to your evaluation. Track “whether a conclusion was adopted by a human as-is” and “whether follow-up questions were needed” as regression metrics. Only then can you tell whether the contract actually works.

One more point: narrow the model’s role in high-risk domains. If a user describes feeling unwell, the assistant shouldn’t guess at causes or suggest medication. It should direct the person to a doctor and explain its limitations. Finance and law work the same way. This is a hard product and compliance constraint. Put it in the system prompt and in post-processing checks, not in the model’s own judgment.

Proactive Planning and Controllability: Two Extra Steps, Never One Out of Bounds

Initiative pays off for real. Planning the next steps in advance, reusing existing patterns from the project, and filling in implicit requirements can cut back-and-forth noticeably. Excessive autonomy is just as common: refactoring on the side into a different style, upgrading dependencies with no real need, or editing files outside the task’s scope.

The engineering answer is to separate the workflow into phases: planning (read-only), writing (after human approval), and verification.

Stage 1 (read-only): allow reading code, searching, running tests and static checks; no writes or installs.
                     Output must include: the list of files to change, the reason for each change, the risks, and how to roll back.
Stage 2 (write): after a human approves the plan, write only within whitelisted paths;
                 reject any write request outside the list and log it.
Stage 3 (verify): automatically run tests and linting, produce a diff for review, and do not merge automatically.

Back this up with a few hard rules:

• A whitelist of paths that may be changed (for example, only src/ and tests/).
• CI configuration and dependency lockfiles are off-limits.
• Each task’s diff has a size limit.
• After two consecutive failures, stop and report instead of endlessly tweaking.

Together, these keep “initiative” within a range a human can review.

Permissions and Audit for Autonomous Execution: Lessons from One Unexpected Scan

A case that came up in practice: while working on a task, an assistant happened to find a process unrelated to that task. It judged the process to be a risk program and asked to delete it outright. It then extended the scan across the machine, looking for other suspicious records and outbound network connections. Whether or not that judgment was correct, the same structural problem is exposed: when an assistant can read and execute, the cost of a mistake is too high to fix after the fact. A false positive can damage a healthy environment, a deletion can take business data with it, and the model’s reasoning is often opaque.

That leads to four risk classes and their controls:

1. False positives and destructive operations
Deny by default anything that deletes, formats, bulk-rewrites, terminates processes, or changes registry or service settings. Allow it only after a person has confirmed each item and a dry run has been executed first.

2. Supply chain and dependency risk
The assistant may introduce package names that don’t exist (hallucinated dependencies), which an attacker could register and poison. Controls:

• Lockfiles with checksum comparison
• Private mirror repositories
• Generating an SBOM
• Blocking unregistered dependencies in CI

3. Credential and data leakage
Source snippets, environment variables, and secrets can be carried out along with prompts. Controls:

• Keep secrets only in a secrets management service, injected as temporary credentials
• Scan and redact sensitive information before sending a prompt
• Never use production data as a debugging sample

4. Indirect prompt injection
Third-party docs, dependency READMEs, ticket text, or web content that the assistant reads may carry embedded instructions. Controls:

• Treat external content as untrusted data, and only summarize or retrieve from it
• Never let its content change the tool-calling policy

Command interception can be a separate gateway sitting between the assistant and the operating system:

import re, shlex
DANGEROUS = [
    (r"\brm\s+(-[a-zA-Z]*r[a-zA-Z]*\s+)+", "recursive deletion"),
    (r"\b(mkfs|dd\s+if=|shutdown|reboot)\b", "system-level destruction"),
    (r"\b(curl|wget)\b.*\|\s*(ba)?sh", "remote script executed directly"),
    (r"\b(chmod\s+777|chown\s+.*root)\b", "permission change"),
    (r"\b(ps|netstat|lsof)\b.*\bkill\b", "process inspection followed by kill"),
]
def guard(cmd: str, policy: dict) -> dict:
    argv = shlex.split(cmd)
    for pattern, reason in DANGEROUS:
        if re.search(pattern, cmd):
            return {"allow": False, "reason": f"Matched a forbidden rule: {reason}", "cmd": cmd}
    if policy.get("read_only") and any(w in argv[:1] for w in policy.get("write_bins", [])):
        return {"allow": False, "reason": "Currently in the read-only stage", "cmd": cmd}
    if not policy.get("network", False) and re.search(r"\b(curl|wget|nc|ssh)\b", cmd):
        return {"allow": False, "reason": "Outbound network access is disabled by default", "cmd": cmd}
    return {"allow": True, "reason": "", "cmd": cmd}

At the runtime level, set these as defaults:

• A dedicated low-privilege account
• A containerized, one-shot execution environment
• Source code mounted read-only
• Outbound traffic through an allowlist proxy
• No credentials injected
• Session destroyed when it ends

Log every command, file change, and session record to an audit log, and connect it to a logging service and monitoring. Set threshold alerts for abnormal frequency, such as large numbers of file reads in a short time or repeated attempts to escalate privileges.

Multi-Model Collaboration: Keep Your Assets Outside the Vendor

Architecturally, avoiding lock-in to a single vendor is the fundamental way to deal with capability differences, cost swings, and access stability. Build a thin unified adapter layer and keep the differences in capability declarations and prompt templates:

class LLMBackend(Protocol):
    name: str
    strengths: set[str]          # e.g. {"backend_debug", "frontend", "planning"}
    max_context: int
    def complete(self, prompt: str, context: str, contract: dict) -> str: ...
def route(task_type: str, backends: list[LLMBackend]) -> LLMBackend:
    ranked = [b for b in backends if task_type in b.strengths]
    ranked.sort(key=lambda b: b.max_context, reverse=True)
    if not ranked:
        raise RuntimeError(f"No backend available for task type: {task_type}")
    return ranked[0]

The operational strategy that goes with it:

• Circuit breaking and fallback: when the primary backend times out or is unavailable, switch to a backup. In extreme cases, fall back to a local small model or rule-based handling.
• Caching: reuse results for the same context and prompt.
• Token budget and context pruning
• Tiered calls by task value: use a low-cost tier for drafts and a high-capability tier for reviews.

Track at least these metrics:

• Call volume
• Success rate
• First-pass rate
• Revision rounds
• Human adoption rate
• Cost per task

On access stability and compliant use, the practical approach:

• Use official purchasing and commercial channels rather than shared credentials or proxy workarounds.
• Follow the terms of service.
• Make data export and retention requirements explicit.
• Keep two-vendor redundancy on critical paths.
• Keep your most valuable assets in your own repositories and cloud environment: evaluation sets, context docs, prompt templates, constraint rules, and audit logs.

This is also where the real “high switching cost” comes from. When the workflow lives on the vendor’s side, migration is very expensive. When it lives on your side, model selection can be made rationally.

Running this pipeline in the cloud doesn’t require anything exotic, because general-purpose services cover it. Container platforms provide isolated execution environments. Object storage archives evaluation results and prompt template versions. Logging services carry the audit trail. Monitoring and alerting catch abnormal frequency and sudden cost spikes. Products like those from Baidu AI Cloud (containers, storage, logging, and monitoring) are enough to support this architecture, with no dependency on any single model service.

Rollout Plan

• Weeks 1–2: Pull real bug tickets and requirements to build an evaluation set of at least 60 samples, and capture baseline metrics.
• Weeks 3–4: Write the three-layer context docs and the output contract template, re-run the evaluation, and track changes in first-pass rate and human adoption rate.
• Weeks 5–6: Set up a containerized sandbox and a command interception gateway. Enable the read-only stage, whitelisted writes, and audit logging.
• Weeks 7–8: Connect the unified adapter layer and task routing. Configure circuit breaking and fallback, caching, and cost alerts, and set up a monthly capability review.

Conclusion

The real barrier to using AI coding assistants isn’t the model itself but the engineering around it: measurable evaluation, reusable context, executable output contracts, reviewable initiative, and an isolated execution environment. Once those five are solid, “which one is stronger” stops being a matter of feel and becomes an engineering problem you solve per task type, cost, and risk constraints.