Selecting AI Coding Agents: Weighing Models vs. Execution Frameworks

    This article explores the key factors in selecting AI coding agents, distinguishing between underlying model capabilities and execution frameworks. It breaks down the five components of an execution framework, discusses native model adaptation, cost calculation, and evaluation methods using controlled experiments, and provides practical recommendations to help development teams make informed decisions based on their specific task structures.

    1. Distinguish Two Layers of Capabilities

    The final effectiveness of a coding agent is determined by two parts: the underlying model’s reasoning and coding capabilities, and the outer execution framework that organizes the model into a continuously working system. These two are often conflated. The model determines how complex requirements it can understand and how correct the code it writes is; the execution framework determines whether it can read the correct files, run commands safely, and maintain its direction in long-running tasks.

    The same model can perform noticeably differently in different frameworks. Therefore, when discussing “which solution is better,” it is necessary to isolate the variables and examine the model and the framework separately.

    2. Five Components of an Execution Framework

    1. System Prompts

    System prompts define the agent’s role, behavioral boundaries, and output specifications—for example, whether to provide a plan before making modifications, whether to proactively ask questions when encountering ambiguities, and how to report test results. The quality of the prompt directly affects whether the model follows engineering conventions.

    2. Toolchains

    The toolchain is the channel for the agent to interact with the development environment, typically including file read/write, code search, terminal execution, test running, and version control queries. The more complete the tools, the more the model can work based on the actual state of the code rather than guessing from memory. Whether the error messages returned by tools are clear also affects the efficiency of the model’s self-correction.

    3. Project Instruction Files

    These files record the project’s coding conventions, build and test commands, directory structure, and prohibitions. Once loaded into a session, the model does not need to repeatedly ask for basic information, which is especially valuable for large repositories. Instruction files should be kept short and verifiable; overly long descriptions will squeeze out context space.

    4. Sub-agents

    Sub-agents offload subtasks such as retrieval, test analysis, and code review to independent contexts, returning only the conclusions to the main process. The main context is thus kept cleaner, alleviating attention dispersion issues in long tasks. The trade-off is increased scheduling complexity; information transfer between subtasks must be clearly designed in advance.

    5. Permission Management

    Permission management determines which files the agent can read, which commands it can execute, which network resources it can access, and which operations require manual confirmation. The principle of least privilege applies here as well. Reading regular code can be allowed freely, while deleting files, pushing branches, and accessing external services should be authorized step-by-step.

    Below is an illustrative generic permission policy, demonstrating the design philosophy without corresponding to any specific product’s configuration format:

    1. policy:
    2. file_read: allow
    3. file_write: confirm # Requires manual confirmation before modifying files
    4. shell_exec:
    5. allowlist: # Pre-approved commands by the team
    6. - "project test command"
    7. - "project build command"
    8. default: confirm # Commands not in the allowlist require confirmation
    9. network: deny # External network access is denied by default
    10. destructive_ops: deny # Destructive operations like deletion and force overwriting are denied by default

    3. Value of Native Model Adaptation

    When the framework’s tool-calling format and context organization method match the model’s training habits, the model can often more stably output valid tool calls, with fewer parameter errors and format drifts. This adaptation is typically reflected in details such as how tool descriptions are written, the truncation strategy for long contexts, and how reasoning content is separated from the final answer.

    The advantage of native adaptation is predictability. Teams can be more confident in determining whether a failure in a certain type of problem is due to insufficient model capability or improper framework configuration. The disadvantage is deep binding. When migrating to other models, some prompts and workflows may need to be re-tuned.

    4. How to Calculate Costs

    Cost should not only look at the list price but also the total consumption to complete a task. A simple estimation formula is as follows:

    Single Task Cost = Input Token Count × Input Unit Price + Output Token Count × Output Unit Price + Retry Count × Single Cost

    Among them, the retry count is the easiest to underestimate. If a framework has a high failure rate for tool calls, the model may need to repeatedly correct itself, and the actual cost may far exceed that of a solution with a lower unit price. Billing methods also vary; pay-as-you-go, subscription packages, and fixed quotas suit different usage intensities and should be selected after calculating based on the actual monthly task volume.

    5. Evaluation Using Controlled Experiments

    A single spectacular case can easily lead to overestimating a solution. A more reliable approach is to use the same set of tasks to conduct controlled experiments under different solutions.

    The implementation steps are as follows:

    • Select 10 to 20 representative tasks, covering bug fixing, feature addition, minor refactoring, and test completion.
    • Fix the initial code and acceptance tests for each task to ensure different solutions face the same conditions.
    • Record the pass status, number of manual interventions, time consumed, and cost for each task.
    • Group and analyze statistics by task type to observe whether differences are concentrated on a specific type of task.

    The summarization script can be written simply:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class TaskResult:
    4. passed: bool
    5. interventions: int
    6. minutes: float
    7. cost: float
    8. def summarize(results: list[TaskResult]) -> dict:
    9. n = len(results)
    10. if n == 0:
    11. raise ValueError("results cannot be empty")
    12. return {
    13. "pass_rate": sum(r.passed for r in results) / n,
    14. "avg_interventions": sum(r.interventions for r in results) / n,
    15. "avg_minutes": sum(r.minutes for r in results) / n,
    16. "avg_cost": sum(r.cost for r in results) / n,
    17. }

    6. Common Pitfalls

    1. Looking only at single demonstrations. One success does not represent stability; averages must be taken after multiple runs.
    2. Comparing only models, ignoring frameworks. The same model may yield very different results in different frameworks.
    3. Ignoring hidden costs. Retries, manual reviews, and rework time should all be included in the total cost.
    4. Treating prompts as a panacea. Issues with permission design and tool capabilities usually cannot be solved by modifying prompts.

    7. Selection Recommendations

    There is no one-size-fits-all answer for selection; it depends on the team’s task structure.

    • For scenarios involving large repository maintenance, multi-person collaboration, and strict permission control, priority should be given to examining the framework’s permission management, sub-agents, and project instruction mechanisms.
    • For personal rapid prototyping and small-scale script development, priority can be given to cost and the performance of native model adaptation.
    • Regardless of the scenario, controlled experiments should be used to gather data before making a decision.

    Evaluation conclusions should be documented in team files and re-tested after model or framework version updates. Tools iterate rapidly, and the validity period of a single evaluation conclusion is typically short.