Coding Agents: Choosing Execution Frameworks and Model Pairings

    Coding agent performance depends on both the underlying large language model and the execution framework around it. This article explains the five core components of an execution framework, compares mature general-purpose frameworks with vendor-native frameworks, and provides task-based selection guidance. It also covers how to design controlled experiments, record key metrics, and write effective project instruction files for reliable adoption.

    1. The Core Question: More Than the Model

    The real-world performance of a coding agent is determined by two parts: the reasoning capability of the underlying large language model, and the execution framework that wraps the model. The model itself only outputs text. The execution framework is responsible for turning that text into real actions, such as reading source code, searching symbols, running tests, modifying files, and calling external tools.

    This division of labor shapes how to evaluate options. The model determines whether the agent can think clearly. The framework determines whether it can execute reliably. A strong reasoning model can still underperform if the framework cannot provide accurate code context or has no retry mechanism after a failed tool call. Therefore, models and frameworks should be evaluated separately, and their effects should not be conflated.

    2. Five Core Components of an Execution Framework

    1. System prompts

    System prompts define the agent’s role, behavioral rules, and output requirements. They specify when the model should read code before modifying it, whether it should ask questions or make assumptions when requirements are unclear, and whether tests should be run after changes. Prompt quality directly affects the model’s decision-making habits.

    2. Toolchain

    The toolchain is the set of capabilities the framework provides to the model, including file read/write, code search, terminal command execution, and web retrieval. Toolchain design should focus on clear parameter formats, structured return values, and sufficiently specific error messages. Whether the model can correct its next action depends on how readable those error messages are.

    3. Project instruction files

    Project instruction files are plain-text descriptions stored in the code repository. They record project structure, build and test commands, coding conventions, dependency versions, and prohibited actions. The framework loads them automatically at the start of each session, so the model does not need to repeatedly ask for basic information. The more accurate and concise these files are, the lower the probability of invalid model actions.

    4. Subagents

    For large tasks, the framework can split work into subagents running in independent contexts. Each subtask has its own context window and returns only a summary to the main context when finished. This prevents the main context from being filled with large amounts of intermediate logs and makes it easier to process independent modules in parallel. Subagent scheduling and result aggregation are important indicators of framework maturity.

    5. Permission management

    Permission management defines which operations can run automatically and which must wait for human confirmation. Typical high-risk operations include deleting files, overwriting uncommitted changes, pushing code, and accessing external networks. Reasonable permission tiers preserve the efficiency of continuous execution while limiting the damage from mistakes.

    3. Why Strong Models Depend on Mature Frameworks

    Strong reasoning capability is necessary but not sufficient. In the following scenarios, framework quality directly changes the final outcome.

    Context management for long tasks. As task steps increase, conversation history keeps growing. The framework needs to compress early irrelevant output and preserve key state, such as the list of modified files, tests that have not yet passed, and confirmed design decisions. Poor compression strategies can cause the model to forget earlier constraints and repeat mistakes.

    Reliability of multi-step tool calls. Tool parameters generated by the model may have format errors, and commands may fail because of environment differences. A mature framework validates parameters, returns structured errors, and guides the model to retry within a reasonable number of attempts. If this layer is weak, a task may fail midway even when the model’s judgment is correct.

    Information gathering before edits. A good framework encourages the model to read relevant files and find function definitions and callers before making changes. Without this guidance, the model may assume interface signatures from memory and generate code that does not compile.

    Clarity of permission boundaries. Users will only allow the model to execute multiple steps continuously when permission boundaries are clear. Frameworks with crude permission design either interrupt the workflow frequently or allow too many dangerous operations.

    These areas require substantial engineering work. As a result, the same strong model can perform differently across frameworks depending on how complete the frameworks are in these areas.

    4. Comparing Two Routes

    Route A: A mature general-purpose framework with a third-party strong model

    This route has three main advantages. First, the framework has undergone long-term iteration and generally has a more complete toolchain, subagent system, and permission mechanism. Second, the same framework can switch between different models for side-by-side comparison, making it easier to evaluate new models without changing the workflow. Third, the team can focus on project instruction files and workflow conventions, which are assets that can be reused across models.

    The disadvantage is the adaptation layer. Protocol differences exist between models and frameworks, such as tool-call message formats, handling of reasoning content, and context-length calculation. Some models may require additional configuration or compatibility adjustments, and certain unique behaviors may conflict with the framework’s default assumptions.

    Route B: A vendor-native framework

    The advantage is deep adaptation. A native framework is usually designed specifically for its own model’s prompt format, tool-calling protocol, and reasoning characteristics. Parameter tuning and version upgrades are synchronized, so adaptation costs are lower. For teams that use only one model, this consistency is often simpler.

    The disadvantage is a narrower ecosystem. Migrating across models requires adapting to a new way of working, some advanced features may be less mature than those in general-purpose frameworks, and the team’s workflow can easily become tied to a single vendor.

    Differences in cost

    The two routes have different cost structures. Some options bill uniformly by model usage, while others require separate charges for framework usage and model calls. Framework efficiency also affects cost: if a framework generates many invalid repeated calls, total cost can remain high even when the unit price is low. Therefore, cost comparisons should be based on the total cost of completing the same set of tasks, not on unit price alone. Specific prices are subject to each service’s official pricing.

    5. Selection Guidance by Task Type

    There is no need to seek a single answer. Tools can be allocated according to task characteristics.

    • Small feature development, script writing, and localized changes. The task chain is short, and orchestration requirements are limited. When cost-sensitive, a native framework is often a practical choice.
    • Large repository refactoring, multi-module collaboration, and parallel subtasks. The value of subagent scheduling and context management is more prominent. Mature general-purpose frameworks generally have an advantage.
    • Unified team standards and long-term maintenance. Permission policies, project instruction files, and workflow templates should be reusable across models. If the team may switch models in the future, prioritize frameworks that support multiple models.
    • High security and compliance requirements. Focus on permission management and log auditing. Whichever route is chosen, sensitive operations should be traceable.

    6. Use Experiments Instead of Intuition

    Subjective impressions are easily influenced by task selection and usage habits. A more reliable approach is to design controlled experiments and compare results after controlling variables.

    Step 1: Build a task set. Select 20 to 30 tasks from real projects, covering four categories: bug fixes, new features, code refactoring, and test writing. Task descriptions should be fixed to avoid manually rewritten requirements.

    Step 2: Fix experimental conditions. Each experiment group should use the same code repository version, project instruction files, and acceptance tests. Only the framework or model under evaluation should change, while other factors remain consistent.

    Step 3: Record key metrics. Record the first-pass rate, number of human interventions, completion time, input and output token usage, and actual cost. For failed cases, classify them as reasoning errors, tool-call errors, or configuration issues so that improvement areas can be identified.

    The following is a structural example of an evaluation record. The field values are for format illustration only:

    1. # Evaluation record structure (sample data, for field meaning only)
    2. task_result = {
    3. "task_id": "refactor-042",
    4. "framework": "framework_a", # execution framework under evaluation
    5. "model": "model_x", # model under evaluation
    6. "passed_first_try": False, # whether acceptance tests passed on first run
    7. "human_interventions": 2, # number of human interventions
    8. "wall_time_sec": 418, # completion time in seconds
    9. "input_tokens": 152340,
    10. "output_tokens": 18702,
    11. "failure_category": "tool_call", # reasoning / tool_call / config / none
    12. }

    Step 4: Analyze the results. Compare how different combinations perform across task categories rather than looking only at overall averages. A framework may be clearly stronger on refactoring tasks while costing more on small edits. Category-level statistics help teams make more detailed decisions.

    7. Key Points for Writing Project Instruction Files

    Regardless of which route is chosen, project instruction files are the lowest-cost way to improve results. A practical instruction file usually includes the following:

    1. Project directory structure and the responsibilities of each module;
    2. Standard commands for installing dependencies, building, and running tests;
    3. Coding conventions, such as naming rules, error handling approaches, and log formats;
    4. Prohibited actions, such as not modifying auto-generated files and not committing secrets;
    5. Common pitfalls and historical decisions, such as compatibility constraints for a module.

    Keep the file concise and record only information the model cannot infer directly from the code. Excessive length consumes the context window and can reduce the weight of key information. The instruction file should also be updated regularly as the project evolves and included in code review.

    8. Adoption Recommendations

    When introducing coding agents, teams can proceed in the following order: first define the main task types, then run small-scale experiments to compare two or three combinations, then standardize permission policies and project instruction files, and finally expand usage gradually in real projects. Each stage should retain human acceptance checks to ensure that agent output meets engineering quality requirements.