Which Execution Framework Should You Pair with a Strong Model? A Guide to Choosing a Coding Agent Harness

Having a large language model write code in the terminal has become a real working style for many teams. But a recurring frustration is that models that score well on benchmarks often perform inconsistently in real repositories. Sometimes a single conversation produces a high-quality refactor; other times, even a simple bug fix comes out incomplete. Most people blame the model, but they overlook a key variable: the execution framework, or harness, wrapped around it. The model sets the ceiling of capability. The execution framework determines how much of that ceiling is actually realized.

What an execution framework is: the engineering system beyond the model

A complete coding agent consists of two layers:

• Model layer: the large language model responsible for understanding, reasoning, and generation.
• Framework layer: the runtime system built around the model, typically including five major parts: system prompts, toolchain, project-level context files, sub-agent orchestration, and permission management.

The core job of the execution framework is to turn a conversation into a deliverable engineering action. It determines what the model can see, what it can call, what constraints it operates under, and how it recovers from errors. The same model placed in different frameworks can produce a full tier of difference in engineering performance. Below, we break down the five key components.

1. System prompt: the model’s engineering handbook

The system prompt is the first layer of constraint a framework places on the model. It usually includes role definition, workflow, tool usage rules, and output conventions. Mature frameworks embed substantial engineering experience into this prompt. For example: perform read-only exploration before making changes; run tests after modifications; ask for confirmation when uncertain instead of guessing. These details determine whether the model behaves like a rigorous engineer or a passionate but careless intern.

2. Toolchain: the model’s hands and feet

Reading files, writing files, executing commands, searching code, running tests, checking version control status — the design granularity of the toolset, the quality of error messages, and the truncation and summarization strategy for returned results all directly affect the model’s decision quality. A typical difference: a search tool that returns verbose, unsummarized output quickly fills the context window and overwhelms the model. In contrast, an execution tool with clear error messages and structured output lets the model locate problems quickly and self-correct.

3. Project-level context files: write once, benefit long-term

Mainstream frameworks support maintaining an instruction file in the repository root that tells the model about coding conventions, directory structure, common commands, and forbidden actions. Instead of repeating the same instructions in every conversation, it is better to write them down once. A minimal viable version looks roughly like this:

# Project conventions
- Build: run ./scripts/build.sh
- Test: run ./scripts/test.sh; the task is complete only when all tests pass
- Style: follow the style of existing files; do not introduce new dependencies without explicit requirements
- Directory: src/core contains core logic; explain the reason for any changes
- Forbidden: do not modify any auto-generated files

Experience shows that documenting build commands, test commands, and what not to do is the highest-return framework configuration action you can take.

4. Sub-agents: preserving the signal-to-noise ratio of the main context

When facing large tasks, mature frameworks delegate exploration, retrieval, and verification work to separate sub-agents that run in isolated context windows and return only conclusions to the main conversation. This keeps the main context information-dense at all times. A crude implementation, by contrast, dumps raw retrieved content into the main conversation. After a few turns, critical information gets buried, and the model starts losing track of the task. This is where framework design quality shows most visibly.

5. Permission management: balancing efficiency and safety

The permission system determines which operations can be executed automatically and which require human confirmation. Mature designs are typically tiered: read-only operations are allowed by default; writes within the working directory are allowed as needed; dangerous operations, such as deletion, overwriting, changing production environments, or force-pushing branches, are intercepted and require confirmation. This preserves execution efficiency while maintaining a baseline of safety — the latter being a prerequisite for broader adoption in enterprise environments.

Why framework maturity determines a model’s capability conversion rate

A rough but useful formula describes the actual effectiveness of a coding agent:

Actual effectiveness ≈ model capability × framework conversion rate

The same strong model can reliably complete cross-file refactors in a well-tuned framework, yet struggle to untangle module dependencies in a crude one. The reasons come down to three points.

First, context engineering determines what the model sees. Code tasks are extremely information-dense, and context windows are scarce resources no matter how large they get. A poor framework fills the window with irrelevant search results. A good framework uses layered retrieval, sub-task isolation, and structured summaries to reserve the limited window for what truly matters. For a strong model, seeing the right information is often more important than being slightly smarter.

Second, process constraints determine how the model works. A mature framework guides the model through an explore-plan-execute-verify loop through prompt and tool design. Without such constraints, models tend to start writing immediately, producing changes that are hard to review, hard to roll back, and occasionally touch things that should not be touched.

Third, error recovery determines what happens after something goes wrong. In real execution, command failures and failing tests are the norm. Whether the framework feeds complete error output back to the model, guides it to diagnose before retrying, and stops after repeated unproductive loops directly decides whether a task ultimately succeeds or fails.

Choosing between two routes: native clients vs. mature general-purpose frameworks

Around the same model, developers typically have two types of execution carriers to choose from, each with a fundamentally different logic.

Route one: the model provider’s native client

• Advantages: cost-friendly — usually tied to a subscription, so marginal cost approaches zero under heavy use, with no need to count token costs for every tool call. Native adaptation — the provider’s own client usually has the most complete and timely support for its model’s context window, output control, and inference parameters. Synchronized iteration — model upgrades and client upgrades happen together, so new capabilities are available immediately.
• Drawbacks: the depth of framework-layer polish varies. In areas such as prompt engineering, sub-agent orchestration, and permission tiers, some products are still catching up with industry best practices.

Route two: mature general-purpose coding agent frameworks

• Advantages: high conversion rate — system prompts, toolchains, project memory, sub-agents, and permission management often represent current best practices, more fully unlocking the potential of strong models. Mature ecosystem — they have accumulated substantial workflow patterns and extension mechanisms, offering strong customizability. Model-agnostic — they can work with different models, making horizontal evaluation and fallback options easier.
• Drawbacks: pricing is usually usage-based. Agentic tasks easily involve dozens of tool-call rounds, so costs under heavy use are noticeably higher than subscriptions. Additionally, support for specific features of non-native models may lag.

Putting the two routes side by side reveals a straightforward conclusion: for an already strong model, the conversion-rate improvement from a mature framework is often greater than the adaptation benefit from a native client. For teams with high usage intensity and cost sensitivity, the subscription model plus native adaptation of a native client is equally compelling. This is the root cause of the phenomenon where the same model performs drastically differently when switched between frameworks.

Cost perspective: structural differences between subscriptions and usage-based pricing

• Subscription: fixed and predictable costs, suitable for individuals and small teams with daily high-frequency usage. Based on publicly available information, mainstream subscription tiers commonly fall in the range of one to two hundred yuan per month. For a full-time developer, a single successful multi-file refactor can often cover the monthly cost.
• Token-based pricing: usage is transparent and can be attributed to projects and teams, making financial accounting and cost allocation easier. This suits enterprise-level governance. However, costs for high-frequency agentic tasks can fluctuate significantly, so budget alerts are necessary.

A pragmatic approach is hybrid usage: route high-frequency, low-risk tasks such as writing tests, scripts, and minor fixes through a subscription channel. Reserve the budget for high-value, high-complexity tasks, using a strong model paired with a mature framework. On the enterprise side, teams can also choose to deploy model capabilities on their own cloud infrastructure or access models through a cloud provider’s managed LLM service, gaining finer control over cost, compliance, and data security.

A practical evaluation method for framework selection

Do not rely on a single benchmark score. Instead, use a fixed set of baseline tasks for quantitative comparison.

1. Prepare a task set: take 5 to 10 typical tasks from real repositories, covering cross-file bug location and fixing, small feature development including new files and tests, behavior-preserving refactoring, and failure-recovery scenarios where an error is intentionally introduced.
2. Standardize measurement: record first-pass success rate without human intervention, interaction rounds and tool-call counts, whether changes follow project conventions, and whether dangerous operations are correctly blocked.
3. Observe behavior, not just results: does the framework explore before acting, does it wander through irrelevant files, does it fall into infinite loops after errors? These behavioral details reveal framework maturity most clearly.
4. Keep variables consistent: use the same model, the same sampling parameters, and the same context settings during comparison. Change only the execution framework.

for task in benchmark_tasks:
    result_a = run_agent(task, framework=A)
    result_b = run_agent(task, framework=B)
    record(pass_at_first, rounds, diff_scope, safety_events)

Practical recommendations: capturing the full framework dividend

1. Write a good project instruction file: this is the single highest-return action. Document build commands, test commands, and forbidden actions clearly.
2. Use permission tiers well: tighten defaults and grant access as needed, balancing safety and smooth operation.
3. Make good use of sub-agents and task decomposition: break large tasks down before execution, keeping the main context clean.
4. Build your own regression baseline set: rerun it after every model or framework upgrade to quantify whether the upgrade is worth it.
5. Orchestrate costs in a hybrid model: subscription channels handle high-frequency tasks; usage-based strong models handle high-value tasks.

Conclusion

Back to the original question: where a model is plugged in matters as much as the model itself. The model determines the ceiling of capability; the execution framework determines the conversion rate. Native clients win on cost and native adaptation. Mature general-purpose frameworks win on fully realizing the potential of strong models. There is no single correct answer. First clarify your task profile and cost structure, then validate with a fixed baseline task set. Making decisions based on data rather than reputation is the right way to turn intelligent coding into real efficiency gains.