Why the Same LLM Performs Differently Across Agent Frameworks (and How to Choose the Right One)

One phenomenon that developers have verified again and again: plug the same capable model into two different coding agent frameworks, run the same batch of coding tasks, and the results can differ dramatically. The cause usually isn’t the model itself, but the often-overlooked “execution framework” wrapped around it—commonly known in the industry as the harness. This article breaks down the core components of a harness, explains how models and frameworks fit together, and offers a practical, actionable method for evaluating and choosing one.

1. Behind the Phenomenon: Model Capability ≠ Delivery Quality

Many developers have run this experiment: the same batch of bug fixes, feature development, and refactoring tasks, given to the same model, but executed in two different agent frameworks. One framework delivers a clearly higher first-pass success rate; the other repeatedly hits tool-call failures, misunderstood requirements, and lost context. This points to a fact that’s easy to miss: what we experience day to day as “how good the model is” is really the combined performance of “model + framework.”

A simplified formula captures this:

Final delivery quality = base model capability × framework amplification factor × task fit

• Base model capability: the underlying level of reasoning, code generation, and instruction following;
• Framework amplification factor: how effectively the harness draws the model’s ability out—it can be below 1 (suppressing performance) or above 1 (amplifying it);
• Task fit: how well the task type matches the framework’s design orientation.

When the same model behaves very differently across frameworks, the amplification factor is usually what’s at work.

2. The Five Building Blocks of an Execution Framework

2.1 The System Prompt: Defining How the Work Gets Done

The system prompt is the framework’s “job description” for the model. It typically specifies the role and capability boundaries, a standard workflow (for example: “read the relevant code first, draft a plan, make small changes, verify each step”), output format constraints, and exception-handling conventions. In mature frameworks, system prompts have been polished against huge volumes of real tasks, which markedly reduces the odds of the model going off track. A schematic fragment:

# Role and goal
You are a senior software engineer responsible for completing coding tasks
in the current repository.
# Workflow
1. Search for and read the relevant code first to understand the existing implementation;
2. Draft a modification plan; for large-scale changes, state the plan first;
3. Make small, incremental changes, running the relevant tests after each step;
4. Summarize the changes and the verification results.
# Constraints
- Do not modify code unrelated to the task;
- State the purpose before executing any command;
- When dependencies are uncertain, verify before acting.

2.2 The Toolchain: Defining What the Agent Can Do

File reading and writing, terminal command execution, code search, web queries, external tools connected through a general-purpose extension protocol—the breadth and quality of the toolchain directly cap what the agent can do. One easily underestimated point is the quality of the tool definitions themselves: whether descriptions are clear, parameter schemas are well structured, and granularity is sensible all have a significant impact on call success rates. With vaguely defined tools, even the strongest model will fail calls repeatedly and burn context and time on endless retries.

2.3 Project Memory Files: Solving Cross-Session Forgetting

Forgetting everything once a session ends is inherent to general-purpose LLMs. Mature frameworks therefore commonly support project-level instruction files that persist long-lived information: build commands, coding standards, directory structure, known pitfalls, and so on. A typical project instruction file looks roughly like this:

# Project instructions (for the agent)
## Build and test
- Install dependencies: ./scripts/install.sh
- Run tests: ./scripts/test.sh (stop on failure)
- Start locally: ./scripts/dev.sh
## Coding standards
- Follow the coding standards document in the repository root; format before committing;
- Every new API must come with unit tests.
## Directory structure
- src/services/: business logic layer; never assemble storage-layer statements directly;
- src/api/: API layer; parameter validation and forwarding only.
## Known pitfalls
- Test fixtures depend on the DB_HOST environment variable; when it is missing,
  test cases silently skip, creating the illusion that "everything passes."

Maintenance advice: update the file in step with refactoring; keep it short and include only information the model would get wrong without; and manage it in layers—global, project, and module.

2.4 Sub-Agents: Divide and Conquer Against Context Bloat

A single session’s context window is finite. When a task involves wide-ranging code searches or bulk modifications, intermediate process information quickly crowds out the context. The sub-agent mechanism applies divide and conquer: the main agent handles planning and summarization, while exploratory subtasks (such as “locate every call site of this API across all modules”) are delegated to independent sub-agents. Each sub-agent rummages through the code within its own context and returns only conclusions. This preserves the main context and keeps process noise from polluting the main task’s judgment. Typical use cases include large-scale code searches, batch refactoring, and parallel investigation of multiple approaches.

2.5 Permission Management: Balancing Safety and Efficiency

Once an agent can read files and execute commands, permission management becomes the safety baseline. Mature frameworks usually provide tiered controls: read-only operations pass automatically, command execution goes through a whitelist or manual confirmation, file writes are confined to designated directories, and sensitive operations require explicit approval. A schematic configuration:

permissions:
  read:
    - "src/**"        # source code reads: auto-approved
  write:
    allow:
      - "src/**"
      - "tests/**"
    deny:
      - "infra/**"    # infrastructure configs: no automatic changes
  execute:
    allow:
      - "run_tests"   # test commands: whitelisted
    confirm:
      - "deploy*"     # deployment commands: manual confirmation required
      - "rm *"        # deletion commands: manual confirmation required

Permissions that are too tight interrupt the workflow constantly; too loose, and you invite mistakes and security risks. Strike the balance according to your team’s security requirements and desired level of automation.

3. Model–Framework Fit: Why Stronger Models Benefit More from Mature Frameworks

One important pattern: the more capable the model, the more it stands to gain from a mature framework. Stronger models follow instructions better and plan complex work better, so they can “absorb” fine-grained system prompts, multi-level tool orchestration, and sub-agent collaboration—the framework’s maturity converts directly into task completion rates. Conversely, a relatively limited model may suffer instruction overload inside an overly complex framework and become less stable; a simple, lightweight framework is sometimes the better choice.

Another easily overlooked dimension is native adaptation. When a framework and a model come from the same team and are tuned jointly, the system prompt can be polished sentence by sentence for that model’s characteristics, tool-call formats and parsing logic align exactly, and the context window and truncation strategy match each other. This “factory-grade” coordination often yields extra stability. When a third-party model is plugged in, the framework’s prompts and tool definitions may not be optimized for it, and a certain ramp-up cost is unavoidable.

So “which framework is better” is really a false question. The right question is: given your target model, target tasks, and target budget, which model–framework combination has the highest fit? A strong model paired with a mature framework usually reaches the performance ceiling; a model and framework from the same source often secures a more dependable floor; and in budget-sensitive, light-usage scenarios, cost may end up dominating the decision.

4. Evaluation and Selection: A Practical Decision Method

4.1 Build an Eval Set from Your Own Tasks

Public benchmarks differ greatly from real work. The most reliable approach is to build your own evaluation: pick 15–30 real tasks from your daily work, covering bug fixes, new feature development, local refactoring, adding missing tests, and script writing. Set uniform acceptance criteria, then measure the first-pass success rate, the number of human interventions, and total time on each framework. One round of data will give you a more trustworthy answer than any review article.

4.2 A Checklist of Key Evaluation Dimensions

Beyond task completion rate, it’s worth tracking:

• Tool-call success rate: failed calls mean retries and wasted context;
• Context management: whether long sessions get forgetful and whether key files are reliably recalled;
• Observability: execution logs, change previews, and failure rollback;
• Safety mechanisms: whether tiered permissions and sensitive-operation confirmations are sufficient;
• Extensibility: whether internal team tools can be connected through a general-purpose protocol;
• Cost structure: whether the billing model matches your usage frequency.

4.3 Cost Trade-offs: Subscription vs. Pay-as-You-Go

Subscriptions charge a fixed monthly fee, suit high-frequency heavy users, and keep budgets predictable. Pay-as-you-go bills by token consumption and suits low-frequency or highly variable workloads. A rough estimate: assume a typical task consumes about 200K tokens end to end and you run 10 tasks per day—that’s roughly 60 million tokens per month on pay-as-you-go. If the subscription price is clearly lower than the equivalent token cost, the subscription wins; otherwise pay-as-you-go is more flexible. Billing rules vary considerably across platforms, so defer to the official documentation. The core idea: model the decision on your real task frequency and token consumption, not on gut feeling.

4.4 A Simplified Decision Flow

def choose(model, tasks, budget):
    candidates = load_frameworks()                  # candidate harnesses
    for fw in candidates:
        score[fw] = benchmark(fw, model, tasks)     # own eval: pass rate / interventions / time
    affordable = [fw for fw in candidates
                  if monthly_cost(fw, tasks) <= budget]
    return max(affordable, key=lambda fw: score[fw])

5. Practical Advice: Getting Real Value from Your Framework

1. Treat the project memory file as a living document. Write back every pitfall and every standards change so the agent understands your project better the more it is used;
2. Loosen permissions gradually. Start restrictive and observe behavior, then whitelist high-frequency safe operations to cut needless manual confirmations;
3. Use sub-agents for exploratory work. Hand searches and research to sub-agents to keep the main context clean;
4. Review failed tasks regularly. Categorize the causes (prompt defects, vague tool definitions, missing information) and feed the findings back into framework configuration;
5. Recalibrate after model upgrades. A stronger model may need adjustments to prompt detail and tool granularity—old configurations aren’t necessarily optimal.

Major cloud providers are also continuously improving their managed model services and supporting AI development infrastructure. The same evaluation method applies here as well: verify with your own benchmark and pick the combination that best matches your stack and budget.

6. Closing Thoughts

The execution framework is an amplifier for model capability: the system prompt determines how the work gets done, the toolchain defines the capability boundary, project memory fights forgetting, sub-agents fight context bloat, and permission management holds the safety line. When choosing, don’t put faith in any single “best” framework. Focus instead on the fit among model, framework, task, and budget, and let data from your own evaluations decide. Configure the five harness components well, and you can truly convert model capability into stable engineering delivery.