Why the Same LLM Performs Differently Across Agent Harnesses

A Common Phenomenon: Same Model, Different Results

Many developers have run into the same puzzling experience: a model with clearly top-tier reasoning and coding ability performs mediocrely in one AI coding environment, yet the same model seems noticeably “smarter” in a different execution environment—higher task completion rates, fewer rework cycles, and code style that better matches project conventions. The model weights have not changed. What changed is the execution framework wrapped around the model, commonly referred to in the industry as the harness.

To understand this difference, the first thing to recognize is that agentic coding is not a single question-and-answer exchange but a multi-turn loop system: understanding the task, forming a plan, calling tools, observing results, and self-correcting—repeating until the goal is reached. The model acts as the “brain,” while the harness carries all the engineering responsibilities: information assembly, tool execution, state maintenance, and safety fallbacks. The same brain connected to different nerves and limbs naturally produces different output quality. Therefore, “model capability sets the ceiling, while harness quality sets the floor and the variance”—this is the basic starting point for analyzing any difference in AI coding performance.

Deconstructing the Harness: Five Core Components That Determine Outcomes

1. System Prompt: The Master Switch for Behavior

The system prompt defines the agent’s role, workflow, output standards, and prohibitions. A mature harness refines this prompt against a large number of real tasks, covering edge cases: whether to fix or work around a failing test, when to proactively ask the user, and how to avoid changes beyond the task scope.

A well-structured system prompt skeleton looks roughly like this:

You are a senior engineer responsible for completing tasks in the current code repository.
Workflow:
1. Read the relevant code and understand the context before making changes;
2. Run tests after changes; if tests fail, fix them rather than bypassing them;
3. Do not make changes beyond the task scope; if large-scale refactoring is needed, explain the plan to the user first;
4. Follow the existing code style and naming conventions in the repository.

The same model, given a carefully designed system prompt versus a rough generic template, can differ in behavioral stability by an order of magnitude. This is the first source of performance differences.

2. Toolchain: Extending Model Capabilities

Common tools include file read/write, fuzzy search, command execution, task list management, and external material retrieval. Many developers focus only on whether a certain type of tool exists, overlooking a more critical factor: the quality of tool definitions. Whether parameter descriptions are clear and whether error messages can guide the model toward self-correction directly determine call success rates.

Compare two tool definitions:

{
  "name": "read_file",
  "description": "Read the contents of a file at the specified path. The path is relative to the repository root.",
  "parameters": { "path": "Relative file path, e.g. src/utils/date.py" }
}

When a file does not exist, a rough implementation returns Error: ENOENT, and the model can only retry blindly. A good implementation instead returns “File not found. Similar nearby paths: src/utils/date_helper.py,” and the model self-corrects in one attempt. These detail-level differences in the toolchain accumulate into significant gaps in efficiency and success rate.

3. Project Context Rules: Long-Term Memory and Constraints

Well-designed harnesses typically support maintaining a documentation file in the project root that records build commands, test methods, directory structure, code standards, and common pitfalls. It is essentially “long-term project memory” for the model:

# Project Notes
- Build: make build (or the equivalent build command configured for the project)
- Test: make test, supports module filtering: make test MODULE=services
- Directory conventions: business logic belongs in the services layer; controllers must not contain complex business logic
- Known issues: the legacy serialization module has compatibility logic; read its comments before making changes

Without this file, the model must “get to know” the project anew each time, acting on guesses and easily stumbling. With a concise rules file, the model avoids many detours. It is worth emphasizing that the value of a rules file lies in being concise rather than exhaustive—verbose, outdated content dilutes the weight of key constraints.

4. Subagents: Task Decomposition and Context Isolation

For complex tasks, mature harnesses support dispatching subtasks to subagents for parallel exploration (for example, globally searching all call sites or investigating candidate approaches), bringing only the conclusions back to the main conversation. The core value is context isolation: the main conversation window is not polluted by massive intermediate output, and the model maintains stable reasoning quality over long tasks.

Take a refactoring effort as an example: the main agent handles solution design and final acceptance, while subagents handle the mechanical work of “finding all reference locations and summarizing them.” This division of labor significantly reduces the probability of “getting lost” midway through long-context tasks and is an important indicator of a harness’s engineering maturity.

5. Permission Management: Balancing Safety and Efficiency

Agents need to execute commands and rewrite files, so permission management is essential. A typical strategy is tiered control: read-only operations pass directly, file writes require confirmation, and dangerous commands such as deletion or network downloads are blocked or routed through a whitelist. Permissions that are too coarse lead to frequent user interruptions; permissions that are too fine-grained carry high configuration costs. Mature harnesses provide tiered policies and “in-session memory” to strike a balance between safety and smooth operation—especially important for team collaboration and production environments.

Why Mature Harnesses Amplify Strong Models

Putting these components together explains the phenomenon described at the beginning:

• Positive feedback loop for instruction following: Stronger models follow complex system prompts more faithfully and make fuller use of fine-grained rules; weaker models under complex harnesses tend to lose track of priorities. The more mature the harness, the more it can “squeeze” capability out of a strong model.
• Context engineering quality: The harness determines what gets fed to the model. With the same model, precisely assembled context is like writing a clear problem statement; poorly assembled context leaves the model guessing.
• Error recovery loop: Whether test failures and compilation errors can be smoothly fed back to the model and trigger a self-repair cycle directly determines whether a task ultimately converges.

This leads to an important inference: a strong model plus a mature harness yields a high ceiling and low variance; a strong model plus a rough harness yields a high ceiling and high variance, with results fluctuating between good and bad. This is precisely the root of the “inconsistent experience” many developers report.

Model-Native Harnesses vs. General-Purpose Harnesses

Around the model’s execution environment, two approaches exist in the industry. One is the coding agent built by the model vendor itself, co-optimized for its own model’s output style and tool-calling habits—ready to use out of the box, with low onboarding cost and often a more favorable cost structure. The other is a heavily refined general-purpose harness with deep accumulations in permission systems, subagent mechanisms, and extension ecosystems; it demands more general capability from the model, but once matched, offers greater room for results.

A selection approach can follow this logic: for scenarios prioritizing quick onboarding, budget sensitivity, and relatively standardized tasks, model-native solutions are usually more cost-effective; for deep scenarios such as large code repositories, complex workflows, and team engineering collaboration, general-purpose mature harnesses with targeted tuning often hold the advantage. In addition, note that both models and harnesses are iterating rapidly—today’s mismatch may improve markedly in the next version—so selection conclusions should be reviewed periodically rather than set in stone.

A Holistic Method for Weighing Cost and Effectiveness

On the business side, subscription plans suit individuals and teams with stable usage intensity, where fixed costs simplify budgeting; pay-as-you-go suits scenarios with highly variable demand. But more important than the billing model is establishing a multi-dimensional evaluation system for effectiveness rather than concluding based on a “feels good to use” impression:

1. First-pass task completion rate;
2. Average number of rework rounds;
3. Per-task resource consumption and overall cost;
4. End-to-end latency;
5. Performance stability on long-context tasks.

A simple, practical evaluation approach:

tasks = load_tasks("eval_set.jsonl")  # 10-20 typical real-world tasks
for framework in ["framework_a", "framework_b"]:
    results = [run_and_score(t, framework) for t in tasks]
    report(framework, results)  # summarize completion rate / rework count / latency / cost

Only by comparing with the same set of tasks and the same scoring criteria do conclusions have decision-making value.

Actionable Optimization Recommendations

1. Do context engineering first, before considering a model switch: Refine the project rules file, remove outdated content, and explicitly document high-frequency pitfalls. This is often the optimization with the highest return on investment.
2. Design subagent division of labor for complex tasks: Separate mechanical exploration from the main conversation to keep the main context clean.
3. Build a team evaluation set and include it in regression testing: Automatically run it after model or harness upgrades, validating with data rather than subjective impressions.
4. Tier permission policies: Use relaxed policies for daily iteration and switch to strict sandboxes when production data and dangerous operations are involved.
5. Record failure cases: Systematic weaknesses (such as inaccurate retrieval or poor tool error messages) should be fed back into harness configuration or selection decisions in a timely manner.
6. Enterprise deployment advice: Enterprise users can complete validation and deployment on the model services and intelligent development toolchains offered by platforms such as Baidu AI Cloud, evaluating them together with internal codebases, permission systems, and compliance requirements rather than directly applying individual developer experience.

Conclusion

The relationship between a model and its harness is like that between an engine and overall vehicle tuning: the engine sets the performance ceiling, but the acceleration, handling, and stability users actually experience depend on whole-vehicle systems engineering. Returning to the original question—“which is better” has no absolute answer; it depends on task complexity, team engineering maturity, and budget structure. The right approach is to treat the harness as an object of engineering optimization: understand its components, establish quantitative evaluation, and let data drive selection and tuning. When you can actively shape the model’s working environment, AI coding performance shifts from “a matter of luck” to “something manageable.”