Same Model, Different Results? A Look at the AI Coding Harness

1. A Commonly Misjudged Phenomenon: Is the Problem the Model or the Harness?

Many developers have had this experience: a model looks impressive in benchmarks and leaderboards, but after subscribing, it performs only average in coding tool A. Switch to another tool calling the same model, and code quality, task completion, and rework rates improve noticeably. It is tempting to conclude that “this model just isn’t good.”

A more accurate explanation: the LLM is only the engine; the harness is the whole car. The same engine mounted on chassis with different levels of tuning can produce very different lap times. This is especially true for AI coding, where agent task chains are long, tool calls are dense, and context is complex. Every design decision at the harness level gets faithfully amplified by the model—either into efficiency or into errors.

2. Five Core Components of an Execution Harness

1. System Prompts and Task Orchestration

System prompts determine how the model understands its role, workflow, and output standards. A mature harness typically specifies: explore the repository structure before acting, make small incremental changes, verify each step, and know when to stop and confirm with the user. A common weakness in weak harnesses is overly simplistic prompts—the model receives a vague goal and starts writing immediately, without guidance to locate, plan, implement, and verify first. The result is often large changes with large errors, and small changes with small errors.

2. Toolchain and Tool Description Quality

The model cannot directly “see” your project; it can only observe and modify the world indirectly through tools. The completeness of the toolset (reading files, precise editing, code search, running commands, viewing test output) and the accuracy of tool descriptions directly determine whether the model can form stable behavioral patterns. For example: if the edit tool only supports whole-file overwrites, the model is highly error-prone on long files; if precise block-level editing and clear failure feedback are provided, success rates improve significantly. When tool descriptions are vague, the model frequently “guesses parameters,” and a simple task gets split into a dozen invalid calls.

3. Project-Level Context Files

This is essentially “onboarding documentation for a new colleague”: build commands, test commands, directory conventions, off-limits areas, and team coding standards. If the harness supports letting repository maintainers write these rules into a file that is automatically injected into every session, the probability of the model making “common-sense mistakes” drops substantially. Otherwise, team rules can only be corrected repeatedly by users in conversation—inefficient and unstable.

4. Sub-Agents and Task Decomposition

For long tasks such as cross-module refactoring or large-scale bug hunting, a mature harness assigns planning, retrieval, implementation, and review to isolated sub-agents, preventing the main context from being flooded with irrelevant details. The context window is a scarce resource, and a harness’s task orchestration capability is essentially context engineering: what information should stay resident in the main context, what should be fetched on demand, and what should be pruned promptly.

5. Permission Management and Security Boundaries

Which commands can run automatically, which require confirmation each time, and which are forbidden entirely. Good permission design balances reducing interruptions with maintaining guardrails: too coarse, and it is either unsafe or causes confirmation fatigue; too fine, and the agent can barely move—humans become button-pushers on an assembly line.

3. Why Mature Harnesses Amplify Strong Models More

A simple but accurate observation: the stronger the model, the more leverage the harness provides. A weak model with a good harness still hits the model’s ceiling; a strong model with a poor harness wastes much of its capability on rework and invalid calls. Four specific mechanisms:

• Feedback loops: The harness can automatically run builds, tests, and static checks, then feed error information back to the model—only then does the model have a chance to self-correct. Without a loop, even the strongest model can only “write blind.”
• Context management: Automatic pruning, summarization, and on-demand retrieval prevent “amnesia” late in long tasks—gradually forgetting the original requirement constraints.
• Failure recovery: Retry and fallback strategies after command failures, plus structured parsing of error messages, determine whether a failure is resolved “within three calls” or gets completely stuck.
• Confirmation cadence: When to proceed autonomously and when to stop and confirm directly determines the flow experience of human-AI collaboration.

4. Advantages and Costs of Native Harnesses

In-house harnesses are not without value. Their advantages typically include:

1. Cost control: Co-designing the harness and model allows compression in prompt length, number of tool calls, and repeated context injection, consuming less compute and cost for the same task;
2. Native adaptation: New model features (longer effective context, new tool-calling protocols, etc.) can be enabled in the in-house harness immediately without waiting for third parties;
3. Iteration speed: Joint tuning of model and harness shortens the problem-locating chain and enables fast closure on issues.

The cost: polishing an execution harness is quintessential “dirty work,” depending on accumulated trial and error from many real-world scenarios. Catching up with mature harnesses refined over years is hard in the short term. So “own model + own harness” is not necessarily better than “strong model + mature harness,” and vice versa—it depends on task types, cost constraints, and team situation.

5. How to Run a Fair Comparative Evaluation

Rather than debating “which is better,” run the test yourself with controlled variables:

1. Fix a task set: Pick 5–10 representative tasks (new feature development, bug fixing, small-scale refactoring, adding tests, understanding an unfamiliar module) covering the difficulty range of daily work;
2. Change one variable at a time: Fix the model and switch harnesses, or fix the harness and switch models;
3. Record quantitatively: First-pass success rate, number of manual interventions, total token consumption, wall-clock time, and rework rounds;
4. Attribute failure modes: Was the requirement misunderstood (prompt/rules file issue), was the tool used incorrectly (toolchain issue), or was context lost (context management issue)? Different failure modes point to different improvement directions.

6. Three Optimizations You Can Apply Immediately

1. Write a Good Project Rules File

# AGENT.md — Example Project Collaboration Rules
## Build
- After modifying code, run ./build.sh; on failure, fix before continuing
## Testing
- Run ./test.sh before committing; ensure all tests pass
- New features must include corresponding unit tests
## Style
- Follow the repository's existing static analysis configuration; do not introduce new style conventions
## Boundaries
- Do not modify auto-generated code under generated/
- For database schema changes, output a migration plan first and wait for confirmation

2. Configure Fine-Grained Permissions

{
  "permissions": {
    "allow": ["read:src/**", "write:src/**", "run:./build.sh", "run:./test.sh"],
    "ask":   ["write:infra/**", "run:deploy*"],
    "deny":  ["write:.env", "run:rm -rf *"]
  }
}

3. Use Sub-Agents to Isolate Long Tasks

planner  = Agent(role="planning", tools=["read", "search"])
coder    = Agent(role="implementation", tools=["read", "edit", "run_test"])
reviewer = Agent(role="review", tools=["read", "run_lint"])
plan = planner.run("Add refund logic to the order module and output a step-by-step plan")
for step in plan.steps:
    change = coder.run(step)
    review = reviewer.run(f"Review changes: {change.diff}")
    if not review.passed:
        coder.run(f"Fix per review comments: {review.comments}")

7. Conclusion: Treat Harness Maturity as a First-Class Selection Criterion

Model and execution harness are multiplicative, not substitutable. When choosing tools, look beyond model leaderboards and weigh harness maturity—toolchain completeness, context engineering capability, permission systems, and sub-agent orchestration—as equally important as model capability. For enterprise teams, model access, auditing, and permission governance should also be part of unified infrastructure: for example, using general-purpose platform capabilities such as container platforms, log services, and monitoring and alerting to host the operation and observability of self-built agents, so that “model + harness + platform” work together and turn model capability into sustainable engineering efficiency.