Agent Harness Design: Context, Tools, and Model Decoupling

1. What Is a “Harness”: The Software Layer Outside the Model

In discussions about agent systems, the model usually gets most of the attention, while the software environment wrapping it is often overlooked. The industry refers to this environment as the “harness”: the software layer that wraps the model, manages context, handles file I/O, and executes tool calls.

A useful analogy is that the model is like a high-performing horse, while the harness is all the equipment that determines how much weight it can pull and how steadily it can run.

At the interface level, a large language model is essentially a stateless function: it receives a sequence of tokens and returns a sequence of tokens. It does not know the directory structure of a code repository, which files were just modified, or what the user really wants to verify. All of that “knowing” happens in the harness layer:

• How prompts are assembled;
• How tools are described;
• How tool results are trimmed and returned;
• How errors are translated;
• How permissions are decided;
• How session state is persisted.

This leads to an engineering fact that is often ignored: the same model, placed inside different harnesses, can have success rates that differ by several times on tasks such as code repair, refactoring, and question answering. The difference mainly comes from four areas:

1. Context orchestration determines what the model can see;
2. Tool protocols determine what the model can do, and how costly recovery is when something goes wrong;
3. Error semantics determine whether a failure ends the session or becomes the starting point for self-repair;
4. Permission and workspace design determine whether the system dares to let the model act.

The conclusion is that the harness is not the model’s shell; it is the entire engineering layer that turns a general-purpose model into a usable product. As models iterate faster, the quality of the harness layer increasingly determines the final product experience.

2. Commoditization Pressure: The Entry Point Is the Lever

A clear competitive landscape has emerged in the industry.

On one side, when users access models through general-purpose third-party AI coding interfaces, the underlying model is extremely easy to replace. If Model A performs poorly, users can simply change the configuration and switch to Model B. The model thereby degrades into a commodity that can be purchased in bulk, with competition centered on price and latency, and the bargaining power of vendors is significantly weakened.

On the other side, model providers respond by guiding users into their own harnesses to achieve vertical integration. When a provider controls the runtime, editing experience, and agent orchestration at the same time, it can optimize the model and the harness together. This kind of bundling improves experience consistency, but it also raises migration costs for users.

For engineering teams, the direct implication is that the first selection question is not “which model should we use?” but “at which layer should we place switching costs?”

• If business logic is deeply coupled with a specific model, every model upgrade may mean rewriting prompts and tool protocols;
• If business logic is built on stable abstraction contracts, the model becomes a hot-swappable component, allowing teams to choose freely according to task type, cost budget, and compliance requirements.

The design principles discussed below are built around this goal: unified contracts, explicit negotiation, graceful degradation, and observability.

3. Model Decoupling: Treat the Model as a Replaceable Component

1. Define a Stable Internal Contract

An adapter layer should be inserted between business code and model interfaces, so that differences among vendor APIs are contained at the boundary. Internally, only a unified message model, tool descriptions, streaming events, and error categories are needed. An example is shown below:

from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import AsyncIterator, Literal
@dataclass(frozen=True)
class Message:
    role: Literal["system", "user", "assistant", "tool"]
    content: str | list[dict]          # Plain text or structured multimodal content
    tool_call_id: str | None = None    # Links tool result messages to calls
@dataclass(frozen=True)
class ToolSpec:
    name: str                          # Globally unique, verb_object naming
    description: str
    parameters: dict                   # Parameter schema described in JSON Schema
    permission: Literal["read", "write", "network"]
@dataclass(frozen=True)
class Capability:
    max_context: int
    parallel_tools: bool               # Supports multiple tool calls in a single turn
    stream_reasoning: bool             # Streams reasoning output
    prompt_cache: bool                 # Supports prompt prefix caching
@dataclass(frozen=True)
class Event:
    kind: Literal["delta", "tool_call", "usage", "error"]
    payload: dict = field(default_factory=dict)
class ModelAdapter(ABC):
    """Normalize different API protocols into internal contracts."""
    name: str
    @abstractmethod
    def capabilities(self) -> Capability: ...
    @abstractmethod
    async def stream(
        self, messages: list[Message], tools: list[ToolSpec]
    ) -> AsyncIterator[Event]: ...

2. Replace Hardcoded Assumptions with Capability Negotiation

Different models differ in context length, parallel tool calling, reasoning-content output, caching mechanisms, and tool-call formats. Hardcoding these differences into business logic is a common form of technical debt. A more robust approach is to let adapters declare their capabilities, then negotiate and degrade at runtime as needed:

from dataclasses import dataclass
@dataclass(frozen=True)
class Requirement:
    max_context: int = 128_000
    parallel_tools: bool = False
    stream_reasoning: bool = False
    prompt_cache: bool = False
REGISTRY: dict[str, ModelAdapter] = {}
ROUTING: dict[str, list[str]] = {           # Declares preferred models and fallback chains per task type
    "code_edit": ["primary", "fallback"],
    "chat":      ["fast", "primary"],
    "review":    ["primary", "fast"],
}
def adapter(cls):
    REGISTRY[cls.name] = cls()
    return cls
def select(task: str, req: Requirement) -> ModelAdapter:
    errors: list[str] = []
    for name in ROUTING.get(task, ["primary"]):
        a = REGISTRY.get(name)
        if a is None:
            errors.append(f"{name}: not registered")
            continue
        c = a.capabilities()
        missing = [f for f, ok in {
            "context": c.max_context >= req.max_context,
            "parallel_tools": (not req.parallel_tools) or c.parallel_tools,
            "stream_reasoning": (not req.stream_reasoning) or c.stream_reasoning,
            "prompt_cache": (not req.prompt_cache) or c.prompt_cache,
        }.items() if not ok]
        if not missing:
            return a
        errors.append(f"{name}: does not satisfy {','.join(missing)}")
    raise RuntimeError("No model backend satisfies the requirements: " + "; ".join(errors))

This layer brings direct benefits:

• Fault isolation: when one backend is rate-limited or unavailable, the system can automatically fall back to a backup backend;
• Cost optimization: lightweight tasks such as summarization, classification, and code completion can be routed to lower-cost models;
• Regression comparison: during evaluation, different backends can be switched across the same test cases.

4. Context Engineering: Budgeting, Compression, and Caching

Context is the scarcest resource in an agent. In design, the first step is to set a total budget, then allocate proportions by role. An example is shown below:

BUDGET_RATIO = {               # Starting point for code editing; tune based on evaluation results
    "system": 0.10,            # System instructions: role, rules, and output format
    "tools":  0.15,            # Tool descriptions and parameter schemas
    "code":   0.35,            # Retrieved code and documentation
    "history": 0.30,           # Conversation history
    "headroom": 0.10,          # Reserve for model replies and tool outputs
}

1. Layering: Stable Content First, Volatile Content Later

The context should be organized in the following order:

1. System instructions;
2. Tool definitions;
3. Retrieved code or documentation;
4. Conversation history;
5. The latest user instruction.

This arrangement not only matches the model’s attention distribution, but also improves cache hit rates. Most APIs define prompt caching as a “longest common prefix” match. If system instructions and tool descriptions remain byte-for-byte stable, and history is only appended to rather than rewritten, the reusable prefix becomes longer, which can significantly reduce cost and time-to-first-token.

Therefore, two types of hidden damage should be avoided:

• Embedding content that changes every time, such as current timestamps or random task IDs, into the system prompt;
• Rewriting earlier messages when compressing history.

Both approaches can invalidate the entire prefix.

2. Compaction Order When the Budget Is Exceeded

When the context exceeds its limit, content should be trimmed by information value from low to high, rather than simply truncated:

def compact(history: list[Message], budget: int, count) -> list[Message]:
    """Trim low-value content first when over budget; preserve recent original messages."""
    recent, older = history[-6:], history[:-6]
    out: list[Message] = []
    for m in older:
        if m.role == "tool":
            continue                       # 1. Old tool outputs: lowest information value, largest size
        out.append(Message("user", f"[History summary] {summarize(m, count)}"))  # 2. Compress old dialogue into key points
    return trim(out + recent, budget, count)  # 3. Preserve recent originals first, then enforce the length limit

The reasoning behind this trimming order is based on information value:

• Old tool outputs, such as early directory listings and logs, have usually already been consumed by later steps;
• Earlier dialogue intent is often easier to recover from summaries than the most recent turns;
• Recent original messages usually contain the specific identifiers currently being processed, such as file paths, line numbers, and variable names. Once lost, the model can easily generate “hallucinated paths”.

3. Governing Tool Outputs

A single tool output can consume a large portion of the budget, so a unified convention is needed:

• Truncate and annotate: explicitly tell the model that the result has been truncated, and include the original length;
• Paginate or filter: provide parameters for keyword filtering and line-range reading;
• Prefer structured results: return compact structures first, such as diff fragments or match locations, rather than full files;
• Two-stage reading: for large files, first return a directory or signature overview, then let the model read specific fragments as needed.

5. The Tool Layer: Treat the Protocol as a Contract

Tool descriptions are the interface documentation between the model and the real system, and they should be treated as rigorously as public APIs:

• Naming: use a verb-object structure, ensure global uniqueness, and avoid near-synonym tools with overlapping semantics that make selection difficult for the model. Examples include search_code and apply_patch;
• Description: explain the purpose, applicable scenarios, non-applicable scenarios, and return structure, with short examples when necessary;
• Parameters: use JSON Schema to constrain types and required fields, explicitly list enum values, and make default values clear;
• Results: control length, keep structure stable, and include status fields rather than raw text.

On the execution side, a unified error semantic layer is also required. The key to model recovery is whether the model can receive “actionable information” when a failure occurs:

class ToolError(Exception):
    def __init__(self, code: str, message: str, recoverable: bool = True, hint: str = ""):
        super().__init__(message)
        self.code, self.recoverable, self.hint = code, recoverable, hint
def to_result(call, exc: ToolError | None = None, value=None) -> dict:
    if exc is not None:
        return {"status": "error", "code": exc.code, "message": str(exc),
                "recoverable": exc.recoverable, "hint": exc.hint}
    return {"status": "ok", "result": clip(value, limit=call.spec.max_result_bytes)}
async def execute(call, ctx) -> dict:
    if ctx.permission_level < call.spec.permission:
        return to_result(call, ToolError("PERMISSION_DENIED", f"{call.spec.permission} permission required"))
    try:
        value = await dispatch(call.name, call.arguments, ctx)
        return to_result(call, value=value)
    except ToolError as e:
        return to_result(call, e)
    except TimeoutError:
        return to_result(call, ToolError("TIMEOUT", "Execution timed out; retry with a smaller scope"))

Exceptions are converted into structured tool results and returned to the model, allowing the model to adjust parameters and retry instead of interrupting the entire session. In addition, several engineering details should be considered:

• Idempotency: write-type tools should carry request identifiers to avoid duplicate writes caused by timeout retries;
• Concurrency control: limit the number of tools executing at the same time, and serialize tools with read-write conflicts;
• Loop protection: set a maximum number of consecutive failures for the same tool and the same parameters, and escalate to human handling when the limit is exceeded.

6. Runtime and File I/O: Let the Agent Act Safely

1. Workspace Isolation and Change Tracking

• Each task should run in an isolated workspace or container instance, and the instance should be recycled after the task ends;
• Before writing, create a baseline snapshot of the workspace, such as a local index based on version control or a file-system snapshot, so that single-step rollback is possible;
• Model output changes should be applied as patches or diffs, written to disk by the tool layer, and displayed as diff views. Letting the model rewrite entire files wastes output tokens and increases the risk of accidentally overwriting unrelated code.

Lightweight validation should also be performed before writing to disk, so that errors can be exposed earlier:

def apply_edit(path: str, patch: str, check) -> dict:
    original = read_file(path)
    updated = apply_patch(original, patch)
    if updated is None:
        raise ToolError("PATCH_NOT_APPLIED", f"No target snippet found in {path}",
                        hint="Re-read the file and generate the patch based on its actual content")
    issues = check(updated, path)
    if issues:
        raise ToolError("CHECK_FAILED", f"{path} failed validation: {issues[:3]}", recoverable=False)
    write_file(path, updated)
    return {"diff": make_diff(original, updated)}

2. Permission Tiers and Least Privilege

Permissions should be divided according to the impact scope of operations:

• Reads and searches are allowed by default;
• Writes inside the workspace are controlled;
• Access to paths outside the workspace and network requests require explicit authorization;
• High-risk operations, such as deletion or executing external scripts, require human confirmation or mandatory sandboxing.

Credentials should not appear in the context. Authentication should be completed by the tool layer, and the model should only see desensitized results. Secrets should be managed by a secret-management service and issued as temporary credentials with least privilege for the task scope.

3. Reproducible Execution Environments

Dependency versions should be fixed, build scripts should be scripted, and environment variables should be declared explicitly, ensuring that the same code produces the same result on any execution machine. If the harness layer needs to be deployed independently, general cloud capabilities can be used to build it:

• Container platforms host execution instances;
• Object storage archives task artifacts and baseline snapshots;
• Message queues decouple task scheduling from execution;
• Logging services and monitoring alerts cover tool calls and model requests.

General-purpose cloud products such as Baidu AI Cloud’s container, storage, logging, and monitoring offerings can also be considered as infrastructure options for this type of architecture.

7. Observability and Evaluation: Make “Good Enough” Measurable

Agent behavior is probabilistic, and “it feels better” cannot be used as an acceptance criterion. A complete trace should be built for each task:

• Trace structure: one task is the root span, and each model request, tool call, and retrieval is a child span;
• Recorded fields: model name, input/output token counts, cache hits, latency, tool parameters and result summaries, and final status.

with tracer.span("task", task_id=tid) as root:
    root.set({"model": backend.name, "user": tenant_id})
    for step in loop(max_steps=30):
        with tracer.span("model_call", step=step) as s:
            reply = await call_model(...)       # Record input/output tokens and cache hits
            s.set({"tokens_in": reply.usage.input, "tokens_out": reply.usage.output})
        if reply.tool_calls:
            with tracer.span("tool_batch", tools=[c.name for c in reply.tool_calls]):
                results = await run_tools(reply.tool_calls, ctx)
            continue
        break

Based on this, two layers of evaluation should be established:

1. Offline evaluation set: anonymized samples drawn from real traces should cover typical tasks and historical failure scenarios. After each adjustment to prompts, tool descriptions, or context strategies, automatic regression should compare task success rate, average interaction rounds, cost per task, and P95 latency;
2. Online gray release: new versions should first be exposed to a small percentage of traffic, while monitoring tool failure rate, human intervention rate, and user interruption rate.

Evaluation results also have an implicit value: they provide a basis for model routing. When evaluation shows that most tasks can be completed by lower-cost models with the same success rate, the cost structure will change accordingly.

8. Conclusion: The Harness Is Engineering, Not Packaging

The idea that “the harness is the product” is essentially a reminder to engineering teams that the model determines the ceiling of capability, while the harness determines the actual delivery level. For implementation, teams can use five principles for self-checking:

1. Stable contracts: business logic should depend only on internal unified contracts, so that models can be replaced;
2. Explicit capabilities: differences should be handled through capability declarations and negotiation, rather than scattered conditional branches;
3. Controlled budget: context should have a clear allocation, trimming order, and caching strategy;
4. Recoverable failures: errors should be returned to the model in a structured and actionable way, supported by loop protection and human escalation;
5. Measurable effects: key metrics should be continuously tracked, and every adjustment should be validated through regression.

If these five points are met, the product experience can remain stable regardless of how the underlying model changes, and the team can focus its energy on the business logic that truly creates value.