Agent Harness: Why Execution Architecture Determines Agent Performance

1. What Is an Agent Harness?

A question that comes up frequently among agent development teams is this: what ultimately determines how well an agent performs on real tasks? Model vendors tend to attribute the result to the capabilities of the underlying model, but many agent teams have observed that the execution framework matters just as much. The same model, when placed inside different execution frameworks, can show completion rates on complex tasks that differ by several times.

The engineering system that wraps around the model and enables it to actually perform work is commonly called the Harness, or execution harness. It operates inside the same inference loop as the model. It assembles input, schedules model inference, executes tool calls, writes results back into the context, and decides when to terminate. A typical agent turn can be summarized in four steps:

1. Assembly: combine system instructions, user goals, historical messages, the current environment state, and available tool descriptions into a context window.
2. Inference: the model outputs reasoning, a tool-call plan, or a final response.
3. Execution: the framework parses tool calls, executes them in the appropriate environment, and writes the results back into the context.
4. Evaluation and looping: determine whether the task is complete, whether the plan needs adjustment, or whether another turn is required.

In other words, the Harness can be understood as the runtime of an agent. The model is the engine, while the Harness is the chassis, transmission system, and steering mechanism. No matter how good the engine is, if the transmission is mismatched, the vehicle will not go fast.

1.1 From Script Orchestration to Execution Harness

The form of agent systems has evolved through several clear stages:

• Script orchestration stage: developers predefined fixed workflows, such as retrieval first, summarization next, and generation last. The large model handled only specific natural language processing nodes. The flow was controllable, but a new task often required rewriting the process.
• Model-driven orchestration stage: the model planned steps and selected tools according to the goal. Autonomy increased substantially, but failure modes became harder to predict.
• Execution harness stage: the Harness becomes the center of attention. The focus shifts to how agents can work reliably over long, multi-step tasks. Engineering effort moves from “write a workflow” to “design an environment in which the model can work stably.”

This shift is driven by improvements in model reasoning. Once a model can think before acting, including scenarios where mid-sized and open reasoning models are available through open weights, engineering teams gain more freedom to include the reasoning process itself in orchestration. This expands the design space for the Harness.

2. What Are the Main Components of an Agent Harness?

A mature Harness usually contains six tightly coupled components. Poor design in any one of them can drag down overall performance.

2.1 Lifecycle and Session State Management

The agent’s full lifecycle, from startup to completion, needs to be managed:

• Initialization: load system instructions, available tools, environment configuration, and user preferences.
• Execution: maintain runtime state, such as the current subgoal, completed steps, retry counts, stage durations, and token consumption.
• Recovery: after crashes, restarts, or network interruptions, resume from the most recent consistent state instead of starting over.
• Termination: distinguish between task completion, user cancellation, budget exhaustion, timeouts, and safety policy blocks, and handle each case appropriately.

In implementation, a useful pattern is to record every model inference, every tool call, and every result written back into the context as an immutable event, appended to a log. State can then be represented as a fold over those events:

state = fold(events)

This makes recovery easier and also supports replay and debugging. Event sourcing is not necessarily required in every system, but “every step should be replayable” is a practical baseline.

2.2 Context Engineering

Context is the only world the model can see. One of the Harness’s core responsibilities is deciding what to put into that world. Context engineering usually involves four subproblems:

• Assembly: organize system instructions, tool descriptions, conversation history, task-relevant information, and the current environment state.
• Compaction: when content approaches the context limit, summarize, truncate, or archive it without losing critical information.
• Recall: retrieve tool outputs, document fragments, or past experience and inject them back when needed.
• Focus: control the model’s attention and reduce interference from irrelevant content.

A practical principle is: organize first, compact later.

# Example pseudocode: organize context entries according to budget
COMPACTION_BUDGET = 6000  # Input token budget available for context organization. Tune this value according to the actual scenario.
def assemble_context(session, budget: int = COMPACTION_BUDGET) -> dict:
    history = []
    # 1) Compact: keep only summaries of older messages.
    # The final version and the summaries themselves usually still need to be retained.
    for summary in session.old_summaries():
        history.append({"role": "user", "content": f"[Historical summary] {summary}"})
    # 2) Trim: for a single overly long output, such as logs or test reports, keep only the beginning and end.
    for msg in session.recent_messages():
        history.append(trim_middle(msg, keep_chars=4000))
    # 3) Recall: retrieve notes and documents most relevant to the current subgoal based on task semantics.
    for note in session.recall_relevant(limit=5):
        history.append({"role": "user", "content": f"[Reference] {note}"})
    # 4) Validate: if the budget is still exceeded, drop the oldest droppable entries one level at a time.
    while estimate_tokens(history) > budget and session.droppable(history):
        drop_oldest_droppable(history)
    return {
        "system": session.system_prompt,
        "tools": session.enabled_tools(),
        "messages": history,
        "env": session.environment_digest(),  # Environment summary: directories, language runtimes, dependency versions, etc.
    }

Compaction should not be one-directional. The details of decisions made in recent turns are often exactly what subsequent execution depends on, so they should be preserved completely. For especially important results, such as the version number of a passing test, the Harness can “pin” a summary message and keep it fixed so that later compaction does not rewrite it. The specific partitioning ratios and tool description methods should be validated repeatedly through evaluation sets rather than copied from other systems.

Prefix stability also directly affects inference cost. Many mainstream model services support caching of repeated prefixes and offer discounts for requests that meet cache conditions. Slowly changing content, such as system instructions, tool definitions, and long-term memory, should be placed at the beginning. Frequently changing environment state should be placed later. This increases the length of the repeated prefix and improves cache hit rates, reducing both latency and cost.

2.3 Tools and Execution Environments

Tool calls are the channel through which a model “acts.” Their design determines the capability boundary of an agent.

• Interface design: use as few parameters as possible, with clear semantics. Describe when the tool should be used and what kinds of failures may occur. Vague descriptions are one of the most common causes of tool misuse.
• Result format: output should be structured and concise, filtering out noise. If a thousand-line log is pushed directly back into the context, it risks truncation and quickly consumes budget.
• Granularity choice: interfaces that are too coarse, such as a single universal endpoint, increase misuse. Interfaces that are too fine, such as one endpoint per sub-step, increase the number of calls and latency. Tools can be grouped into cohesive capability clusters by task, and a “batch execution” interface can be provided when necessary to reduce round trips.
• Idempotence and retries: read operations can usually be retried safely. Write operations require idempotence keys or “confirm before execution.” Otherwise, a transient network failure could lead to duplicate orders or duplicate submissions.

The execution environment determines what an agent can actually do. Read-only code analysis is very different in risk from an environment that can modify code, run tests, or operate a browser. A common industry approach is layered isolation:

1. Read-only analysis: restricted file views and read-only commands.
2. Controlled writes: modifications inside a temporary working directory or container, restricted network access, and auditable, rollback-able write operations.
3. High-privilege operations: actions involving production systems, funds, or external releases must require human confirmation or dual review.

2.4 Memory and State Persistence

Conversation context is “short-term memory.” What is retained across sessions is “long-term memory.” Long-term memory usually falls into three categories:

• Factual memory: stable information such as user preferences, project conventions, and business rules.
• Experiential memory: successful paths and lessons learned from past tasks, for example, “unit tests for this module require a specific type of preloaded data.”
• Procedural memory: operation sequences that have been validated and can be solidified into fixed processes.

The key decisions are not whether to record memory, but what to record, when to retrieve it, and how to update it. Writing too much creates retrieval noise that can bury useful information. Writing without updating allows outdated conclusions to continue causing errors. A safer approach is to attach source, time, and confidence level to each memory entry, periodically merge conflicting entries, and decide what to inject based on retrieval relevance rather than write time.

2.5 Evaluation and Feedback Loops

Without evaluation, there is no clear direction for improvement. Harness evaluation should cover at least four dimensions:

• Task success rate: for a fixed task set, what is the pass rate on the first attempt, and what is the pass rate after retries?
• Trajectory quality: number of steps, tool calls, and error retries, used to detect detours or infinite loops.
• Resource consumption: token usage, elapsed time, and cost per task.
• Security and compliance: whether there are unauthorized accesses, leakage of sensitive data, or execution of dangerous instructions.

The principle is: offline evaluation first, online gradual rollout second. Offline cases help control regressions. Gradual rollout, exposing a new version to a small portion of traffic first, validates behavior under real-world distributions. Evaluation cases should cover typical paths and known failure cases, and they should be continuously expanded based on production issues.

2.6 Permissions, Identity, and Auditability

Once an agent can execute real operations, it is no longer just a “conversational tool.” It becomes an executor that needs a clear identity within an organization. At least four things must be addressed:

• Least privilege: each task should request only the minimum permissions needed to complete it, and those permissions should be revoked when the task ends.
• Human gates: irreversible or high-impact operations should require confirmation.
• End-to-end auditing: who performed what operation, at what time, with what identity, and with what result should all be traceable.
• Identity management: agent access credentials should be centrally managed by a platform and issued as short-lived credentials per task, rather than long-term keys being scattered across scripts or configuration files.

3. Will Models Eventually “Eat” Agents?

This is one of the most frequently asked questions. A reasonable answer is that improving model capabilities will change the shape and focus of the Harness, but will not make it disappear. There are four reasons:

First, the execution environment is not inside the model weights. Reading files, running commands, querying databases, and operating browsers all happen outside the model. No matter how capable the model is, something still needs to execute those actions, handle timeouts, and validate results.

Second, context limits will always remain scarce. Larger windows raise the ceiling, but cost and attention dilution still exist. A single complex code review task can generate hundreds of thousands of tokens of tool output. Without management, this can quickly exceed the budget.

Third, permission and responsibility boundaries require engineering constraints. A model does not automatically gain permissions just because it becomes “smarter.” Approval workflows for high-risk operations, data classification, and audit trails are fundamentally organizational and compliance requirements. They must be implemented by the Harness.

Fourth, attributing failures requires observability. When a model produces a wrong answer, engineers need to distinguish among reasoning errors, tool-call parameter errors, environment failures, and missing context. Without event-level logs and tracing, the problem cannot be diagnosed.

At the same time, the reverse effect also matters. Stronger models allow the Harness to become “thinner”: fewer fixed scripts, fewer heuristic rules, and more reliance on autonomous model decisions. The relationship is therefore co-evolution, not zero-sum replacement.

4. General-Purpose Agents and Vertical Agents

The distinction between general-purpose and vertical agents usually does not lie in the model itself, but in how much domain constraint is embedded in the Harness. A vertical agent for data analysis, for example, creates value not because the model understands SQL better, but because the Harness comes preloaded with table schema knowledge, metric definitions, data permission rules, and result validation templates.

The two are better understood as a “base plus plugin” relationship. A general-purpose Harness provides the inference loop, tool calling, context management, and auditing capabilities. Vertical capabilities are then layered in through domain-specific toolsets, domain memory, domain evaluation sets, and domain workflow templates. For teams, the key judgment is whether vertical knowledge is “long-tail and high-frequency” or “clearly defined and slow-changing.” The former is suitable for continuous accumulation through memory and retrieval. The latter is suitable for solidification into reusable processes.

5. Why Model Teams and Agent Teams Should Both Work on Both Ends

Model teams build agents because feedback from real tasks is one of the most important data sources for model iteration. Only by running real long-horizon tasks can they learn where the model tends to “drop the ball” during multi-step reasoning. Agent teams adapt or even fine-tune models because general-purpose models may not have the optimal tool-call format, output length, or latency characteristics for a team’s specific scenario. Adaptation may be required at the level of inference parameters, prompt templates, or even post-training.

However, there is a clear boundary: building a foundation model from scratch and adapting a model to a specific scenario are two levels of investment. Most teams should focus on adaptation and Harness design rather than trying to recreate a general-purpose large model.

6. When Is Multi-Agent Collaboration Worthwhile, and When Is It a Burden?

Multi-agent collaboration is often described as “division of labor,” but in engineering it is more accurately a set of tradeoffs around context isolation and parallelism:

• Benefits: context isolation, where subtasks complete in their own contexts and only return conclusions to the main agent; parallel execution; and role specialization, such as separating planner, executor, and reviewer.
• Costs: coordination overhead, information loss during context transfer, error amplification across the chain, and increased debugging difficulty.

A practical rule of thumb is this: if a single agent can complete the task stably, do not introduce multi-agent complexity. Only when a task can be clearly divided into relatively independent subgoals, and the budget of a single context cannot accommodate all the work, does splitting become worthwhile. A common and relatively stable structure is for the main agent to handle goal decomposition, task dispatch, and result aggregation, while sub-agents are responsible only for completing work within limited scope and returning structured conclusions.

# Example pseudocode: main agent dispatches subtasks and receives only structured conclusions
plan = planner.decompose(goal)                      # Decompose the goal into mutually independent subgoals.
results = pool.map(
    plan.subtasks,
    lambda t: worker.run(t, tools=t.allowed_tools, max_turns=15),
)
return summarizer.merge(goal, results)              # The main agent merges conclusions and does not absorb all intermediate details.

7. Where Are the Opportunities in Agent Infrastructure?

When competition at the agent application layer becomes intense, structural opportunities often appear in the infrastructure layer. These can be summarized into five general capabilities:

1. Sandboxing and execution environments: isolated environments that can start in seconds, support snapshots, and roll back changes. These are essential for long-running agents.
2. Memory and context storage: long-term memory services that support semantic retrieval, version management, and permission-based isolation.
3. Observability: tracing, replay, and cost attribution for inference plus tool-call chains, which differs from traditional APM.
4. Evaluation platforms: construction of task-level evaluation sets, regression testing, and production quality monitoring.
5. Identity and permission governance: agent identity, credential management, dynamic authorization, and operation auditing.

Cloud providers such as Baidu AI Cloud can provide general infrastructure such as container platforms, object storage, message queues, logging services, monitoring, and alerting, which can serve as the foundation for these scenarios. Specific technology selection should be based on actual testing rather than brand perception. For example, teams can validate startup latency and isolation strength by building their own load tests.

8. Security and Compliance: Questions That Cannot Be Avoided

Identity and Traceability

For enterprise-facing agent products, requiring identity verification for users is not merely a technical restriction. It is a responsibility-attribution requirement. Once an agent can operate code repositories, production systems, or externally published content, the platform needs to be able to answer: “Who authorized this action?” For enterprise users, this means agent accounts should be incorporated into existing identity and access management systems rather than managed separately.

Client-Side Code Readability

Many agent products are distributed as clients or local tools. Their prompts, tool-call logic, and orchestration strategies can be read or reverse-engineered. This creates two types of risk:

1. Intellectual property risk, such as leakage of prompts and workflows.
2. Security risk, such as exposing the attack surface. If attackers understand the toolset and permission model, they are better positioned to craft inducement-based inputs.

Mitigation measures include moving sensitive logic and credentials to the server side, applying code obfuscation and integrity checks to clients, and, most importantly, setting defensive boundaries at the permission layer rather than the prompt layer. Relying on “the prompt says this is not allowed” to control risk is itself a fragile design.

Credential Leakage

Agents read configurations and execute commands inside environments. Once the context is polluted by malicious content, secrets may be sent back externally. In practice, teams should use short-lived credentials instead of long-term keys, redact sensitive information from tool outputs, restrict network egress through allowlists, and provide an independent confirmation channel for high-risk operations.

9. Turning Methods into Reusable Assets

A worthwhile practice is to turn repeated individual and team workflows into process descriptions that agents can directly invoke, instead of verbally explaining them every time. These assets usually contain three parts:

• Applicability: when the workflow should be used.
• Operational steps: what to do and which tools to use.
• Acceptance criteria: how to determine whether the work was done correctly.

The value of this approach is that it turns “implicit experience” into “engineering assets that can be versioned, evaluated, and shared.” These assets can be understood as a combination of an operating manual and test cases for agents. As such assets accumulate, organizational marginal costs can decline significantly, and small teams can produce output that is substantially higher than traditional structures would allow. This is one engineering expression of the “extremely small team” discussion. However, it is important to recognize what can and cannot be replaced: processes and judgment criteria can be sedimented; goal definition and responsibility cannot be replaced.

10. How to Tell Whether the Bottleneck Is the Model or the Harness

Here is a practical troubleshooting order:

1. Quantify first: measure success rate, average steps, token cost, and failure-type distribution on a fixed task set.
2. Inspect the failure distribution: if failures concentrate on tool-parameter errors, missing context, or repeated retry loops, prioritize improving the Harness. If failures concentrate on reasoning errors or planning errors, consider changing the model or adjusting prompts.
3. Run single-variable experiments: change only one variable at a time, such as model, prompt, tool description, or context strategy. Otherwise, conclusions will not be reliable.
4. Watch cost and latency: Harness design often has a larger impact on these two dimensions than switching models does.

Conclusion

When people discuss the Harness, they are ultimately discussing one thing: how to make a probabilistic reasoning system produce deterministic results stably in an uncertain environment. The model provides the upper bound of capability, while the Harness determines how much of that capability can actually be delivered.

Debates about model capability gaps, open-source versus closed-source models, and changes in agent paradigms are all important background context. But for individual teams, more practical questions are: how to manage context, how to design tools, how to constrain permissions, and how to review failures. Doing these engineering details solidly is closer to the answer than chasing any single model or any single framework.