Agent Harness Engineering: The Second Moat Beyond the Model

1. Reframing the Problem: Why We Need Harness More, Not Less, Once Models Are Strong Enough

Over the past year, discussions about AI agents have often drifted toward two extremes. One view holds that as foundation models keep getting stronger, the engineering scaffolding around them, the harness, will be flattened automatically. The other view holds that engineering details matter far more than model capability.

From conversations with frontline model teams and agent platform teams, a more grounded conclusion emerges: after model capability moves up the stack, the harness does not become obsolete. Its job changes.

In the early days, the harness’s main responsibility was to “fill gaps.” When the model could not call tools, engineers hardcoded prompt templates; when it could not hold long context, they built chunking and summarization; when it could not write files, they wrapped everything into a few fixed function calls. This kind of work naturally decays as models get stronger.

Once models have solid autonomous planning and tool-use abilities, the problem the harness must solve shifts to determinism and controllability:

• If the same task runs a hundred times, are the success rate and latency stable?
• Will a single erroneous tool call cause irreversible data damage?
• Does the right information appear at the right moment in the context of a complex task?
• Can the cost of a long-chain task be predicted and capped?
• After a failure, can you pinpoint which step or which piece of context caused it?

None of these answers live in the model weights. They live in the harness. Put differently, the model sets the ceiling of an agent’s capability, while the harness determines what fraction of that ceiling can be reproduced reliably. The gap between a task that works once in a demo environment and an agent that runs a task tens of thousands of times a day in real production is exactly this engineering.

2. The Anatomy of a Harness: A Practical Capability Map

Breaking the harness into seven layers is more actionable than vaguely calling it “scaffolding.”

1. Context Layer

This decides what the model “sees” at any given moment. It includes how system instructions are organized, the granularity of tool descriptions, the compression strategy for message history, the injection timing of retrieval results, and intermediate-state summaries for long tasks. The core principle is to prioritize the context budget for the information needed for the current decision, rather than dumping everything potentially relevant in at once.

Three common anti-patterns show up here in practice: stuffing all documentation into the system prompt; failing to truncate tool results, which blows up the context; and keeping the previous round’s erroneous output verbatim, so the model keeps getting disturbed by noise.

2. Tooling & Runtime Layer

This covers tool registration and description, parameter validation, execution timeouts, concurrency control, result normalization, and the isolation and resource reclamation of the execution environment (local process, container, remote sandbox). The tool description is itself a kind of prompt; if it is vague, the model’s call accuracy drops significantly.

3. State & Memory Layer

Three kinds of storage need to be distinguished: short-term in-session state (current task progress), long-term cross-session memory (user preferences, project conventions), and retrievable external knowledge (documents, code, tickets). The key design point is that writing and evicting long-term memory both need explicit policies; otherwise memory degrades into an unstructured pile that actually lowers accuracy.

4. Governance Layer

Least privilege, action tiering, secondary confirmation for high-risk operations, and audit logs. A mature practice is to tier tools as “read-only / writable / irreversible” and force irreversible actions through human confirmation or an approval chain, rather than relying on a single “please be careful” sentence in the prompt.

5. Orchestration Layer

Single-agent loops, task decomposition, sub-agent scheduling, and human-in-the-loop intervention points. The goal of orchestration is not to make the flow as complex as possible, but to keep each subtask’s context as clean as possible and to isolate failures.

6. Evaluation Layer

Offline evaluation sets, regression tests, online canary releases, and bad-case backflow. This is the layer most often skipped and the one that best reflects a team’s engineering maturity. Without an evaluation set, any change to the harness is a blind change.

7. Observability & Cost Layer

Trace tracing, token and tool-call metering, per-task cost, P95 latency, and failure attribution. In real production, “why is this agent slower today than yesterday” is often resolved by tracing a single full run.

A minimal harness skeleton can be abstracted like this:

class Harness:
    def __init__(self, model, tools, memory, policy, evalset):
        self.model = model          # pluggable inference backend
        self.tools = ToolRegistry(tools)
        self.memory = MemoryStore(memory)
        self.policy = Policy(policy)  # permission tiers / high-risk action blocking
        self.evalset = evalset        # regression baseline before each release
    def run(self, goal, ctx_budget=32000):
        state = {"goal": goal, "steps": [], "spent": 0}
        while not self.is_done(state):
            prompt = self.compose(state, ctx_budget)   # context assembly
            action = self.model.plan(prompt)           # planning and tool selection
            decision = self.policy.check(action)       # permission decision
            if decision.needs_confirm:
                action = self.wait_human(action)       # human confirmation point
            result = self.tools.invoke(action, timeout=30)
            self.memory.append(state, action, result)  # persist state for traceability
            state["spent"] += self.cost(prompt, result)
            if state["spent"] > self.budget:           # cost cap
                return self.degrade(state)
        return state

Not a single line in this skeleton is “model capability,” yet every line directly shapes production behavior.

3. General Harness vs. Vertical Harness: Layered Coexistence, Not Mutual Consumption

One question comes up repeatedly in the industry: will general-purpose agents swallow vertical agents?

A more precise judgment is this: a general harness will absorb the repetitive runtime pieces of vertical scenarios, while the value of a vertical harness will converge on “domain knowledge + toolchains + evaluation sets.”

General harnesses excel at model-agnostic infrastructure: context management, tool-call protocols, sandboxed execution, state persistence, trace tracing, and cost accounting. This part is highly repetitive, and building it in-house is duplicated labor for any team.

A vertical harness truly accumulates three things that are hard to copy:

• Private tools and data sources: internal business systems, proprietary data, and industry rule engines;
• Domain constraints and failure-mode libraries: knowing which operations are absolutely off-limits in your business;
• High-quality evaluation sets: a few hundred annotated real tasks that are far closer to your production distribution than any leaderboard.

So the two are not a replacement relationship but a combination of “general runtime + domain plugins.” A team’s healthy structure is to reuse mature infrastructure for general capability as much as possible and concentrate vertical capability on knowledge and evaluation, rather than rewriting the scheduler.

4. Why Model Teams Should Also Build Agents

There is a plain engineering reason behind this: real task distribution is the scarcest data for model iteration.

Model training and evaluation depend on task distribution, and there is a large gap between public benchmarks and real business. When a team operates its own agents, it continuously collects three kinds of high-value feedback:

1. Which failures come from model reasoning and which come from harness defects (attribution data);
2. The distribution and error patterns of real tool calls (tool-use capability data);
3. Whether the intermediate steps of long-chain tasks drift from the goal (process supervision data).

The second kind of data is especially critical: away from a real execution environment, it is hard to judge how good a model’s multi-step planning actually is. This is also the real driver behind “unified model + agent” development in terms of R&D efficiency, not merely market packaging.

From an engineering perspective, there is also clear room for co-optimizing the model and the harness: context-length strategy, tool-description format, how the reasoning process is exposed, and failure-retry strategy all feed back into model-side training objectives and inference configuration. When the two sides are developed separately, these parameters can only be coordinated through documentation; when developed together, they can form a closed loop.

5. The Opportunity Boundary of Agent Infra: Filter by “High-Frequency” and “Strong-Constraint”

For infrastructure startups around agents, common opportunity points can be evaluated along two dimensions: call frequency (the higher, the more worth platformizing) and constraint strength (the stronger, the harder for model capability to replace).

This yields a priority order:

• High priority: execution sandbox and isolation (strong safety constraints, and needed more as models get stronger); tool gateway and permission governance (a hard requirement for enterprise compliance); evaluation and regression platforms (every team is rebuilding them); observability and cost accounting (inevitable once the business scales).
• Medium priority: memory and state storage; orchestration and scheduling; multi-model routing and fallback; agent identity and key management.
• Low priority / easily flattened: single-point prompt-template management; simple task queues; generic RAG assembly, all of which can be absorbed directly by upper-layer frameworks or model capability.

A counterintuitive but important judgment: the stronger the model capability, the higher the value of sandboxing and permission governance. Because the more a model can do, the larger the blast radius of a single misoperation. Security is not a “patch for insufficient capability” but a “precondition for releasing capability.”

6. Multi-Agent Collaboration: When to Split and When Not To

Multi-agent systems are often treated as the default architecture, but introducing them carries costs: context-synchronization overhead, error propagation across agents, higher debugging difficulty, and higher total cost.

Three legitimate reasons to split:

1. Hard need for context isolation: the main agent does not need to see all the details of a subtask, only the conclusion. For example, code retrieval, data analysis, and document reading each run in their own context and return only structured results.
2. Permission boundaries need physical isolation: different agents use different credentials and different tool sets, reducing the risk of lateral privilege escalation.
3. Clear parallelization gains: subtasks with no dependencies can run concurrently, shortening end-to-end latency.

The signals for when not to split are equally clear: the task itself has strong serial dependencies; subtasks need to frequently exchange full context (in which case communication cost exceeds the benefit); or the team does not yet have a stable single-agent evaluation baseline (multi-agent systems only make attribution harder).

A pragmatic evolution path is to first make single-agent evaluation and observability solid, then introduce a limited orchestration of “one main agent + several read-only sub-agents,” and only then consider dynamic orchestration. Skipping the first two steps and going straight to multi-agent orchestration usually buys you a harder-to-debug black box.

7. Distilling Experience into Skills: Engineering Team Knowledge Assets

An increasingly common practice is to consolidate high-frequency, stable workflows into reusable skills, rather than describing them ad hoc with prompts each time. A skill is essentially a self-contained capability package, usually made up of four parts:

my-skill/
  instruction.md      # when to use, input/output contract, prohibitions
  reference/          # domain knowledge, templates, examples
  scripts/            # deterministic logic goes to code, not the model
  evals/              # 10-50 acceptance cases and pass criteria

Design points:

• Give the deterministic parts to code. Logic such as format conversion, field validation, and numeric computation should be written as scripts, which is more reliable and cheaper than having the model “compute it” every time.
• Write failure experience into constraints. The most valuable content of a skill is often not “how to do it” but “which ways of doing it are wrong.”
• Ship with evaluation. Without acceptance cases, you cannot tell whether a change is an improvement or a regression.
• Keep the granularity moderate. Too large a skill is hard to maintain; too small and it degenerates into an ordinary tool wrapper.

The strategic implication is that a team’s core assets will migrate from “prompt tricks” to “versionable, testable, distributable capability packages” that go through the same review and release process as code.

8. Compliance and Security: Two Real Issues That Cannot Be Avoided

Account and identity compliance. Some coding and agent tools now begin to require real-name or business-entity verification, driven by the needs of call-volume attribution, abuse prevention, and compliance auditing. Engineering-wise, this should be treated as an “identity and credential management” problem: use separate credentials for different environments, manage credentials through a unified store rather than hardcoding them in config or scripts, and keep all calls logged and auditable. Equally worth watching is leakage in build artifacts and workspaces: exposing credentials, internal addresses, or proprietary code inside uploaded or shared artifacts is the most realistic risk surface for these tools.

Least privilege. The permissions an agent holds by default should be lower than those of a human operator. High-risk actions (deleting data, changing production config, sending messages externally) must go through approval or secondary confirmation, are disabled by default, and should be explicitly enabled with a validity period when needed.

9. A Practical Evaluation System and Build Checklist

Do not judge harness iteration by “it feels better.” It is recommended to establish four groups of metrics:

• Success rate: task-level end-to-end pass rate, bucketed by task type;
• Stability: variance in pass rate over N repetitions of the same task, plus P95 latency;
• Economics: per-task token consumption, number of tool calls, and per-task cost;
• Security: high-risk action interception rate, number of privilege-escalation attempts, and audit coverage.

For build order, here is a pragmatic checklist:

1. First build the evaluation set (start with 100 real tasks); do not touch the architecture without one;
2. Then add full-chain tracing so that every step’s inputs and outputs, latency, and cost are inspectable;
3. Next, solidify the context-assembly and tool-description conventions into team agreements;
4. Then fill in sandbox isolation, permission tiering, and high-risk action approval;
5. Finally, introduce advanced capabilities such as multi-agent orchestration and dynamic routing.

For teams that want to fill in infrastructure quickly, general cloud capabilities can directly cover most of it: container platforms and function compute handle sandboxed execution and elastic scaling; object storage and message queues handle state persistence and async decoupling; log services and monitoring/alerting handle observability; and key management and private networking handle credential hosting and network isolation. What a team truly needs to build in-house should be concentrated in three areas: domain tools, private data integration, and evaluation sets, which are also the only parts that general infrastructure cannot replace. The capabilities Baidu AI Cloud offers in the general categories above can serve as a reference direction when selecting such infrastructure.

10. Conclusion

Back to the original question: discussing the harness is essentially discussing how to turn uncertain model output into deliverable, auditable, and billable engineering artifacts. The model decides what an agent can do; the harness decides how far it can be trusted to do it.

Three takeaways: stronger models will not eliminate the harness, but will shift its center of gravity from “filling capability” to “guaranteeing determinism”; general and vertical harnesses are a layered relationship, and the moat lies in private tools, domain constraints, and evaluation sets; multi-agent systems are an orchestration means, not an architectural faith, and teams without a single-agent baseline should not adopt multi-agent systems first.