Agent Engineering in Practice: The Real Bottleneck Is Infrastructure, Not the Harness

1. A Counterintuitive Conclusion: An Agent’s Ceiling Isn’t Set by the Harness

In discussions about building agents, one judgment keeps recurring: the hardest part is “harness engineering” — the whole scaffolding wrapped around the model, including tool definitions, prompt organization, context trimming, loop control, and result validation.

That judgment almost always holds during prototyping. When you have one demo, one terminal, and one dev machine, every problem looks like “the model isn’t obeying” or “the prompt isn’t good enough.” You tune one version of a prompt, success rate goes from 40% to 55%, the reward is immediate, and the feedback loop is measured in minutes.

But the moment that same prototype takes on real traffic, the problem list changes wholesale. What actually stalls a project is usually not that the model answers poorly, but problems like these:

• Sandbox sessions drop unexpectedly during long tasks; the task is half done and the scene is gone.
• Context and intermediate artifacts are lost, re-runs are expensive, and there is no way to resume from a checkpoint.
• The task shifts from “synchronously waiting for one answer” to “running asynchronously for tens of minutes,” and the caller has no way to see its state.
• When something goes wrong, you can’t localize it — you don’t know what the model saw, which tool it called, or where it went off track.
• Cost, concurrency, quota, and rate limiting are not isolated, so one runaway loop drags down the entire service.

This is what gets called the “infrastructure wall.” Its defining trait: completely invisible during prototyping, erupting all at once at scale, and impossible to route around with a better prompt.

2. Clarifying the Concepts: Where the Harness Ends and Agent Infrastructure Begins

To discuss this cleanly, separate the two layers first.

2.1 The Harness Layer: A Tightly Model-Coupled Adaptation Layer

The harness can be understood as the adaptation layer between model capability and application goals. Its typical responsibilities include:

• Tool contracts: translating external capabilities into call descriptions the model can understand, including parameter structure, error semantics, and return-value trimming strategy.
• Context management: deciding which history enters the window, how to compress it into summaries, and which fragments to retrieve and inject.
• Control loop: after one round of inference, whether to continue, correct, or converge on output — plus how to define iteration limits and termination conditions.
• Output validation: structured parsing, field completion, retry on failure, and fallback paths.

The harness is highly correlated with model behavior. Move the same harness to a newer generation of models and the result usually isn’t “worse” — it’s wrong-shaped: tool descriptions that are too fine-grained end up suppressing the model’s planning ability, or implicit reasoning habits the harness was relying on simply disappear.

2.2 The Infrastructure Layer: A Model-Agnostic Load-Bearing Layer

This layer is concerned with “where the task runs, where state lives, what happens when things fail” — largely independent of which model you use.

Capability domain	Generic capability categories	Core problem to solve
Execution environment	Container platforms, function compute, sandbox instances	Isolation, lifecycle, resource limits
State and files	Object storage, file systems, databases	Intermediate artifacts, resume-from-checkpoint
Memory	Vector retrieval, key-value stores	Context continuity across sessions
Scheduling	Message queues, task orchestration, scheduled triggers	Long-running tasks, async callbacks, retries
Observability	Log services, distributed tracing, monitoring and alerting	Attribution, replay, cost accounting
Security	Permission models, secret management, network egress control	Least privilege, audit, data boundaries

The key judgment: the harness sets an agent’s capability ceiling, while infrastructure sets its availability floor. The ceiling can be approached through iteration; if the floor collapses, everything goes to zero.

3. Why the Infrastructure Wall Always Shows Up Late

3.1 The orders-of-magnitude gap in feedback cycles

Prompt iteration has a feedback cycle measured in seconds to minutes. By contrast, a problem like “the sandbox was reclaimed at minute 47 of the task” may need hours or days to reproduce even once. Human attention is naturally drawn to short-cycle feedback, so long-cycle problems are systematically underestimated.

3.2 Failure modes are combinatorial

A single point of failure is easy to handle. What makes agents painful is combinatorial explosion: network jitter + tool timeout + retry logic + state never persisted, which finally surfaces as “the task silently vanished.” In the logs this looks like a model problem; the actual root cause is in the orchestration layer.

3.3 Team skill mismatch

Agent teams are usually made up of people with backgrounds in algorithms and application engineering — naturally strong on the harness, weak on distributed systems operations. Infrastructure problems need exactly the latter.

4. Four Typical Collapse Points and General Remedies

4.1 Execution environment: sandbox lifecycle and “disconnect means amnesia”

Long tasks are the norm for agents. A single code-repair task may involve dozens of tool calls plus several compile and test cycles. If the execution environment is bound to the task lifecycle as “one question, one answer,” then losing a single heartbeat makes the scene unrecoverable.

The general remedy is to decouple the execution environment from the inference loop:

inference loop (control plane)  <--->  task store (state)
        |
        +--->  execution environment (data plane, rebuildable and snapshotable)

The essentials: allocate a persistent working directory per session; snapshot after critical steps (filesystem snapshot or upload artifacts to object storage); allow the execution environment to be reclaimed and then restored from a snapshot; record every tool call as a structured event rather than leaving it only in the context.

4.2 State and memory: treat “context” and “state” as two different things

Many implementations put all their hope in the context window, which produces two consequences: the window gets stuffed with history, and state cannot survive a process restart.

A more robust approach splits storage into three classes:

• Short-term working memory: the current task’s goal, constraints, and a summary of completed steps, with a lifetime tied to the task.
• Long-term memory: cross-session user preferences and project conventions, stored independently and retrievable on demand.
• Artifacts and evidence: code patches, test output, retrieved document fragments — stored in object storage, with only references and summaries kept in context.

A sketch of the state structure:

{
  "task_id": "t-8f21",
  "goal": "fix failing boundary conditions in unit tests",
  "constraints": ["do not change public interface signatures", "keep existing dependency versions"],
  "steps": [
    {"idx": 1, "action": "read_file", "ref": "s3://bucket/t-8f21/ctx/src_utils.md"},
    {"idx": 2, "action": "run_tests", "result_ref": "s3://bucket/t-8f21/evidence/test_01.log"}
  ],
  "checkpoint": {"at_step": 2, "sandbox_snapshot": "snap-7c19"},
  "budget": {"tool_calls_used": 12, "tool_calls_limit": 60}
}

With a structure like this, resuming a task is just “read state + rebuild environment,” not “hope the model still remembers.”

4.3 Asynchronous execution: from “request-response” to “job-event”

Agent task durations follow a long-tailed distribution. Serve them with a synchronous interface and you will inevitably hit timeouts, broken connections, and duplicate submissions. The common approach is to model an agent run as an asynchronous job:

1. Submit the job and return a job identifier immediately.
2. Drive execution with a state machine: ACCEPTED → QUEUED → RUNNING → WAITING_HUMAN → SUCCEEDED / FAILED / CANCELLED.
3. Push progress events through a message queue or callback, supporting incremental consumption.
4. Fetch results via signed links or a query API, guaranteeing idempotency and replayability.

A minimal state machine constraint:

ALLOWED = {
    "QUEUED":     {"RUNNING", "CANCELLED"},
    "RUNNING":    {"WAITING_HUMAN", "SUCCEEDED", "FAILED", "CANCELLED"},
    "WAITING_HUMAN": {"RUNNING", "CANCELLED"},
    "SUCCEEDED":  set(),
    "FAILED":     {"QUEUED"},   # retry only allowed after an explicit rebuild
    "CANCELLED":  set(),
}
def transition(state, target, ctx):
    assert target in ALLOWED[state], f"illegal {state}->{target}"
    ctx.append_event(state, target)   # write the event first
    ctx.persist_state(target)         # keep state consistent with the event log

The core principle here: write the event first, then change the state. On crash recovery, treat the event log as the source of truth so state and facts never diverge.

4.4 Observability and result acceptance: no acceptance means no delivery

There is a wide gap between agent output that “looks right” and output that “is right.” Engineering needs three layers of acceptance:

• Format acceptance: does it match the agreed structure, are the required fields present — decidable directly by a program.
• Behavioral acceptance: does it satisfy executable assertions, such as tests passing, compilation succeeding, checks all green.
• Semantic acceptance: performed by a stronger model or by human spot-checks, used to assess quality and risk, usually sampled rather than exhaustive.

You must also record a replayable trace of every step: the context summary the model saw, tool call inputs and returns, latency, and token consumption. Without traces, production problems can only be guessed at.

5. Managed Runtime vs. Self-Built Scaffolding: Where the Trade-Off Lies

When teams keep hitting the infrastructure wall, one natural option is to stop building it themselves and adopt a managed agent runtime from a platform provider, with the execution environment, file system, skill extensions, and memory capabilities all supplied from the cloud.

This isn’t merely about “saving effort” — it’s a redrawing of responsibility across layers.

When a managed runtime fits

• The team is small and has no dedicated platform engineering capability.
• Task shapes are generic (retrieval, writing, code modification, process handling) and align well with the platform’s assumptions.
• You need to validate business value quickly and don’t require deep control over the underlying stack.
• Compliance permits the data to enter the chosen runtime environment.

When you still need to build it yourself, or go hybrid

• You have strong determinism requirements and need custom scheduling, prioritization, and quota policies.
• Data cannot leave your own network boundary, requiring private execution environments and egress control.
• You need deeply customized execution semantics, such as a dedicated compile toolchain or a special permission model.
• You already have a mature observability stack and need agent events wired into your own logging and alerting pipelines.

The pragmatic path is usually hybrid: keep core inference and scheduling in-house, host generic execution and memory capabilities, and connect to your own logging, monitoring, secret management, and permission systems through standard interfaces. That way you avoid re-treading the same infrastructure pitfalls while retaining control over the critical path.

6. An Active Trend: Harness and Model Are Becoming Co-Designed

A few months ago the mainstream thinking was: build one “universal harness,” then hot-swap models at will and use whichever is strongest. That made sense when model capabilities were fairly close and interface paradigms were converging.

The situation has changed. Different labs have taken more divergent technical paths in training objectives, tool-calling paradigms, long-context utilization, and reasoning budget allocation. As a result:

• Tool granularity carefully tuned for model A may over-constrain the planning space on model B.
• Context organization that depends on a particular model’s implicit reasoning habits can lose its benefit — or even turn negative — after a model swap.
• The same termination conditions and loop limits behave very differently depending on how “overconfident” each model is.

So engineering practice is shifting from “universal harness + swappable model” toward co-designing the harness and the model: changes in model capability drive harness adjustments, and the capability boundaries the harness exposes feed back into optimization priorities on the model side.

This creates a direct engineering requirement: manage the model version as part of the harness. Concretely:

1. Package model identifier, version, sampling parameters, and harness configuration into a single immutable “run configuration,” recorded with every task.
2. Maintain a separate evaluation baseline per run configuration, so every switch triggers a regression.
3. Make the evaluation set cover capability dimensions rather than end-to-end success rate alone — for example tool selection accuracy, context utilization, steps to loop convergence, and cost per task.
4. Preserve canary rollout and fast rollback so that “swapping the model” never becomes an incident.

7. Multi-Agent Orchestration: A High-Frequency Test Bed for Harness Techniques

As orchestration became a standard platform feature, “collaborating with multiple agents” quickly turned into the default way to test harness ideas. Three shapes are relatively mature today.

1. Separating executor and advisor

The executor agent holds tool permissions and changes the environment directly; the advisor agent has read-only access and proposes improvements. They interact through structured messages, and the executor may accept or reject a proposal, but must record its reasoning.

The value lies in isolating the risk of “proposing” from the risk of “acting”: a bad suggestion never becomes a destructive operation directly, while exploratory space is preserved.

{"role": "advisor", "proposal": "switch to binary search to localize the failing case", "confidence": 0.62, "risk": "low"}
{"role": "executor", "decision": "accept", "reason": "upper bound can be reduced at acceptable cost", "action": "run_tests --filter"}

2. The adversarial generator-critic structure

One agent produces content; another audits specifically to “find problems,” and the auditor is explicitly incentivized to discover defects rather than approve the work. This pattern pays off for writing, design review, and code review, because the audit goal is narrow and the criteria are clear.

Watch the cost and control implications: adversarial structures significantly increase token consumption and latency, and an auditor can become overly picky and trigger endless rework. Engineering must set a round limit, define an explicit “acceptable threshold,” and allow early convergence once that threshold is met.

3. Swarm-style parallel exploration

For open-ended search tasks like “where might the problem be,” you can launch several lightweight agents in parallel, each covering a different region or hypothesis, sharing a central list of findings that is deduplicated and then aggregated.

This works when the search space is partitionable, individual probes are cheap, and results can be judged objectively (for example, whether a failure reproduces). If the criteria are subjective or probes are expensive, parallelism amplifies noise and cost instead.

New infrastructure requirements introduced by multi-agent setups should be planned for at the same time: message channels and backpressure between agents, concurrent-write conflicts on shared state, allocation and circuit-breaking of the overall budget, and end-to-end attribution of “which agent did what.”

8. An Actionable Engineering Checklist

Condensing the analysis above into something you can check against directly:

Execution environment

• Session working directories are persistent; snapshot and restore available after critical steps
• Each execution has explicit resource and duration limits
• Environments can be rebuilt, and the rebuild path has been rehearsed

State and memory

• Task state exists independently of the process; runs can resume after restart
• Intermediate artifacts go to object storage; context keeps only references and summaries
• Short-term and long-term memory are separated, and long-term memory supports relevance-based retrieval

Async and scheduling

• The external interface is a three-part “submit + query + callback” contract
• State machine transitions are constrained and illegal transitions are rejected
• The submit API is idempotent, so retries produce no duplicate side effects

Observability and acceptance

• Every tool call has a structured trace and is replayable
• Format and behavioral acceptance are automated; semantic acceptance is sampled
• Cost, latency, and success rate are monitored and alerted on per task and per model version

Security and permissions

• Tool permissions are minimal, and destructive operations require explicit confirmation
• Keys and credentials never enter the context; they go through separate managed storage
• Network egress is controlled, and external content is treated as untrusted input

Harness and model co-design

• Model version and harness configuration are bound into an immutable run configuration
• Every run configuration has its own evaluation baseline, and switching triggers a regression
• Canary release and fast rollback are available

9. The Picture a Year Out: Simplification Is an Outcome, Not a Starting Point

Asked what things will look like in a year, a reasonable expectation is this: as managed runtimes mature, the scaffolding teams have to write themselves will shrink noticeably, a lot of boilerplate will be absorbed into platform capabilities, and painstaking tuning of tool descriptions and prompt wording will give way to more structured task definitions and acceptance criteria.

But the causal direction needs emphasis: simplification is a result of mature infrastructure, not a goal you can pursue directly. If you skip infrastructure today and chase a “minimal harness,” you usually just move complexity out of the code and into incident triage.

A more robust evolution path has three steps:

1. Raise the floor first: complete the execution environment, state persistence, async jobs, and observability plus acceptance, so the agent doesn’t “fail silently” when unattended.
2. Then push the ceiling: iterate on the harness on top of a stable load-bearing layer, at which point prompt and tool-contract changes have a trustworthy measurement baseline.
3. Finally subtract: once generic capabilities are sufficiently absorbed by the platform, delete self-built scaffolding and concentrate the team’s energy on business semantics, acceptance criteria, and domain knowledge.

10. Closing

The value of “the hardest part isn’t the harness, it’s the infrastructure” is not that it dismisses the importance of harness engineering — it corrects the order of investment. The harness determines how high an agent can go; infrastructure determines whether it can keep standing on the ground. The rapid spread of multi-agent orchestration, generator-critic adversarial setups, and swarm search is essentially about giving harness techniques denser test scenarios; whether those shapes run stably still depends on the maturity of the load-bearing layer.

For engineering teams, the single most practical recommendation is this: on day one of designing an agent, write “what do we do when the task fails at minute 47?” into the plan.