Agent Engineering: Why Infrastructure, Not Prompts, Is the Real Moat

1. The Commonly Misjudged Difficulty

In discussions about agents, the most frequent claim is that “prompts and tool wrapping are the hardest parts.” Most attention goes to writing stable system prompts, structuring tool descriptions, and designing retry and fallback logic. These tasks do exist. In the industry, they are commonly called harness engineering: the full scaffolding layer outside the model, responsible for context assembly, tool exposure, execution loop control, error recovery, and result validation.

However, when a team moves an agent from a demo environment to production and tries to increase concurrency from single digits to hundreds or thousands, they hit a different wall. The wall is not that the prompt is not good enough. The wall is that the runtime environment lacks engineering support:

• Context is lost after a session is interrupted.
• Temporary files disappear when containers are destroyed.
• Long-running tasks cannot be awaited reliably.
• Multiple agents overwrite each other when accessing the same state concurrently.

In the prototype stage, these problems are hidden by single-machine, single-session, manually monitored runs. Scaling is the only scenario that exposes them all.

2. What the Infrastructure Wall Looks Like

2.1 Execution Environment Lifecycle Management

Agents need to execute real code, read and write files, and access the network. This requires an isolated sandbox instance. In production, the core question is not whether a sandbox can be started, but how its full lifecycle is managed:

• Is cold-start latency acceptable, and is a warm pool supported?
• After a session times out, should the sandbox be destroyed or suspended? If suspended, can it be restored to its previous state?
• How are resource quotas set for each sandbox to prevent one runaway loop from overwhelming the host?
• Should network egress be restricted through an allowlist to prevent unauthorized access?

2.2 File System and Artifact Persistence

Sandbox disks are usually temporary. If intermediate files, build artifacts, or reports generated by an agent are written only inside the container, they will disappear when the session ends.

A workable approach is to use layering:

• Keep a temporary workspace on local sandbox storage to preserve speed.
• Asynchronously sync long-lived artifacts to object storage at key task milestones.
• Archive artifacts using the session ID as a prefix for unified organization.

2.3 Tiered Design for Memory and State

“Memory” is often treated as a single feature, but in practice it should be divided into three layers, each with its own storage and expiration strategy.

Tier	Content	Lifecycle	Typical storage
Short-term context	Current session messages, tool call records	Single session	In-process storage + session store
Working memory	Task plan, completed steps, pending items	Single task	Structured state store
Long-term memory	User preferences, project conventions, historical conclusions	Cross-session	Retrieval store + periodic distillation

Mixing these layers together is one of the biggest design mistakes. Putting long-term memory directly into prompts can quickly blow up the context window. Writing short-term state into a retrieval store can fill retrieval results with noise.

2.4 Long Tasks and Asynchronous Execution

A code-level task may run for minutes or even longer. A synchronous HTTP request will time out, and the frontend has no reasonable way to wait.

The correct execution model should be split into three stages: submit, poll or receive push updates, and fetch results. Task state should be maintained by an independent storage layer, and after a process restart, execution should be able to resume from a checkpoint.

# Pseudo-code: resumable execution loop with checkpoints
def run_task(task_id, steps, state_store, sandbox):
    for index, step in enumerate(steps):
        # Save a checkpoint before each step so the process can resume from this point after a crash
        state_store.save_checkpoint(task_id, {"step": index, "step_state": step.state})
        result = sandbox.execute(step.command, timeout=step.timeout)
        if result.exit_code != 0 and step.retryable:
            result = retry_with_backoff(step.command, max_attempts=3)
        if result.exit_code != 0:
            state_store.mark_failed(task_id, reason=result.stderr)
            return  # Failure details are fully persisted for humans or upstream agents
        step.state = result
        state_store.append_artifact(task_id, result.artifacts)
    state_store.mark_succeeded(task_id)

The key points are:

• State writes and command execution should be separated.
• Checkpoints should be explicit steps, not implicit side effects.

2.5 Identity, Permissions, and Tool Authorization

When an agent acts on behalf of a user or a team, it needs an auditable identity model. At a minimum, this should include:

• The agent’s own service identity.
• The delegated user identity it acts for.
• Authorization scoped at the tool level.
• Audit logs for every sensitive operation, such as writing files, making external requests, or modifying configuration.

Injecting user credentials directly into a sandbox is one of the most common and dangerous approaches. Code running inside a sandbox is not fully trusted. Once credentials leak, the impact extends beyond a single session.

2.6 Observability and Cost Control

The call chain of a multi-step agent is usually an order of magnitude longer than that of a normal service. Without structured tracing, it is impossible to answer basic operational questions such as:

• Which step did the failure occur in?
• How many tokens were consumed?
• Which tool was slow or stuck?

The minimum useful observability set includes:

• Distributed tracing organized by session and step number.
• Input and output summaries for each step.
• Tool latency distributions.
• Cost aggregation at the task level.

3. Harness Engineering and Models Are Becoming Tightly Coupled

A once-popular assumption was that models could be replaced while the harness remained stable. As long as the interface did not change, switching models was just changing a parameter.

This assumption is weakening for three reasons.

1. Capability differences require different scaffolding.
   Different models perform differently in long-context utilization, tool-call format adherence, and multi-turn self-correction stability. A context compression strategy tuned for one model may actually reduce performance after switching to another.

2. Vendors are absorbing capabilities into the model layer.
   Capabilities that were previously implemented through prompts, such as structured output, tool selection, and multi-turn planning, are increasingly being internalized through training and inference mechanisms. If the harness layer continues to pile on prompts, it is doing redundant work against the model’s own capabilities.

3. Evaluation is converging toward end-to-end outcomes.
   The focus is shifting from “how well the prompt is written” to task completion rate, rework rate, and cost per task. All three metrics are affected by both the model and the harness, making them difficult to attribute independently.

The engineering conclusion is that harness engineering should be modular. Interfaces should remain stable, implementations should be replaceable, and model-related components such as context budgets, tool description granularity, retry strategies, and compression thresholds should be externalized as configuration rather than hard-coded.

4. Multi-Agent Orchestration Is a Testbed for Harness Techniques

The value of multi-agent orchestration is not simply that it uses “more agents.” Its greater value is that it turns different harness patterns into comparable engineering choices.

Three commonly validated patterns are:

• Execution-suggestion separation.
  The main agent performs the work, while a side agent only reviews and gives suggestions without participating in write operations. The benefit is an independent review perspective. The cost is roughly double the latency. This pattern suits scenarios with high change risk.

• Generation-critique adversarial loop.
  One side generates content, while the other side critically looks for problems and repeats until a threshold is reached. This is suitable for copywriting, test cases, and design documents. Strict iteration limits are required to prevent meaningless loops.

• Parallel search-style collaboration.
  Multiple exploration paths are run concurrently on the same problem, each reporting independently, and an upper layer converges the results. This suits tasks where coverage matters more than depth, such as defect localization and large-scale code retrieval. However, it must solve result deduplication and conflict merging.

When choosing a pattern, ask three questions first:

1. Can the task be split into mutually independent subtasks?
2. Do the subtasks need to share intermediate state?
3. Is the cost of an error higher than the additional token and latency overhead?

If all three answers are unfavorable, a single agent with better tools is often better than a multi-agent system.

5. What a Managed Agent Platform Should Own

Providing sandboxes, file systems, skill loading, memory, and orchestration as managed capabilities is a reasonable direction. These components are highly homogeneous, and rebuilding them in every team brings little incremental value.

A reasonable boundary looks like this:

The platform should own:

• Execution environments.
• Storage.
• Identity and authorization.
• Orchestration primitives.
• Observability.
• Quotas and isolation.

These are “error-prevention” capabilities. It is difficult for users to evaluate how well they are implemented, and the cost of getting them wrong is high.

Developers should own:

• Task definitions.
• Concrete implementations of tools and skills.
• Domain knowledge and evaluation sets.
• Business-side success criteria.

These are “value-creation” capabilities and cannot be fully generalized.

One pitfall to avoid is treating a managed platform as a black box. If agent behavior cannot be traced, state cannot be exported, and the execution environment cannot be replaced, teams will be unable to locate problems or perform disaster recovery outside the platform.

6. A Practical Checklist for Self-Built Agent Infrastructure

The following items should be confirmed early in the project, rather than added later during scaling:

1. Sandbox policy
   Startup latency targets, concurrency limits, timeout and suspension semantics, and network egress constraints.

2. State and checkpoints
   Which steps are reentrant, how often checkpoints are written, and what idempotency guarantees exist after recovery.

3. Artifact archiving
   The boundary between temporary and persistent storage, archive trigger points, retention periods, and cleanup strategies.

4. Credential isolation
   Separate agent identity from user identity, enforce tool-level authorization, and avoid injecting credentials directly into the sandbox.

5. Evaluation loop
   Use a fixed task set and measure end-to-end pass rate, rework rate, and cost per task. Regress these metrics whenever the model or harness changes.

6. Degradation paths
   Define fallback behavior when the model is unavailable, the sandbox is unavailable, or a tool times out.

7. Reasonable Expectations One Year From Now

A likely evolution is:

• More boilerplate in the outer scaffolding layer will be absorbed by platforms, reducing the amount of code teams need to write.
• Manual prompt tuning will become less important, replaced by a regressable evaluation system.
• The coupling between models and harness engineering will become tighter, further weakening the assumption that models can be swapped freely.

However, one thing will not change: the success criteria for a task, domain knowledge, and the judgment of what counts as “done correctly” must still be defined by the people using the system. Infrastructure can reduce the cost of building agent systems, but it cannot replace understanding of the problem.

For current practice, the advice is direct: do not over-invest in perfecting every detail of prompts too early. First, make the execution environment, state management, and observability mature enough to support concurrency and failure recovery. These two factors determine whether a project can move beyond the prototype stage. Once the infrastructure is stable, the benefits of prompt tuning will become easier to realize naturally.