AI Agent Deployment: The Real Bottleneck Is the Infrastructure Wall

1. A Common Misreading of the Hardest Problem

After talking with many teams deploying AI agents, a recurring misconception appears: most people believe harness engineering is the hardest part of building an agent—how to design tools, write prompts, split context, and orchestrate steps—while model capability development is seen as the second hardest problem.

That is often true during the prototype stage. But when a system moves from “can demo” to “in production and at scale,” most teams eventually hit not the harness layer, but the infrastructure wall.

The boundary between these two concepts is worth clarifying:

• Harness: The scheduling and constraint layer around the model. It includes tool definitions and calls, context and memory trimming, output-format constraints, retry and correction policies, and multi-step task decomposition. It determines how the model is used.
• Infrastructure: The runtime foundation beneath the harness. It includes isolated execution environments (sandboxes), file systems and state storage, task queues and asynchronous scheduling, identity and permissions, secret management, logging and monitoring, quotas, and cost attribution. It determines whether these components can run reliably, safely, and continuously over the long term.

The harness determines the upper bound of what an agent can do. Infrastructure determines the lower bound of whether it is usable. In production, failures almost always occur at the lower bound.

2. From Prototype to Production: Six Frequent Failure Points

1. Sandbox disconnections and session interruptions

Agents execute code, operate browsers, and read or write files, usually inside an isolated runtime environment. In the prototype stage, that environment may be a developer’s laptop or a long-lived virtual machine. If it disconnects, it can simply be restarted.

In production, the questions change: What happens to running tasks when an instance is recycled? How do you determine whether a long unresponsive period means the task is stuck rather than still reasoning? Once network policies tighten, which outbound requests will be blocked? How should isolation level be balanced against startup latency, given the trade-offs among containers, micro virtual machines, and function compute?

Without clear lifecycle management and health checks, users see tasks fail mysteriously or spin forever.

2. Loss of state and memory

An agent’s “memory” is actually spread across several layers: conversation context, cross-session user preferences, project facts and conclusions, intermediate artifacts, and task progress. Prototypes often keep these in process memory or local disk.

Once deployment becomes stateless—multiple replicas, rolling updates, failover—all in-memory state is lost. Teams then have to add:

• Checkpoints: Persist task state after each critical step and support resuming from any node.
• Externalized memory: Store factual memory externally, isolate it by session and tenant, and apply expiration and deduplication policies.
• Context reconstruction: On resume, first rebuild summaries and key facts instead of stuffing all history back into the context window.

An illustrative structure:

# Persistence contract for task state (illustrative, not a concrete interface)
state = {
    "task_id": "t-8f21",
    "tenant_id": "acme",
    "phase": "executing",          # planning / executing / verifying / done
    "step_index": 7,
    "plan": [...],
    "artifacts": ["oss://bucket/acme/t-8f21/report.md"],  # Artifacts stored externally
    "checkpoints": [
        {"step": 6, "tool": "run_tests", "result_digest": "sha256:...", "cost_tokens": 18320}
    ],
    "resume_policy": {"max_retries": 3, "backoff": "exponential"},
}

3. Asynchronous execution and long-running tasks

A single agent task may last from several minutes to tens of minutes. Synchronous HTTP requests cannot handle this: gateways time out, load balancers drop connections, and client retries cause duplicate execution.

Three things are required:

• Taskification: Wrap an agent run as an asynchronous job, return a task ID immediately, and deliver results through polling, callbacks, or event streams.
• Idempotency: Generate idempotency keys for each tool call and side-effect operation so retries do not produce duplicate writes.
• Progress observability: Stream intermediate events—such as reading a file or running tests—instead of leaving users waiting on a blank screen.

4. Concurrency, resources, and quotas

When multiple users and multiple tasks run concurrently, resource contention appears in strange ways: multiple tasks write to the same branch, compete for the same test environment, or overwhelm rate limits on downstream APIs. This requires concurrency gates, per-tenant quotas, tool-level rate limiting, and serialized queues for dangerous operations.

5. Permissions and data boundaries

Agents hold credentials and access internal systems, making them naturally high-value attack surfaces. The minimum requirements include least-privilege short-lived credentials, tenant-isolated data boundaries, human confirmation for high-risk operations, and a complete audit trail: who did what, with which identity, and at what time.

Prompt injection is the most realistic risk: any external content entering the context—web pages, tickets, logs, dependency files—should be treated as untrusted input. Permission decisions cannot rely on the model’s self-restraint; they must be enforced by an external policy engine.

6. Observability and cost attribution

Traditional APM focuses on request latency and error rates. Agent observability is different: it must track every tool call, every context expansion, and the attribution of every token. Without trace-level monitoring and cost attribution, teams cannot answer why a task failed, which tenant is consuming budget, or how much more expensive a system became after switching models.

The common approach is to connect log services, metrics monitoring, and alerts, distributed tracing, and aggregate by task ID. On the cost side, measurements should be recorded along three dimensions: tenant, task, and tool.

3. Managed Runtimes: Turning Learned Pain Points into Capabilities

Most of the work in the list above is unrelated to business logic; it is the common foundation needed to keep agents running. Writing it once per team produces diminishing returns and increasing risk. The industry trend is therefore to move it upward into a platform capability: a managed agent runtime. Typical capability boundaries include:

Capability area	Common implementation
Isolated execution	Container platforms, function compute, and micro virtual machines, created and recycled per session
Files and artifacts	Object storage plus session-level temporary volumes, with lifecycle policies
Memory and retrieval	Structured storage plus vector retrieval, with tenant isolation and TTL cleanup
Task orchestration	Message queues plus workflow engines, supporting delays, retries, and dead-letter handling
Tool access	A unified tool gateway for centralized authentication, rate limiting, and auditing
Observability	Log services, monitoring and alerting, and distributed tracing
Secrets and policies	Centralized secret management plus an external policy engine

When implementing this kind of system on Baidu AI Cloud, these capabilities usually correspond to general cloud product categories such as container and function compute, object storage, message queues, and logging and monitoring/alerting. The real engineering difficulty is not product selection; it lies in three areas: session-level isolation granularity, state consistency semantics, and the completeness of failure recovery.

4. An Underrated Trend: Harnesses and Models Are Becoming Highly Coupled

A few months ago, the mainstream approach was to build a general-purpose harness, hot-swap models at any time, and decide which model to use through A/B testing. That assumption is being broken.

The reason is straightforward: different labs optimize models in different directions—tool-call formats, structured-output stability, effective context length, reasoning budget allocation, and whether the model natively supports code execution and parallel tool calls. These differences are not merely different levels of the same capability; they are differences in shape. A prompt and tool description tuned for Model A often does not merely degrade on Model B; it may stop working entirely.

A new engineering paradigm is therefore emerging: co-designing the harness with the model. A practical structure is “thin abstraction plus policy registration”:

# Capability abstraction layer: unify semantics, not implementations (illustrative)
class ModelProfile:
    name: str
    tool_call_style: str        # json_schema / native / prompt_parsed
    max_context_tokens: int
    effective_context_ratio: float
    supports_parallel_tools: bool
    supports_streaming_reasoning: bool
    prompt_template_id: str
    verify_retry_policy: dict   # Retry and fallback policy after output validation failures

The key point is that the abstraction layer only unifies interface semantics—calling tools, producing structured results, and reporting progress—while each model’s specific harness strategy (prompt template, tool description style, context trimming algorithm, validation retry count) evolves independently as pluggable configuration. Evaluation must also change accordingly: it becomes a matrix of model × harness strategy rather than single-model benchmarking.

The cost is that the number of harness variants grows, along with maintenance overhead. The mitigation is to push common parts down into the runtime and leave only strategy differences at the upper layer.

5. Multi-Agent Orchestration: Three Proven Patterns

When a single agent’s context and responsibilities become overloaded, splitting becomes necessary. Three orchestration patterns are relatively mature in current engineering practice:

1. Executor / advisor separation

One agent performs execution while another provides analysis and suggestions without producing side effects directly. This decouples “dare to act” from “should act,” and is suitable for high-risk write scenarios. The cost is increased round trips and higher context synchronization overhead.

1. Generator / reviewer adversarial loop

One agent produces output while another finds errors, looping until the output meets standards or hits an upper limit. This is suitable for content generation, code review, and solution design. The key is that review criteria must be decidable—rules, tests, checklists—otherwise the process degrades into two models complimenting each other.

1. Swarm-style parallel scanning

The search space is split into many independent pieces—files, modules, test cases—then dispatched in parallel and aggregated with deduplication. This is suitable for code defect investigation, large-scale log analysis, and cross-repository research. Bottlenecks usually appear in deduplication and ranking quality during aggregation, as well as in cost control.

Common engineering constraints for orchestration:

• Each sub-agent must have an independent sandbox and independent permissions; high-privilege credentials must not be shared.
• Sub-task results should be returned only as structured artifacts, not as entire raw contexts.
• Hard budgets—steps, tokens, time—must be set; when exceeded, execution must terminate and degrade.
• The main agent should only plan, dispatch, and aggregate; it should not perform specific execution.

6. Outcome Acceptance: From “Looks Correct” to “Verifiably Complete”

The easiest place for a harness to fool itself is this: process logs look good, but results are unusable. Mature systems make acceptance a first-class concern:

• Verifiable contracts: Each task declares success criteria—tests pass, type checks pass, schema validation passes, artifacts exist and are non-empty, key metrics remain within thresholds.
• Deterministic checks first: If something can be determined by scripts, do not ask the model to self-evaluate.
• Sampled human review: For parts that cannot be automatically judged, sample them into a human queue according to policy, and turn feedback into evaluation data.
• Regression evaluation sets: Convert production failures into fixed cases and rerun them whenever the harness or model changes.
• Visible failures: When acceptance fails, output structured reasons instead of a generic “task failed.”

outcome_contract:
  task_type: code_change
  criteria:
    - type: test_suite
      command: "make test"
      required: pass
    - type: static_check
      command: "make lint"
      required: pass
    - type: artifact_exists
      path_pattern: "dist/*.whl"
  on_failure:
    - action: retry_with_feedback
      max_attempts: 2
    - action: escalate_to_human
  budget:
    max_steps: 40
    max_wall_clock_seconds: 1800

7. One Year Later: Harnesses Will Slim Down, and Engineering Focus Will Move Up

A reasonable prediction is that as runtime capabilities become standardized, much of the scaffolding code in the harness will be absorbed by the platform. Tool integration, session recovery, context trimming, and retry/validation loops—things that must be written manually today—will gradually become configuration.

But “simplification” does not mean “disappearance”; it means a shift in focus. At that point, team investment will concentrate on:

1. Environment provisioning: High-quality, reproducible execution environments with real dependencies. This is a prerequisite for agents to do real work and is often the most underestimated part of the engineering effort.
2. Permissions and compliance: Fine-grained authorization, auditing, and data boundaries.
3. Evaluation systems: Task-level, regression-capable evaluation sets that reflect real business value.
4. Reliability and cost engineering: Budget control, rate limiting, degradation, and failure recovery.
5. Business semantic modeling: Tools and data structures should fit the domain, not merely fit the model.

In other words, the share of prompt engineering and scaffolding code will decline, while the share of infrastructure and evaluation engineering will rise. Whoever builds the wall first will more easily convert model capability into product capability.

8. Staged Recommendations for Deployment Teams

• Stage 1 (Validation): Use a single agent, read-only tools, and human confirmation for all side effects. The goal is to make the pipeline work, not to maximize automation.
• Stage 2 (Usable): Introduce asynchronous taskification, checkpoints and recovery, structured logging, cost tracking, and externalized memory. The goal is “recoverable failures and diagnosable problems.”
• Stage 3 (Scale): Add multi-tenant isolation, quotas and rate limiting, permission consolidation, acceptance contracts, and regression evaluation sets. The goal is “deliverable and auditable.”
• Stage 4 (Optimization): Pair harness strategies with models, add multi-agent orchestration, and choose orchestration patterns by scenario. The goal is “improve task success rate per unit cost under budget constraints.”

Order matters. Skipping Stage 2 and jumping directly to Stage 4 is the most common failure path for these projects: the more complex the orchestration, the more infrastructure cracks are amplified.

Conclusion: The harness determines what an agent can do; infrastructure determines whether it can be delivered. The former can be assembled quickly with engineering techniques; the latter can only be filled in piece by piece through systematic platform capabilities—and that is the wall most teams will truly hit.