Scaling AI Agents in Production: Infrastructure Is the Real Bottleneck, Not Model Capability

1. The consensus is shifting: the hard part is runtime reliability, not model capability

In a recent public interview on building agent platforms, two platform leads offered a judgment worth returning to: most teams assume the hardest problem is extracting every last bit of model performance, but what actually stops them at the scaling stage is the infrastructure.

That is a counterintuitive conclusion, because for the past two years the industry’s attention has been captured by prompt techniques — how to get a model to call tools reliably, how to fit context into the window, how to make it err less. That work has real value, but it is a problem of the upper bound: it determines how smart an agent can get. Infrastructure determines the lower bound: whether the agent survives the tenth call under real traffic. A prototype runs ten times, so lower-bound problems stay hidden; production runs a hundred thousand times, and the lower bound becomes everything.

A typical pattern: the agent chains through multiple steps cleanly in a demo, then in production you see three failure classes — sandbox sessions dropping mid-task, critical context lost after several turns, and long-running tasks that no one closes out, leaving state dangling. None of these are model capability problems. They are engineering debt in the execution environment, state management, and task orchestration. They stay invisible in a feature demo and only surface together once concurrency and duration go up.

2. What harness engineering is actually responsible for

“Harness” — the carrying and constraining layer around the model — is often treated as a synonym for prompt engineering, but its real scope is much wider. It covers at least six things:

1. Tool semantics encapsulation: wrapping internal systems, APIs, and commands into capability units the model can understand, whose parameters can be validated, and whose side effects are controllable.
2. Context construction: what goes into the window, how to summarize and compress, and how to keep critical information from being pushed out during long tasks.
3. Control flow: when to continue, when to retry, when to stop and ask, when to declare failure.
4. Output structure and acceptance: structured parsing, schema validation, and who decides whether a result is correct.
5. Permissions and safety boundaries: what can be read, what can be written, whether dangerous operations require human confirmation.
6. Evaluation and replay: how failure samples become regression cases.

Here is a skeletal outline (pseudocode, expressing structure only, not any specific product interface):

def run_agent(task, model, ctx, tools):
    plan = model.plan(task, ctx.summary())           # planning
    for step in plan.steps:
        args = schema_guard(step.args, step.schema)  # validate parameters before the call
        result = tools.invoke(step.tool, args,       # controlled tool invocation
                              timeout=30, retries=2)
        ctx.append(step, result, checkpoint=True)    # checkpoint state to external storage
        verdict = checker.verify(step, result)       # verify the result
        if not verdict.ok:
            plan = model.replan(ctx.tail(), verdict.reason)
            break
    return ctx.finalize()

One point deserves attention: what really determines agent reliability is usually not the model calls inside this loop, but the facilities around it — sandboxes, checkpoints, queues, logs. That is why practitioners move the conversation from harness to infrastructure.

3. The wall after the prototype: five infrastructure problems

3.1 Execution environment and sandbox lifecycle

An agent that runs code or manipulates files needs an isolated, rebuildable, reclaimable environment. Typical problems at scale: cold-start latency inflating end-to-end response time; image and dependency bloat slowing pulls; resource quotas exhausted under high concurrency; sessions reclaimed after long idle periods, so tasks lose their working directory midway.

Three things need to be settled explicitly: the environment’s specification and warm-up strategy, its concurrency density and limits, and the recovery semantics after a disconnect — rerun from scratch, or resume from a checkpoint.

3.2 State and memory persistence

“A memory loss” looks to users like the agent suddenly forgetting what it did three minutes ago. The usual root cause is keeping session state in process memory. The workable fix is to externalize state: session history, intermediate artifacts, and pending task lists all go into persistent storage, and writes carry an idempotency key so retries cannot duplicate side effects.

Memory also needs layering: short-term working memory for the current task, task-level artifacts such as files, patches, and reports, and long-term memory across sessions for preferences and facts. The hardest part of long-term memory is not storage but recall strategy and invalidation governance — writing is easy, removing stale or wrong entries is not.

3.3 Long-running tasks and asynchronous orchestration

Real business tasks often run from minutes to hours, and may wait on external system callbacks or human approvals. These do not belong in a single synchronous request; they need event-driven or workflow orchestration, backed by timeouts, retries, compensation, and dead-letter queues.

What is most easily overlooked is the human-in-the-loop checkpoint: where a human must confirm, how state is frozen while waiting, and how to roll back after a rejection.

3.4 Observability and replay

Agent failures are often not errors but wrong results without errors. That requires tracing granular enough to cover every model call, every tool call, and token and cost attribution. The more important step is turning failure samples into a regression suite automatically: every production anomaly becomes a replayable case, and the evaluation set grows with runtime. Without this mechanism, teams tune by intuition and cannot measure whether anything improved.

3.5 Isolation, permissions, and compliance

The more an agent can do, the larger the risk surface. Least privilege, credential management instead of hardcoded secrets, secondary confirmation for dangerous operations, complete audit logs, and data staying within its boundary are hard gates before launch, not optimizations for later.

4. Why harnesses and models are becoming tightly paired

The same interview flagged another trend. A few months ago the fashionable approach was to build one generic harness and swap the underlying model at will. Today models differ noticeably in tool-calling protocols, adherence to structured output, long-context decay curves, handling of conflicting instruction hierarchies, and reasoning budgets (thinking modes). Engineering can only optimize against the specific model if it wants to extract the available capability.

That creates a real dilemma. A generic harness compromises with the weakest model and is usually forced to cap its own expressiveness; a heavily customized harness delivers better results but raises migration cost and tends to leak abstractions, with model-specific branches scattered through business code.

The pragmatic compromise is to converge the differences in one place rather than spread them out:

[Policy configuration layer]  <- model-specific parameters, prompt variants, tool description trimming
     |
[Adapter layer]               <- tool-calling protocol translation, output parsing, retry semantics
     |
[Business logic layer]        <- independent of any specific model
     |
[Evaluation and replay]       <- same tasks across models, driving policy selection

As long as the adapter layer stays thin and the evaluation set is broad enough, switching models becomes a configuration change plus a regression run, not a rewrite.

5. Multi-agent orchestration: a natural testbed for new harness designs

The value of multi-agent orchestration is not only that several agents work together; it also provides a low-cost testbed for new harness structures. Patterns already in practice include:

• Executor/advisor separation: one agent acts while another is read-only, handling review and risk warnings to reduce mis-operation.
• Generation–review adversarial pairing: content generation and content review are held by independent roles, using independent judgment to suppress self-confirmation bias.
• Swarm-style parallel search: cast a wide net across a large codebase or a large dataset, each agent returning candidates, then a merge layer deduplicates, votes, and verifies.
• Routing and aggregation: simple tasks take a lightweight path, complex tasks escalate to a heavier flow, controlling total cost.

These patterns amplify three classes of engineering problems: cost (call volume and duplicated context rise sharply), context contamination (erroneous conclusions from multiple agents reinforce each other), and termination conditions (how parallel tasks decide they have found what they need and should stop). This again confirms the earlier point: pattern design is upper-layer innovation, while whether it runs stably depends on the lower-layer facilities.

6. Build or use managed: a clear decision line

If infrastructure is the main cost center, then “what should we build ourselves” deserves a deliberate boundary.

Lean toward building: strict compliance or data-boundary requirements; execution environments that need deep customization (a dedicated runtime, specialized hardware); core flows that are themselves the competitive advantage; and a team able to sustain investment in sandboxes, orchestration, and observability.

Lean toward managed: the stage between validation and early scale; task shapes close to mainstream scenarios; a wish to concentrate engineering effort on task specifications, acceptance criteria, and evaluation. Managed does not mean abandoning the harness — it means converging harness work from “maintaining scaffolding” to “designing strategy and acceptance.”

For technology selection, it is better not to fixate on proprietary terminology and instead check whether the common capability categories are covered: container platforms or function compute for isolated execution environments; object storage and file services for task artifacts; message queues and workflow orchestration for asynchronous execution and retries; log services plus monitoring and alerting for observability; and key management plus access management for the permission boundary. When building this kind of system on a platform such as Baidu AI Cloud, the same approach applies: go through the capability checklist category by category, then decide what to build and what to reuse directly.

7. What the shape of this will look like a year from now

Based on the judgment in that discussion, a few directional expectations:

• Much less scaffolding code: filesystem, skill loading, memory, and similar common capabilities sink into platform defaults. Teams delete templated glue code, not business logic.
• Prompt engineering shifts to specification engineering: from stacking technique-laden sentences to writing clear task specifications, acceptance criteria, and failure judgments.
• Engineer roles migrate: from writing loops and retry logic to designing evaluation systems, doing failure analysis, and governing cost.
• The capability ceiling is set by evaluation: tool count is no longer a source of differentiation; whoever has a more complete regression set and finer tracing iterates faster.

One thing to stay clear-eyed about: the infrastructure barrier will not disappear, it will only move upward. Today it is sandboxes and state management; tomorrow it may be governance of cross-session memory, cost constraints in multi-agent systems, and the design of human-in-the-loop approval chains.

8. Production readiness checklist

1. Is all state externalized, and can work resume from a checkpoint after a process restart?
2. Are write operations idempotent, and can retries produce duplicate side effects?
3. Have sandbox startup latency, concurrency density, and resource limits been load-tested?
4. Do tasks longer than a few minutes use asynchronous orchestration, with timeout and dead-letter handling?
5. Is tracing granular down to a single tool call, with token and cost attribution?
6. Do failure samples enter the regression evaluation set automatically?
7. Are model differences converged into a single adapter layer instead of scattered across business code?
8. Do dangerous operations have least-privilege enforcement and human confirmation points?
9. Is memory layered with invalidation governance, rather than append-only?
10. Do multi-agent patterns define explicit termination and deduplication rules?

The real value of this discussion is that it pulls attention back from “how do we make the model smarter” to “how do we make the system reliable.” Model capability keeps improving and will absorb part of the upper-layer techniques for us, but execution environment, state, orchestration, observability, and permissions are hard engineering that has to be filled in incrementally as business scale grows. Make the lower bound solid first, then talk about pushing the upper bound — that is the path from demo to production for agents.