Scaling LLM Agents in Production: Engineering Practices and Governance

1. The Competition Has Shifted from Model Scale to Engineering Delivery

The development of foundation models over the past few years can be broadly divided into four recognizable stages:

1. The first stage: “Can it converse?”
   The focus was on general-purpose language understanding and generation.

2. The second stage: “Can it handle long context?”
   The core work involved expanding context windows and improving understanding of long documents and codebases.

3. The third stage: “Can it take action?”
   This stage introduced tool calling, structured output, multimodal perception, and multi-step reasoning.

4. The fourth stage: “Can enterprises trust it?”
   The key questions become reliability, permission isolation, cost control, auditability, and compliance boundaries.

Public information has also included claims such as “an internally deployed interactive agent at a leading foundation-model research organization has accumulated working hours several times those of human employees.” The real value of this kind of statement is not a direct comparison between humans and machines. It shows an important engineering reality: agents are moving from “demo scripts” to “production components that run continuously, 24 hours a day, 7 days a week.”

Once an agent becomes a production component, the evaluation criteria change. Instead of asking whether the answer sounds human-like, teams begin measuring a set of engineering indicators: task completion rate, first-pass rate, human intervention rate, cost per task, and P95 latency.

Public reports have also mentioned that, within one year, the share of junior technical positions at certain R&D organizations dropped from roughly one-quarter to less than one-tenth. This does not simply mean that “AI replaces engineers.” It reflects a change in the structure of work. Repetitive tasks that can be formalized—such as routine coding, test-case generation, and document organization—are increasingly automated. At the same time, the importance of requirement clarification, architecture decisions, result acceptance, and risk ownership increases significantly.

As a result, engineering teams should focus less on “writing more code” and more on “designing better constraints and validation mechanisms.”

2. A Five-Layer Reference Architecture for Enterprise Agents

A maintainable enterprise-grade agent system can usually be divided into five layers. The purpose of layering is not to impose a rigid framework, but to ensure that each layer has clear replacement points, observability points, and failure boundaries.

1. Model Serving Layer

This layer is responsible for model access and scheduling. In practice, it should support:

• Model routing, such as selecting different model sizes based on task complexity;
• Failure retries and fallback mechanisms;
• Redundancy across multiple providers or deployment modes to avoid business disruption when a single model becomes unavailable.

Platforms such as Baidu AI Cloud, where available, may provide managed model services, private deployment options, and general-purpose infrastructure such as object storage, message queues, container platforms, logging services, monitoring, and alerting. These capabilities can serve as the underlying infrastructure for this layer.

2. Orchestration Layer

This layer is responsible for task planning, state progression, and multi-agent collaboration. Its core principle is to constrain “free-form generation” into a “managed state machine.”

3. Tool and Integration Layer

This layer covers function calling, retrieval, code execution, and external system APIs. All write operations must pass permission checks and include idempotency design.

4. Data and Knowledge Layer

This layer handles document parsing, chunking, vectorization, and incremental updates. One of the most important requirements is that retrieval results must be filtered by permissions before they are inserted into the prompt context.

5. Observability and Governance Layer

This layer includes end-to-end logging, evaluation-set regression testing, cost accounting, content-compliance checks, and audit trails.

The orchestration layer can be illustrated with the following simplified code. The key elements are “checkpoints, bounded loops, and explicit termination conditions.”

# Agent orchestration skeleton (illustrative; not tied to any specific product interface)
from dataclasses import dataclass, field
@dataclass
class AgentState:
    goal: str
    plan: list = field(default_factory=list)
    observations: list = field(default_factory=list)
    step: int = 0
class BoundedAgent:
    def __init__(self, planner, executor, checker, max_steps=12):
        self.planner, self.executor, self.checker = planner, executor, checker
        self.max_steps = max_steps
    def run(self, state: AgentState, store=None):
        while state.step < self.max_steps:
            if store:                      # checkpoint/resume: persist state at each step
                store.save(state)
            if not state.plan:
                state.plan = self.planner(state.goal, state.observations)
            action = state.plan.pop(0)
            observation = self.executor(action)      # tool call, including authorization checks
            state.observations.append(observation)
            verdict = self.checker(state)            # done / needs correction / failed
            if verdict.done:
                return state
            state.step += 1
        raise RuntimeError("exceeded max_steps: human intervention or replanning required")

A hard limit such as max_steps may look simple, but it is one of the most effective safeguards in production. It turns the risk of “an agent getting stuck in a loop and causing uncontrolled costs” from an unknown into something measurable.

3. Four Engineering Problems for Long-Running Agents

Context and Memory Layering

A single very long context is both expensive and prone to diluting important information. In engineering practice, memory is usually managed in layers:

• Working memory: intermediate results of the current step;
• Episodic memory: summaries of historical trajectories;
• Semantic memory: knowledge chunks retrieved through vector search;
• Procedural memory: validated operation templates that have proven effective.

Compression strategies must preserve traceability. For example, summaries should retain citation IDs for the original chunks so that auditing and traceback remain possible.

Idempotency and Recovery

Agents interact with real systems. Repeated execution may lead to duplicate orders, duplicate charges, or other side effects. Every side-effect operation should carry an idempotency key and check whether it has already been completed before execution:

def charge(order_id: str, amount: int):
    key = f"charge:{order_id}"
    if store.exists(key):          # if already executed, return the historical result
        return store.get(key)
    result = payment_api.charge(order_id, amount)
    store.put(key, result, ttl=7 * 24 * 3600)
    return result

Typical Failure Modes and Countermeasures

• Goal drift: the agent gradually moves away from the original objective. Countermeasure: periodic goal review.
• Tool misuse: parameter type or semantic mismatch. Countermeasure: strict input validation for tools and error feedback.
• Hallucination propagation: an error from the previous step is treated as fact in the next step. Countermeasure: external fact verification and confidence annotation.
• Permission overreach: the agent performs actions beyond its allowed scope. Countermeasure: use the minimum necessary tool set and require explicit confirmation for write operations.

Cost and Latency Governance

Common methods include:

• Semantic caching to reuse results for similar requests;
• Batching and asynchronous execution;
• Using a small model for initial filtering and a larger model as a fallback;
• Setting latency targets based on business value.

Cost should be treated as a first-class metric and connected to monitoring and alerting systems, rather than being reviewed only through monthly bills.

4. Compliance Red Lines for Generated Content: The Case of “Prohibiting AI Cloning of Existing Works”

Public information has mentioned that a category of AI coding tools prohibits users from directly cloning existing games with AI. The essence of this issue is copyright risk at both ends: training corpus and generated output. This risk can be broken down into implementable technical controls rather than being handled only through a terms-of-use statement.

Input Side: Intent Recognition and Interception

Requests such as the following should be classified and intercepted:

• “Clone an existing game.”
• “Extract all level data from a certain game.”
• “Generate art assets highly similar to a specific IP.”

Interception records should be retained.

Retrieval Side: License Allowlists

Every chunk in the retrieval library should include source and license metadata, such as:

• Commercial use allowed;
• Internal use only;
• Redistribution prohibited.

During retrieval, results should be filtered according to the intended use.

Output Side: Similarity Detection and Source Tracking

Code and asset outputs should be checked for duplication and source attribution before being returned to the user.

def guard_output(text: str, license_mode: str = "internal") -> dict:
    hits = similarity_search(text, top_k=5, threshold=0.86)   # near-duplicate detection
    blocked = [h for h in hits if not h.license_allows(license_mode)]
    if blocked:
        return {"allow": False, "reason": "near_duplicate",
                "sources": [h.source_id for h in blocked]}
    return {"allow": True, "citations": [h.source_id for h in hits]}

Beyond technical controls, organizations should also complete governance-side actions, including:

• Defining an acceptable use policy;
• Conducting red-team testing before launch, especially with prompt sets designed to induce cloning behavior;
• Establishing incident response procedures;
• Defining audit-log retention periods.

Technical controls reduce the probability of violations. Governance and auditing help determine responsibility and scope when an incident occurs.

5. A Three-Phase Rollout Roadmap and Acceptance Metrics

Phase 1: Validate a Single Scenario (Several Weeks)

Choose scenarios with clear boundaries and automatically verifiable results, such as:

• Log root-cause analysis;
• Test-case generation;
• Internal knowledge question answering.

Build an offline evaluation set of 100 to 300 cases. Make the agent measurable before making it production-ready.

Phase 2: Platformization and Scale (Several Months)

Develop reusable components and controls, including:

• A unified orchestration framework;
• A tool registry;
• A permission model;
• An observability dashboard.

Connect the evaluation set to CI. Model changes or prompt changes must pass regression testing. Establish cost accounting and rate-limiting strategies.

Phase 3: Cross-Domain Autonomy and Governance Loop

Allow agents to collaborate across multiple systems while making routine processes part of normal operations, including:

• Human review sampling;
• Compliance checks;
• Incident retrospectives.

Core acceptance metrics should include:

• Task completion rate;
• First-pass rate;
• Human intervention rate;
• Average number of steps;
• Cost per task;
• P95 latency;
• Unauthorized-action interception rate;
• Compliance-detection recall.

Among these, human intervention rate and first-pass rate are especially important. They provide the strongest signals about whether the agent system is truly usable and whether its engineering maturity is sufficient.

6. Conclusion

The value of large models ultimately depends on engineering constraint design, not only on parameter scale. To move from demonstration to scale, agents should be built as production systems that require rate limiting, idempotency, auditing, and evaluation. At the same time, copyright, privacy, and security should be treated as first-class architectural concerns from the beginning, because they determine whether the system can run sustainably over the long term.

For developers, the most worthwhile next investment is not simply waiting for stronger models. It is building more reliable guardrails, clearer observability, and more rigorous acceptance systems.