Hosted Agents Explained: Running LLM Agents in the Cloud

1. From Model APIs to Hosted Agents: The Evolution of Service Form

The service form of large models is undergoing a clear transition. In the early “model-as-a-service” model, developers used APIs to perform single-turn text generation, and the model was essentially a smarter autocomplete engine. This was followed by an “application orchestration” phase, in which developers built retrieval, tool invocation, and flow control around the model themselves to create RAG or workflow-style applications. The current mainstream direction is hosted agents: cloud platforms package model inference, tool execution environments, memory and state management, and secure sandboxes into an integrated service. Developers only need to describe the goal, and the agent autonomously completes the “perceive—plan—act” loop in the cloud.

The driving force behind this evolution is the change in task complexity. Long-horizon tasks such as cross-repository code refactoring, long research report writing, and multi-stage data processing inherently require multiple rounds of iteration and autonomous decision-making. If developers had to manually orchestrate every step, engineering complexity would grow rapidly with task length. Hosted agents push this complexity down to the platform layer, shifting developers from “writing flows” to “defining goals, giving constraints, and doing acceptance.”

Several recent frontier models released by the industry also confirm this trend: parameter scales moving toward the trillions, context windows expanding to millions of tokens, and dual “standard/fast” variants that balance intelligence and speed. All of these point to the same goal—making agents capable of carrying real long-horizon, complex production tasks.

2. The Three Technical Foundations of Hosted Agents

2.1 Long-Context Modeling: The Agent’s “Working Memory”

For agents, long context is not a nice-to-have but a hard requirement:

• Repository-level code understanding: A single refactor often requires grasping the structure and dependencies of dozens of files at once.
• Long-document reading: Scenarios such as financial research and legal review routinely involve hundreds of pages of material.
• Interaction trace retention: After executing dozens of steps, the agent still needs to remember the original goals and constraints.

The engineering challenges are equally clear: the attention computation of a standard Transformer grows quadratically with sequence length; KV cache memory usage grows linearly with length; and positional encoding risks extrapolation failure on extremely long sequences. Common industry approaches include sparse and block attention, hybrid structures combining sliding-window and global attention, KV cache quantization and prefix cache reuse, and rotary positional encoding with extrapolation strategies. Only by stacking these techniques do million-level contexts move from paper metrics to usable capability.

2.2 Long-Horizon Reasoning and Task Planning: From “Answering Correctly” to “Getting It Done”

Single-turn Q&A tests knowledge; long-horizon tasks test the ability to “get things done,” which typically includes task decomposition (breaking a vague goal into executable steps), tool invocation (choosing the right tool at the right time), self-reflection (correcting the route based on execution results), and error recovery (retrying or switching paths after failure). On the training side, the industry commonly introduces reinforcement learning with verifiable rewards in addition to instruction tuning, allowing models to learn more reliable reasoning chains in domains with automatic scoring such as code and math, and then generalizing this ability to open-ended tasks. To measure long-horizon capability, the focus is on task completion rate, step efficiency, and exception recovery performance rather than single-turn answer quality.

2.3 High-Speed Inference Variants: The Latency Requirement of Agent Scenarios

Agent execution is loop-based: a single task can involve dozens of model calls. If a single response takes ten seconds, the total time accumulates to an unacceptable level. Therefore, leading models commonly offer same-source high-speed variants that use general techniques such as quantization (INT8/FP8), speculative decoding, continuous batching, operator fusion, and KV cache optimization to achieve multiples of output speedup while keeping intelligence roughly aligned.

In practice, a smarter approach is tiered mixing:

def choose_model(step_type: str) -> str:
    # Use the high-capability tier for key decisions and the high-speed tier for mechanical subtasks
    if step_type in ("planning", "final_review"):
        return "llm-standard"
    return "llm-turbo"

That is, planning and final review go to the standard tier, while draft generation, batch subtasks, and formatting go to the high-speed tier, balancing quality and latency.

3. Runtime Architecture: What Happens Inside a Hosted Agent

A typical hosted agent platform can be divided into three planes:

• Control plane: session management, task orchestration, budget and policy control (e.g., maximum steps, token limits);
• Data plane: model inference services, tool gateways, isolated sandboxes, memory storage;
• Observability plane: full-link logs, trace tracking, metric monitoring, and alerting.

At its core is an agent main loop (illustrative):

state = init_state(task)
for step in range(max_steps):
    plan = llm.plan(state, tools=tool_schemas)   # Model decision
    action = parse_action(plan)                  # Parse into a structured call
    if action.name == "finish":
        break
    observation = sandbox.run(action)            # Execute inside the sandbox
    state.append(role="tool", content=observation)
return state.summary()

Tool calls are typically passed as structured JSON, for example letting the agent execute a piece of data analysis code in the sandbox:

{
  "name": "run_code",
  "arguments": {
    "language": "python",
    "code": "import pandas as pd\ndf = pd.read_csv('sales.csv')\nprint(df.groupby('region')['amount'].sum())"
  },
  "timeout_seconds": 30
}

Security mechanisms are a key selling point of the hosted model: file system isolation, network whitelists, CPU/memory/duration limits, least-privilege tool authorization, and complete audit logs together ensure that “autonomous execution” does not mean “uncontrolled execution.”

4. Typical Application Scenarios

1. Software engineering: repository-level understanding, defect localization, batch refactoring, and test generation. Long context lets the agent maintain global consistency across the “read code—modify code—run tests” loop, avoiding local changes that break the overall structure.
2. Data analysis and financial research: cross-document reading, multi-table join computation, and generation of research drafts with citation traceability. This scenario requires tight control of hallucinations, requiring conclusions to be traceable back to original sources.
3. 3D and multimedia creation: text-to-3D is essentially a multi-stage pipeline (geometry generation, texture mapping, rigging, render acceptance). The agent handles stage orchestration and iterative optimization based on render feedback, while humans only do final review.
4. Office and process automation: long-chain tasks such as cross-system information aggregation and periodic report generation. The hosted runtime can be driven stably with basic components such as scheduled scheduling and message queues.

5. Engineering Best Practices

• Context management: externalize state storage, compress history with hierarchical summaries, and inject retrieval augmentation on demand, avoiding “stuffing everything into the context”;
• Cost control: reuse system prompts via prefix caching, batch subtasks, and set token budgets and step circuit breakers;
• Evaluation and observability: build trace-level evaluation (step success rate, tool call accuracy) and regression test sets, and combine online full-link tracing to locate failing steps;
• Security and compliance: defend against prompt injection, minimize tool permissions, and define clear data boundaries and desensitization strategies.

6. Challenges and Outlook

Hosted agents still face challenges such as long-horizon reliability decaying with step count, insufficient evaluation benchmarks, and inconsistent multi-agent collaboration protocols. It is foreseeable that they will deeply integrate with cloud-native infrastructure—container platforms, object storage, message queues, and logging and monitoring services—and become the default runtime foundation for agent applications. Currently, cloud platforms such as Baidu AI Cloud commonly provide general capabilities including model services and agent development toolchains. Developers can build their own hosted agent applications quickly on top of these components and focus their energy on business value itself.