Decoding Hosted Agents: Cloud-Native Agent Architecture and Implementation Practices

1. Background: From Model-as-a-Service to Agent-as-a-Service

Over the past year, the capability boundaries of large language models (LLMs) have been continuously pushed to new limits. Based on public industry information, the latest generation of flagship models generally exhibits three key characteristics. First, parameter scales are moving toward the trillion-level, significantly enhancing complex reasoning and task planning capabilities. Second, context windows have expanded to the million-token magnitude, allowing them to accommodate large codebases, lengthy research reports, or long-cycle project materials in a single pass. Third, long-horizon Agent capabilities have become a primary optimization direction, with notable performance improvements in scenarios requiring multi-step execution such as code generation, 3D creation, and financial research. Concurrently, some providers offer “high-speed” inference services, boosting output speeds by approximately 2 to 3 times under the same level of intelligence to cover latency-sensitive online interactive scenarios.

This leap in model capabilities is transforming application paradigms. The focus of AI applications is shifting from “single-turn Q&A” to “goal-oriented multi-step execution.” Supporting such applications requires more than just a powerful model; it necessitates a comprehensive engineering infrastructure, including task orchestration, tool execution, state management, and security isolation. Against this backdrop, “Hosted Agents” have emerged as a new service paradigm for developers.

2. What is a Hosted Agent?

A Hosted Agent can be defined as an agent service where the cloud platform uniformly manages model inference, task orchestration, tool execution, and state management. Developers submit a business objective via an API, and the agent autonomously completes the “plan—execute—observe—revise” loop in the cloud, ultimately delivering the result.

This fundamentally differs from typical self-built Agent solutions:

• Self-built solutions: Developers must deploy or call model APIs themselves, implementing the planning loop, tool registration, session storage, failure retries, concurrent scheduling, and monitoring/alerting. This results in high engineering complexity, requiring the team to possess both algorithmic and platform engineering capabilities.
• Hosted solutions: The platform shoulders the responsibility for the runtime environment, sandbox isolation, state persistence, and elastic scaling. Developers focus on three things: defining objectives, declaring tools, and consuming results. This mirrors the evolution logic of cloud computing from “self-built data centers -> cloud virtual machines -> Serverless,” sinking the execution infrastructure to make the application layer lighter.

3. Core Architecture Layer Breakdown

A mature hosted agent service can typically be decomposed into five layers:

1. Model and Inference Layer: Responsible for providing foundational intelligence. Key metrics include inference quality, context length, output speed, and cost. Long-context capability determines how much “working memory” the agent can hold, while high-speed inference versions dictate the upper limit of the interactive experience.
2. Orchestration and Planning Layer: Responsible for task decomposition and the execution loop. The industry widely adopts paradigms like ReAct (alternating reasoning and action) and Plan-and-Execute. Coupled with reflection and retry mechanisms, these ensure the agent maintains goal alignment without drift and remains recoverable upon errors during multi-step tasks.
3. Tool and Execution Layer: The agent uses tools via structured function calling protocols. Typical tools include code interpreters, file I/O, retrieval, and browser operations. Execution usually occurs in sandboxes formed by containers or micro-VMs to achieve least-privilege permissions and fault isolation.
4. State and Memory Layer: This includes session persistence (retaining context across requests), short-term memory (trajectories within the current task window), and long-term memory (consolidating historical knowledge via external storage and vector retrieval). Context compression and summarization techniques are used to retain the most critical information within a limited window.
5. Access and Governance Layer: Exposes three access methods—synchronous APIs, asynchronous tasks, and callback events. It is complemented by logging, distributed tracing, evaluation baselines, and content safety guardrails to ensure the service is observable, auditable, and rollback-ready.

4. Technical Implications of Three Key Capabilities

1. Long Context and Long-Horizon Tasks: Million-token windows make it possible to “hand over an entire codebase or a full set of research reports to the model.” Long-horizon Agent capability emphasizes maintaining goal consistency across tasks spanning dozens of steps and several hours, relying heavily on checkpointing, breakpoint resumption, and trajectory management mechanisms.
2. Tool Calling Ecosystem: Tools are declared with structured descriptions (e.g., JSON Schema) for parameters and functionality, allowing the model to decide “when to call and what parameters to pass.” The richer the tools and the more standardized the protocols, the larger the actionable task space for the agent.
3. High-Speed Inference: Boosting speed several times under the same intelligence level typically stems from general optimizations at the inference framework level, such as speculative decoding, KV cache management, quantization, and batch scheduling. For interactive scenarios, speed directly dictates product viability.

5. Typical Application Scenarios

• Software Development: Cross-repository requirement understanding, generating-running-debugging closed-loop execution, automatically fixing defects, and submitting change suggestions.
• Data Analysis and Financial Research: Parsing lengthy research reports and financial statements, executing calculations and chart generation, and outputting structured conclusions.
• 3D and Creative Production: Multi-round iterative generation and modification, combined with code interpreters to verify rendering results.
• IT Operations Automation: Determining root causes based on alert contexts, generating remediation scripts, and verifying them in sandboxes.

6. API Call Example (Pseudo-code)

# Submit a long-horizon task to a hosted agent asynchronously 
# (endpoints and fields depend on actual service documentation)
import requests
task = requests.post(
    "https://{agent-endpoint}/v1/agent/runs",
    headers={"Authorization": "Bearer <API_KEY>"},
    json={
        "agent": "research-analyst",              # Configured agent
        "input": "Read the three uploaded industry research reports and output a structured comparative summary",
        "tools": ["code_interpreter", "file_reader"],
        "session_id": "sess-20250101-001",        # Reuse session to retain context
        "mode": "async",                          # Async execution recommended for long tasks
        "callback": "https://my-app.example.com/hooks/agent"
    },
).json()
# Retrieve results via callback or polling: final answer + full execution trajectory

The internal execution loop can be abstracted as follows:

while not finished:
    plan        = llm(goal, history, tool_specs)      # Plan the next step
    observation = sandbox_exec(plan.tool, plan.args)  # Execute in the sandbox
    history.append(plan, observation)                 # Record trajectory
    finished    = judge_done(plan, observation)       # Determine if the goal is achieved

7. Engineering Implementation Recommendations

1. Evaluation Dimensions: Focus on model inference quality, context length, tool richness, sandbox isolation strength, observability, billing transparency, and compliance capabilities.
2. Cost Control: Leverage context caching and batch processing. Simple sub-tasks can be routed to smaller, faster models to form a tiered calling strategy.
3. Evaluation First: Build a benchmark task set aligned with your specific business needs. Continuously conduct regression tests using four metrics: pass rate, step count, time consumption, and cost.
4. Security and Compliance: Minimize tool permissions, filter sensitive information, and retain complete execution trajectories for auditing purposes.

8. Conclusion

Hosted agents represent the “Serverless” trend in LLM application engineering: sinking complex execution infrastructure to the cloud, allowing developers to utilize agents just like cloud functions. In the Chinese market, cloud service providers such as Baidu AI Cloud have also built general platform capabilities around LLM services and agent construction. Developers should comprehensively evaluate these platforms based on their own tech stacks, compliance requirements, and cost budgets. As model capabilities and engineering support continue to evolve, hosted agents are poised to become one of the standard delivery formats for AI applications.