1. Why LLM APIs Are Getting “Thicker”

Early LLM APIs were essentially a “text completion” function: you send in a prompt, you get back text, and developers build everything else themselves around it. Once applications shifted from “generating content” to “completing tasks,” the limits of this single-point interface became obvious quickly. One tool call is tolerable; ten tool calls plus subtask dispatch, failure retries, session state, and context compaction pile up a large amount of business-irrelevant infrastructure code on the application side.

Two evolution paths emerged, sharing the same goal but using different approaches:

• Additive evolution: Keep the stable message interface as the core primitive and layer runtime capabilities on top of it (tool execution, session state, memory, hosted agents). Developers choose how deeply to integrate based on the control granularity they need.
• Replacement evolution: Introduce a new, higher-level interface that pulls the message primitive partly or entirely into the platform, using “submit a task / run an agent” as the new calling semantics.

The former leaves the choice to developers; the latter leaves the complexity to the platform. Understanding the layered structure of these two paths is the prerequisite for making sound LLM application choices.

2. Layer One: The Stable Message Primitive

The message interface is built on a minimalist contract: a request carries a set of role-tagged messages (system instructions, user input, assistant history, tool results), and the response returns a structured list of content blocks rather than a block of raw text. Common block types include text blocks, tool-call blocks, reasoning blocks, and multimodal blocks such as images.

This design looks ordinary, but it delivers three key benefits:

1. Composability: Separating content from structure means the client does not need to parse model output with regexes or delimiters, and can reliably distinguish “what the model says” from “which tool the model wants to call.”
2. Extensibility: Adding a new block type introduces a new capability (such as explicit reasoning) without changing the overall protocol.
3. Auditability: Each interaction has a clear structure, which makes logging, replay, and troubleshooting easier.

On top of this contract, the message interface itself can already carry fairly complex agent workflows.

2.1 Tool Calling: An Explicit Multi-Turn Loop

The model does not execute tools directly. Instead, it outputs a tool-call intent block (tool name + structured arguments); the client executes it in a controlled environment and returns the result as a tool-result block; the model then decides the next step based on that result—continue calling, correct arguments, or give a final answer. Execution authority always stays on the developer’s side, which is both a security boundary and a source of flexibility.

def agent_loop(messages, tools, max_steps=10):
    for _ in range(max_steps):
        resp = client.messages.create(model=MODEL, messages=messages, tools=tools)
        messages.append({"role": "assistant", "content": resp.content})
        calls = [b for b in resp.content if b.type == "tool_use"]
        if not calls:                      # No tool calls -> task converged
            return extract_text(resp.content)
        results = []
        for c in calls:
            output = dispatch(c.name, c.input)   # Local controlled execution + timeout/permission checks
            results.append({"type": "tool_result",
                            "tool_use_id": c.id, "content": output})
        messages.append({"role": "user", "content": results})
    raise RuntimeError("Maximum steps reached, manual intervention required")

This loop is the minimal definition of an “agent.” The engineering problems it must solve on its own include: loop termination conditions, per-step timeouts, how to feed tool errors back to the model, how to truncate overly long results, and how to schedule concurrent calls.

2.2 Explicit Reasoning Blocks

Allowing the model to return its reasoning process as a separate content block, distinct from the final answer, has three benefits: the reasoning can be passed across turns to keep multi-step tasks coherent; the answer region stays cleaner for structured output and front-end rendering; and reasoning content can be retained or discarded selectively by policy, balancing explainability against context cost.

2.3 Context Caching Capabilities

A harsh reality of agent scenarios is this: nearly every request carries almost the same prefix (system prompt, tool definitions, historical messages), and recomputing that part every time is both slow and expensive. Prefix-caching mechanisms let the server reuse the already-computed prefix state, substantially reducing the marginal cost and latency of long conversations and high-frequency tool calls. In practice, you must keep the prefix stable: move volatile content (timestamps, random IDs, user-specific fragments) out of the prompt head and place it at the tail of the message, otherwise cache hit rates will be undermined.

2.4 Structured Output and Constrained Decoding

Mechanisms such as JSON Schema constrain the output structure, turning “try to make the model answer in a given format” into “the model can only answer in that format.” This is the foundation for agent results being directly consumable by downstream code, and is especially useful for tool-argument generation, data extraction, and state-machine decisions.

2.5 Server-Side Hosted Tools

General-purpose capabilities such as search, web scraping, and code execution can be hosted and executed by the platform, removing the need to build your own retrieval system and sandbox. The trade-off: it saves effort, but you do not have full control over execution location and data flow, so compliance-sensitive scenarios require evaluation.

2.6 Advantages and Costs

The advantage of the message primitive is the finest control granularity: the complete agent loop, context management, and tool-execution logic are all in your hands, so you can precisely decide when to compact, when to parallelize, and how failures degrade. The cost is equally direct: multi-step tool orchestration, subtask dispatch, session-state persistence, context compaction, and observability instrumentation must all be implemented by you—and these are exactly the parts that are unrelated to business value yet most prone to problems.

3. Layer Two: Library-Based Agent Runtime

Once a capability is repeatedly validated in leading products, it gets abstracted into a library. Exposing the same runtime that drives mature coding agents to developers for embedding in their own products is the typical second step of additive evolution. Its value lies not in “one more SDK,” but in making the implicit best practices of the agent loop explicit: how to design the toolset, how to validate permissions, when to compact context, and how to recover from failure.

Two typical integration forms:

Integration form	State characteristics	Applicable scenarios
Single-task entry (stateless)	Completes one task per call, no session retained	Scripted tasks, batch processing, generating changes in CI, prototype validation
Session client (stateful)	Multi-turn conversation, session persistence, supports hooks and permission interception	Interactive products, long-cycle collaboration, workflows requiring manual approval

Built-in tools typically cover file reading, writing, and editing, command execution, directory traversal, pattern matching, and web retrieval and scraping, plus reusable “skill” wrappers. Around these, the runtime should provide four kinds of control points:

• Permissions and approvals: Policy checks and manual confirmation before tool execution, deciding which operations can be auto-approved.
• Hooks: Inject custom logic before a request, before a tool call, or after a response, used for auditing, rewriting prompts, or blocking dangerous operations.
• Subtask dispatch: The main loop spawns independent contexts to handle local problems, preventing the main context from being flooded with intermediate artifacts.
• Context compaction: When nearing the window limit, summarize and trim history, trading controlled information loss for sustained operation.

The selection boundary is clear: if you need to directly touch the local file system, code repositories, or terminal environments, or want to embed the agent into an existing developer toolchain, the runtime library is the most suitable choice. When you need multi-tenancy, cross-device sessions, and unified governance, its capabilities fall short.

4. Layer Three: Managed Agent Platform

The third layer raises the semantics one more notch: developers no longer “call a model” but “submit a task and run an agent.” The capabilities the platform takes on typically form this puzzle:

• Session and state management: Session persistence, resume-from-checkpoint, cross-device recovery.
• Memory tiering: Short-term context, session summaries, and long-term memory (user preferences, project conventions), plus write and recall strategies for memory.
• Tool registration and gateway: Tools are registered declaratively, while the platform handles authentication, rate limiting, auditing, and execution-environment isolation.
• Sandboxed execution: Code and commands run in isolated environments, restricting file-system and network access.
• Task scheduling: Long-running tasks, scheduled tasks, and event-triggered tasks, along with retry and degradation strategies.
• Versioning and evaluation: Canary releases and regression evaluation for prompts, toolsets, and model versions.
• Observability: Trace tracking, per-step latency and token consumption, tool success rates, and cost attribution.
• Safety guardrails: Input/output filtering, sensitive-data masking, and blocking of unauthorized operations.

What changes along with this is the cost structure: the billing dimension may expand from pure token usage to task runtime, concurrent sessions, or seats. When planning budgets, you cannot simply extrapolate linearly from token unit price; you must also fold in retry rates, context compaction rates, and the number of tool round-trips.

5. Comparing the Three Paths

Dimension	Message Primitive	Agent Runtime Library	Managed Agent Platform
Control granularity	Finest, controllable per turn	Fairly fine, built-in but pluggable loop	Coarse, task-based delivery
Onboarding cost	High	Low to medium	Lowest
State management	Self-implemented	Provided by the library	Managed by the platform
Tool execution location	Fully self-controlled	Mainly local/self-hosted	Platform sandbox or hosted service
Portability	Highest (stable protocol)	Medium (coupled to the runtime)	Lower
Typical scenarios	Highly customized workflows, compliance-sensitive	Embedding in own products, local-environment operations	Quick launch, multi-tenant SaaS, long-running tasks

The three paths are not mutually exclusive substitutes; they are more like three abstraction layers of the same architecture. Mature systems are often like this: business orchestration runs on a managed platform, critical tools are precisely controlled through the message primitive, and local development-assistance scenarios are handled by an embedded runtime library.

6. Engineering Implementation Recommendations

Layer first, then choose a layer. Break the system into a model-access layer, an orchestration layer, a tool layer, a memory layer, and an observability layer. Wrap the model-access layer with an adapter that unifies request and response structures, so business code does not depend directly on the detailed shape of any one vendor’s interface. When you later switch models or upgrade from the primitive to a platform, changes stay confined to the adapter layer.

Treat context as a budget. Set window-occupancy thresholds (for example, trigger summarization at 70%, force compaction at 85%) and distinguish “must keep” (task goals, hard constraints, key conclusions) from “disposable” (raw intermediate tool output, lengthy retrieval fragments). Truncating or structuring tool return values before they enter the context is the most cost-effective way to reduce cost.

Design for failure, not for success. Tool calls should be idempotent and write operations should be rollback-able; set a timeout and maximum retries for each step; when the step limit is reached, output “completed parts + a to-do list” rather than spinning idly. Place manual approval points before irreversible operations.

Front-load observability. At minimum, record each step’s prompt version, tool name and latency, input/output summaries, token and cache-hit status, and termination reason. Problems in agent-type systems can almost never be located from a single log line; you must link the entire chain by session.

Evaluate by task success rate, not by impression. Build a fixed evaluation set covering typical and edge-case tasks, use offline regression to test prompt and model changes, and use online canary comparison across four metrics: task completion rate, average step count, manual-intervention rate, and cost per task.

Design security and compliance in parallel. Apply least privilege to tools, isolate execution environments, keep sensitive data within bounds or mask it, and retain full audit trails. The stronger the managed platform’s capabilities, the more clearly you must define data flow and the boundary of responsibility.

7. Common Misconceptions

• Premature platformization: Handing orchestration logic to a managed platform before the business is validated, which shrinks tuning space and raises migration costs.
• Treating an agent as a workflow, or a workflow as an agent: Use workflow orchestration for deterministic, fixed-step processes—it is more reliable and cheaper; reserve agents for open-ended goals where the path is uncertain.
• Ignoring cache-friendly prompt design: Mixing dynamic content into the prompt head keeps cache-type capabilities permanently ineffective, losing on both cost and latency.
• Optimizing only the model, not the context: The root cause of most effectiveness and cost problems lies in context quality, not in which model you switched to.

8. Conclusion

The main thread of LLM API evolution is gradually pulling uncertainty out of developers’ hands and into the platform, while keeping a stable protocol foundation available for deep customization. The message primitive guarantees you can always drop down to the underlying layer; the runtime library saves you from writing thousands of lines of duplicate code; the managed platform lets you give your attention back to the business. The practical requirement for developers is: know clearly which layer you are working in, and why you are in that layer. When building on platforms such as Baidu AI Cloud, it is also advisable to assemble the layered structure above using general-purpose capabilities—model serving, container platforms, message queues, logging, and monitoring and alerting—so that the application has the elasticity to evolve across models and across layers.