LLM API Evolution: From Message Primitives to Managed Agent Platforms

Over the past two years, the design focus of major LLM APIs has shifted from “making models talk” to “making models do work.” The evolution paths of mainstream platforms can be summarized into three categories:

• Directly replacing the core interface: introduce a new conversation API that integrates conversation, tool calling, reasoning, and state orchestration into a new primitive;
• Keeping a stable core and adding runtime layers: the message API remains unchanged for a long time, while agent development kits, context compression, memory, managed agents, and other capabilities are added layer by layer;
• Building agent capabilities into the platform: delegate multi-step task orchestration, execution, and governance to the platform.

The goals of the three paths are not fundamentally different: to let developers build reliable agents with less code. However, their migration costs, control granularity, and coupling models differ sharply. For developers preparing to build agent applications, the key is not to memorize a particular product, but to understand the layered model of “message primitives—runtime—managed platform.”

1. Two Evolution Paths: Core Replacement vs. Layered Runtime

Path 1: Replace the core interface

The new interface consolidates capabilities that were originally scattered in the client into the protocol layer. The model side is responsible for planning and advancing state, while the developer side only provides the environment and tools. The benefit is lower mental overhead, but upgrades require migration:

• Request and response structures, streaming protocols, error codes, and authentication methods must be adjusted accordingly;
• Surrounding toolchains, observability, and evaluation baselines need to be rebuilt;
• Maintaining multiple versions in parallel for a long time creates maintenance costs and can create gaps in documentation and community knowledge.

Path 2: Keep the stable primitive and add runtime layers on top

Take the message API as an unshakable foundation, and make all new capabilities optional layers. Migration is often incremental: developers first use the message API to get the chain working, then gradually introduce runtime and managed capabilities when the complexity of tool calling, conversation length, or operational pressure reaches a bottleneck. The cost of this approach is deciding “at which layer to integrate” during technology selection; the benefit is that each layer can be evaluated and replaced independently.

What is more noteworthy is the commonality between the second and third paths: they move the hardest parts of agent engineering from the “protocol” to the “runtime.” Loop control for multi-step tasks, allocation of context budgets, persistence of session state, permissions and security for tool execution, and observability of the entire chain are what determine whether an agent can go live. The message protocol itself is only the carrier.

2. Layer 1: Message Primitives, the Layer with the Strongest Control

Message APIs from mainstream LLM platforms share a basic paradigm: developers submit a list of messages composed of roles such as system, user, and assistant, and the model returns structured results composed of several content blocks. Content blocks may include text, tool-call requests, and, when reasoning capability is enabled, reasoning content blocks. The message list can therefore be viewed as a continuously growing structured conversation tree rather than a flat sequence of text.

Several common capabilities have formed around this primitive:

Capability	Mechanism	Main benefit	Usage notes
Tool calling	The model returns a tool-call block (tool name and arguments); after the developer executes it, a tool-result block is sent back	Allows the model to operate in a real environment and supports multi-step tasks	Clear tool descriptions and parameter schemas are required
Reasoning content	The model returns the reasoning process in a separate content block, which can be sent back in requests per turn	More stable complex reasoning and long-chain tool selection	Some platforms apply signature validation to returned content; reasoning content is not the final answer
Prompt caching	Stable and unchanged prefixes in the prompt, such as system prompts, tool definitions, and knowledge fragments, are marked as cacheable	Reduces token cost and first-token latency for repeated prefixes	The prefix must remain stable, and cache breakpoints should be organized according to the platform’s minimum-length threshold; pricing for cache hits should follow the platform documentation
Structured output	Mechanisms such as JSON Schema constrain the model’s output format	Results can be consumed directly by programs, reducing parsing failures and retries	Complex nesting should be accompanied by examples; business semantics still need validation
Server-side tools	Platform-hosted execution of web search, web fetching, code execution, etc., with results returned directly to the conversation	No need to build your own crawler, search, or sandbox	Result reliability requires secondary validation, and additional charges may apply

A minimal agent loop

MAX_STEPS = 20
def run_agent(messages, tools, exec_tool, client, model):
    for _ in range(MAX_STEPS):                      # Step limit: fallback termination condition
        resp = client.messages.create(
            model=model, system=SYSTEM_PROMPT, tools=tools,
            messages=messages, max_tokens=4096,
        )
        messages.append({"role": "assistant", "content": resp.content})
        if resp.stop_reason != "tool_use":          # No tool request: task complete
            return next(b.text for b in resp.content if b.type == "text")
        for block in resp.content:                  # Execute each tool requested by the model
            if block.type == "tool_use":
                result = exec_tool(block.name, block.input)
                messages.append({"role": "user", "content": [{
                    "type": "tool_result", "tool_use_id": block.id,
                    "content": result, "is_error": not result.ok,
                }]})
    raise RuntimeError("Maximum step count exceeded; task not completed")

This code is less than twenty lines and can already complete the “think—call—observe” loop. However, it is only a prototype:

• There is no failure retry or timeout;
• There is no context-length budget, so long tasks will inevitably hit the window limit;
• Session state exists only in memory and is lost when the process restarts;
• Tool execution has no permission validation, so risks such as parameter injection and path traversal must all be handled by the business logic.

These gaps correspond exactly to the capabilities that the next runtime layer needs to fill.

3. The Missing Middle: Capabilities That Must Be Built Between Primitives and Production

1. Loop control and termination: maximum steps, per-task token and time budgets, duplicate tool-call detection, and circuit breaking when the same error occurs repeatedly. Without these constraints, an agent can easily fall into an infinite loop of repeatedly attempting the same operation.
2. Context engineering: estimate token usage in real time, compress earlier turns that exceed the budget into structured summaries, truncate oversized tool results, and preserve task goals, constraints, and unfinished items during compression. This mechanism is usually called compaction, and it directly determines whether long tasks can complete.
3. State persistence: write sessions and intermediate artifacts to storage, support checkpoint recovery, human intervention, replay auditing, and handle idempotency during retries.
4. Permissions and sandboxing: authorize tools with least privilege, put command execution and code running in isolated environments, and constrain file-write scope, network egress, and key access.
5. Observability and evaluation: record the full trace of every model call and tool call, measure tokens, latency, cost, and tool error rates, and use a fixed task set for regression testing to prevent prompt or model-version updates from causing performance regressions.

A common mistake is underestimating items 2 and 4: the former determines “whether the task can be completed,” while the latter determines “whether it is safe enough to deploy.”

4. Layer 2: Agent Development Kits, Turning Mature Runtime Into Libraries

After the runtime behind products such as intelligent coding tools and automated office assistants has been repeatedly refined, the industry has naturally productized it: expose the same runtime that powers these products as a development kit for developers to embed in their own applications. This packages most of the five gaps above into reusable infrastructure.

Two common integration methods

Method	Form	Suitable scenarios
One-shot task invocation	Stateless; one call completes one task and returns the final result	Batch processing, code checks in CI, offline data processing
Persistent session client	Stateful; maintains multi-turn context and supports hooks and permission control	Interactive assistants, human-computer collaborative editing, long-running tasks

Built-in tools and lifecycle hooks

Such kits usually come with a set of default tools for code and documentation scenarios, such as reading and writing files, editing specific locations, locating files by glob pattern, searching by content, executing shell commands, and web search and page fetching. The benefit for developers is that they do not need to implement file system access, command execution, and search logic one by one to obtain an agent capable of working in real engineering projects.

More importantly, there are lifecycle hooks. Before a tool call, you can validate parameters, rewrite the request, or reject it directly; after a tool call, you can desensitize or truncate the output; there are also interception points when a sub-agent starts, before context compression, and when a session ends. Their value lies in “changing behavior without rewriting the loop”:

• Auditing: write every tool call and result to a logging service to meet compliance traceability requirements;
• Interception: require human confirmation before executing commands, or directly reject high-risk parameters;
• Modification: inject key task points before compression to avoid losing goals in long tasks.

from some_agent_sdk import HookMatcher, tool
async def guard_bash(input_data, tool_use_id, context):
    cmd = input_data.get("command", "")
    if any(k in cmd for k in ("rm -rf", "git push --force")):
        # Returning a decision blocks the call and provides a rejection reason the model can understand
        return {"decision": "block", "reason": "High-risk commands require human confirmation"}
    return {"decision": "allow"}   # Allow without modification
opts = AgentOptions(
    allowed_tools=["Read", "Edit", "Grep", "Bash"],
    hooks={HookMatcher(matcher="Bash", hooks=[guard_bash])},
    permission_mode="acceptEdits",  # File edits pass automatically; command execution follows permission rules
)

The limitations are equally clear: such kits are usually deeply tied to a specific model family. Their default tools, prompt strategies, and context-compression logic are tuned for that model, so replacing the model may not be equivalent. For highly customized business loops, developers still need to fall back to message primitives and implement them themselves.

5. Layer 3: Managed Agent Platforms, Delegating Operations and Governance to the Platform

When agents need to serve multiple users, run for long periods, and call many external tools, the engineering burden shifts from “writing loops” to “running services.” Managed platforms target the latter:

• Session and state hosting: the server persists conversations and intermediate state, naturally supporting recovery, replay, and multi-user collaboration;
• Execution environment hosting: an isolated sandbox is allocated for each task, responsible for lifecycle, resource limits, and cleanup;
• Built-in tool hosting: platform-executed web search, page fetching, code execution, etc., with results returned;
• Governance and evaluation: unified trace tracking, datasets, and evaluation pipelines, with permission models covering tools and data resources;
• Capacity and quotas: elastic scaling, concurrency quotas, and cost allocation reduce capacity-planning work.

The cost is reduced control and observability granularity: underlying prompt and compression strategies are not fully transparent, customization points are limited by the platform’s openness, and deep binding increases migration costs. Therefore, managed platforms are better suited for scenarios with “clear business value, fast iteration, and teams that do not want to build their own infrastructure,” and are not ideal for algorithm teams that need fine-grained control over every model call.

Dimension	Message primitives	Agent development kits	Managed agent platforms
Control granularity	Finest; loop and context are fully autonomous	Relatively fine; can be customized through hooks	Constrained by platform-exposed capabilities
Initial development effort	High; requires building the runtime yourself	Low to medium	Lowest
Session persistence	Implement yourself	Provided by the kit	Hosted by the platform
Model replaceability	High	Medium; tied to a model family	Lower
Main risks	High engineering effort, slow launch	Vendor lock-in, customization boundaries	Low transparency, high migration cost
Suitable teams	Algorithm/platform teams	Application development teams	Business teams, rapid validation

6. Selection and Architecture Recommendations

Adopt layered integration based on control granularity. During the validation phase, use managed capabilities directly to prove the value loop. Once in production, abstract the model-call layer, tool-execution layer, and state-storage layer into your own internal interfaces, with managed platforms and self-built runtime as replaceable implementations. This way, you can enjoy the benefits of hosting while retaining an exit path.

Hybrid modes are often optimal. Use development kits to handle common capabilities such as code operations, search, and command execution; use message primitives for core business reasoning and customized loops; for critical paths with extremely high stability requirements, retain human confirmation and fallback switches.

Put cost governance first. Enabling caching for stable prefixes, routing models of different specifications by task complexity, and setting length limits on tool results are usually the three optimization points with the highest return on investment.

Tighten security defaults. Default to deny rather than allow; require explicit authorization for write operations and operations with external side effects; treat external content read by the model (web pages, issue descriptions, file contents) as untrusted input to prevent prompt injection from inducing the model to perform unauthorized operations.

When deploying on Baidu AI Cloud, you can reuse existing cloud capabilities to support the self-built portion:

• Model services: text, multimodal, and vectorization capabilities provided by the LLM platform are integrated through a unified adapter layer for easier switching as needed;
• Execution and runtime: container platforms host sandboxes and agent services, scaling elastically based on actual tasks;
• Async and storage: message queues handle asynchronous decoupling and peak shaving for long tasks, while object storage stores session snapshots, intermediate artifacts, and evaluation datasets;
• Observability: logging services and monitoring alerts cover tool-call chains, and dashboards track tokens, latency, task success rates, and human-intervention rates;
• Permissions and security: combine the access control system to implement least privilege, and have credentials managed by a dedicated credential management mechanism to avoid writing them into code or prompts.

At the same time, it is recommended to abstract the model-call layer, tool layer, and session-storage layer into internal interfaces so that platform implementations are replaceable. This ensures that business code does not need to be rewritten when capabilities are upgraded or the solution is adjusted.

7. Conclusion

Message primitives, agent development kits, and managed platforms are not three generations of products that replace one another. They are three layers on the same capability stack: the former defines “how the model is called,” the middle solves “how tasks are executed reliably,” and the latter answers “how systems are operated and governed.”

Once this layering is understood, discussions about API selection become more concrete: Do you lack control over every model call, a context-management mechanism that can complete long tasks, or an operational system that can be delivered to the business? Different answers lead to different integration depths. There is only one approach to be truly wary of: directly exposing managed agents to end users without having governance capabilities in place.