Agent Output Design: From Linear Text to Interactive UI

1. The starting point: a text stream is a poor fit for expressing “process”

For the past two years, LLM applications have defaulted to essentially one output shape: linearly arranged rich text, with code blocks and lists as the only embellishment. When the model’s role is “responder,” this shape is efficient enough — the reader consumes information in reading order, the model releases it in generation order, and the two align naturally.

The moment the assistant’s role becomes “executor,” however, three weaknesses of linear text surface immediately:

• No state. Tables, progress, branch decisions, and items awaiting confirmation are structured objects. Once flattened into paragraphs, a program cannot parse them reliably and is left depending on regular expressions or fragile prompt conventions.
• No interaction. A user who wants to tweak a parameter, approve a high-risk action, or compare two candidate approaches has no choice but to type out a new description — turning what should be a single click into a full conversational round trip.
• Unverifiable intermediate states. Long-running tasks often take minutes to hours. When the intermediate results, evidence citations, and failure reasons exist only in the chat transcript, they are hard to audit and impossible to resume from after a disconnection.

More fundamentally, once humans join the execution loop (human-in-the-loop), the role of output changes: it is no longer merely a carrier for the answer, but a control plane — carrying state, carrying authorization, carrying human decisions. This is also the root reason more and more products emphasize rich media and interactive output (HTML rendering included): the chat interface is being demoted from endpoint to entry point, while the real work happens inside the rendered interface.

2. A layered design for rich interactive output: structure first, rendering as the fallback

Do not treat “emit HTML” as the goal in itself. The sound approach is to split output into three layers with clear responsibilities that can evolve independently:

Layer	Responsibility	Typical form
Semantic	Express intent and data	Structured JSON (intent, parameters, state)
Rendering	Turn data into a visible interface	Component tree, declarative UI description, HTML fragment
Degradation	Guarantee readability on any client	Markdown text summary

An illustrative output envelope might look like this:

{
  "type": "interactive_card",
  "version": 1,
  "intent": "confirm_destructive_action",
  "summary": "About to delete 3 production data tables, 12.4 GB in total",
  "data": {
    "tables": ["orders_2024", "orders_2025", "user_profile"],
    "size_gb": 12.4,
    "reversible": false
  },
  "actions": [
    { "id": "confirm", "label": "Confirm deletion", "style": "danger", "requires_typed_input": true },
    { "id": "cancel",  "label": "Cancel", "style": "default" }
  ],
  "fallback_text": "About to delete 3 production data tables (12.4 GB total). This is irreversible. Reply \"confirm deletion\" or \"cancel\"."
}

Three practical constraints are worth writing into team conventions:

1. fallback_text must exist and be self-consistent. Client capabilities vary widely; terminals, email, and push notifications may only be able to display plain text. Omitting the degradation layer is equivalent to staking availability on a frontend implementation.
2. Actions must be idempotent and replayable. Repeated clicks, network retries, and page refreshes all cause duplicate submissions. Bind action IDs to task IDs, and have the server return the same result for an already-completed action rather than executing it again.
3. Rendering contracts must be versioned. A version field plus an explicit ignore-unknown-fields policy lets older clients degrade instead of crashing.

3. Security is the first constraint of rich interactive output, not an afterthought

Any HTML generated by a model and ultimately executed in the user’s browser must be treated as untrusted content. This is not conservatism but an objective fact about the attack surface: the model reads web pages, documents, issues, and email bodies, and prompt injection in any of them can readily induce the model to produce a fragment carrying scripts or references to external resources.

Protections that have to land in the implementation:

• Iframe sandbox isolation. Host generated content in <iframe sandbox> + srcdoc; do not grant allow-same-origin, allow-top-navigation, or allow-forms. Script capability is off by default.
• Content Security Policy. Declare default-src 'none' explicitly, and allow by exception only style-src 'unsafe-inline' and img-src data:; forbid external scripts, fonts, and form submission.
• Resource allowlisting. Route all images, charts, and external links through a server-side proxy or data URIs, so that rendering itself cannot leak the visitor’s identity or trigger outbound requests.
• No access to credentials. Inject no cookies or LocalStorage tokens into the sandbox; any data requiring authentication is fetched by a backend endpoint and passed in as read-only data.
• Size and complexity ceilings. Cap fragment size, DOM node count, and inline style count so rendering cost cannot be weaponized as a denial-of-service vector.

<iframe
  sandbox="allow-scripts"
  referrerpolicy="no-referrer"
  csp="default-src 'none'; style-src 'unsafe-inline'; img-src data:"
  srcdoc="<!-- server-sanitized render fragment -->">
</iframe>

Add auditing on top of this: record the model version for each generated fragment, a hash of the rendered output, the user’s actions, and the final execution result, forming a traceable chain.

4. Putting humans in the loop: interrupt at the right moment

Human participation is not “one extra confirmation step.” It requires distinguishing three intervention patterns and designing an interface for each:

• Prior authorization. High-risk, irreversible, or costly actions (deleting data, initiating a payment, changing production configuration) must be confirmed before execution. The interface should present evidence, scope of impact, and rollback possibility — not just a natural-language sentence.
• Parallel approval. When a branch of a long task needs human judgment, the task should enter a pausable state and wait for the decision rather than occupying a synchronous session.
• Post-hoc review. Present the execution result together with intermediate artifacts, logs, and diffs, and support one-click rollback or conversion into a new task.

This leads to an architectural principle: session state and task state must be separated. The session records interaction history; the task records execution state (queued, running, awaiting human input, succeeded, failed, cancelled). The two are linked by task ID, and the task itself has an independent lifecycle — viewable across devices, resumable across sessions, restartable after disconnection. Stuffing long-running tasks into the conversation context is the root cause of most “it just vanished midway” problems.

5. From a single chat box to an agent workbench

A mature workbench typically spans desktop, mobile, browser extension, terminal, and a background long-task runtime. These surfaces are not redundant:

Surface	Strength	Typical scenario
Terminal	Same environment as the dev toolchain, scriptable	Code search, builds, batch fixes
Browser	Holds page context and session state	Information gathering, form-based operations
Desktop	Local filesystem and process access	Working on repositories, long compiles
Mobile	Always reachable, suited to approval	Receiving notifications, one-shot decisions
Background runtime	Not bound by surface lifecycle	Scheduled jobs, batch processing, long-chain orchestration

The key engineering move is to decouple capabilities from surfaces: a surface only handles rendering and session management, while capabilities register as plugins or skills that declare their required permissions (filesystem read/write scope, network egress, allowlisted executable commands) along with timeouts and concurrency limits. The same set of capabilities can then be reused across surfaces, and the permission model has only one place to review.

6. Standard protocols: reducing connection cost from M×N to M+N

Connecting models, tools, resources, and external systems has long been a Cartesian-product integration problem — every model against every system. An open model-context access protocol exists to reduce that to linear cost: implement a client once on the model side, and a server once on the system side.

A usable protocol should contain at least four primitives and five engineering requirements.

Primitives

• Tools: invocable actions, each with a machine-readable parameter schema and a human-readable description.
• Resources: readable context such as files, database views, and document fragments.
• Prompts: reusable task templates with parameter definitions.
• Sampling: allows the server to make a reverse request for model completion, for lightweight inference needs on the server side.

Engineering requirements

• Capability discovery and version negotiation: the client enumerates server capabilities at startup and degrades explicitly on incompatibility instead of failing silently.
• Least privilege and authentication: authorize at tool granularity; tokens are short-lived, revocable, and session-scoped.
• Validate parameters at the boundary: model-produced parameters are always untrusted, and the server must independently validate type, range, and path.
• Observability: record latency, input summary, return code, and error class for every tool call, and feed them into unified logging and monitoring alerts.
• Normalized error semantics: distinguish parameter errors, insufficient permission, missing resource, rate limiting, and internal faults, so the model can decide whether to retry or stop.

The biggest benefit of adopting a standard protocol is not a few saved lines of code, but avoiding deep coupling to a single model implementation: models become replaceable, tools reusable, and the ecosystem composable.

7. Skills: from one prompt to a small application

Once a team starts accumulating experience in “how to do a class of task,” pure prompts quickly become insufficient. A maintainable skill package typically contains:

my-skill/
├── manifest.yaml      # name, version, description, trigger conditions, declared permissions
├── instructions.md    # model-facing manual: steps, decision criteria, failure handling
├── scripts/           # deterministic scripts: prefer code over step-by-step model reasoning
├── assets/            # templates, configuration, reference docs
├── schema/            # input/output contracts
└── tests/             # golden samples and regression cases

Five engineering conventions go with it:

1. Progressive disclosure. Context is a scarce resource. Only metadata and trigger conditions stay resident; the instruction body and scripts load when actually matched. This also means the manual must be layered: summary first, details after.
2. Determinism first. Anything a script can compute or validate precisely should not be worked out token by token by the model. Scripts are testable and reproducible; the model handles judgment and orchestration.
3. Explicit runtime constraints. Dependencies, executable permissions, network egress, timeouts, and artifact write paths are declared in the manifest, and the runtime generates a sandbox from them rather than relying on convention.
4. Versioning and rollback. Semantic versioning plus a compatibility policy plus gradual rollout. A skill change is a code change and must go through review and regression.
5. Evaluability. Maintain a sample set with expected results, and use pass rate, average step count, and per-task cost as release gates.

8. Reference architecture and cloud capability mapping

A five-layer division that works:

• Access layer: multi-surface session ingress, authentication, rate limiting, protocol conversion.
• Orchestration layer: task decomposition, state machine, human intervention points, retries and compensation.
• Capability layer: standard-protocol clients, tool registry, skill repository with version management.
• Runtime layer: task execution and skill sandboxes, decoupled from surface lifecycle.
• Data and observability layer: session and task persistence, artifact storage, logs, metrics, audit.

Common cloud building blocks map onto those layers as follows:

• Container platform: hosts the long-task runtime and skill sandboxes, combined with resource quotas, image isolation, and elastic scaling.
• Object storage: holds task artifacts, skill resources, and attachments, exposing read-only access through time-limited authorized links.
• Message queue: decouples task dispatch from result reporting, supporting asynchronous long chains and peak shaving.
• Log service + monitoring and alerting: unified collection of tool-call and model-call metrics, with threshold alerts on failure rate, P99 latency, and per-task cost.
• Key management and private network egress: centralized credential custody, on-demand injection, and egress traffic narrowed to an allowlisted proxy.

On Baidu AI Cloud, these capabilities can be composed into a complete agent infrastructure along the layering above: the container platform carries the task runtime, object storage and the message queue handle artifacts and asynchronous chains, logging and monitoring provide observability, and key management plus network policy complete the security boundary.

9. Risks and boundaries

• Usability of generated interfaces. Model-produced layouts are inconsistent in quality. Core flows should be carried by platform-provided components, with the model filling in data; reserve free-form generation for non-critical paths.
• Accessibility and internationalization. Interactive controls need semantic labels, keyboard reachability, and compliant contrast; reserve space for text length differences in advance to avoid truncation and misalignment across languages.
• Context bloat. Rich media output, long-task logs, and tool return bodies all crowd the context. Summaries should replace raw text, retrieval should be on demand, and details should live in external storage fetched only when needed.
• Supply chain risk. Skill packages and plugins are executable code. Sources need signature verification, tightened directory permissions, pinned dependency versions, and diff review for every new version.

10. Summary

Three lines of evolution point in the same direction: an agent’s deliverable is turning from “text” into “programmable interfaces and processes.” Three transitions follow. Design output as structured, interactive objects with a degradation layer. Strip task state out of the session and hand it to an independent runtime. Distill capabilities into skills that carry contracts, carry versions, and are testable, and connect them to external systems through a standard protocol. Do those three things, and an agent stops being a demo and becomes an operable, auditable, continuously improvable production system.