AI Agent Workflows: From Conversation to Execution in Production
This article explains how AI agent execution layers move LLM applications beyond chat by connecting models to business systems with structured task planning, tool invocation, permission boundaries, approvals, recoverable state, and audit trails. It covers the limits of pure conversation, core runtime capabilities, a reference execution loop, rollout steps, evaluation metrics, and key risks.
1. Background: Evaluation Criteria for AI Applications Are Changing
Over the past two years, the main form of LLM applications has been the chat box. Users enter a question, the model returns an answer, and product metrics tend to focus on active users, time spent, and number of questions asked. This model works well during user acquisition, but in enterprise scenarios the problem quickly becomes clear: no matter how good an answer is, its value is hard to calculate if it cannot be tied to a concrete business action.
Enterprises really care about questions such as these: Can an expense report be automatically checked and submitted for approval? Can a customer ticket be assigned directly to the right team with the status written back? Can a weekly report pull metrics from data sources, generate a draft, and send it to the owner? These are requirements for “getting things done.” They require AI to have execution capability, not only expression capability.
As a result, industry attention is shifting from “what the model can say” to “what the system can complete.” This shift has given rise to a new technical layer: the agent execution layer.
2. Limitations of Pure Conversation
Pure conversation faces four structural problems in enterprise settings.
- Lack of business context. General chat models do not understand an organization’s structure, approval rules, or data definitions, so they can only provide generic suggestions.
- No access to systems. Answers remain at the text layer, and the final step still requires manual copy-and-paste into business systems.
- Uncontrollable permissions. If a model is given account credentials directly, its permission boundary cannot be audited.
- Hard-to-quantify value. The output is text and cannot be directly tied to measurable metrics such as order volume, work hours, or defect counts.
These problems show that simply adding model capability or expanding chat entry points does not automatically solve enterprise deployment. The key is to connect the model to existing systems and establish explicit execution boundaries for it.
3. Core Capabilities of an Execution Layer
A usable agent execution layer typically includes the following capabilities.
3.1 Task Understanding and Decomposition
User goals are often vague, for example, “organize last week’s production incidents and generate a postmortem draft.” The execution layer needs to break the goal into executable steps: query incident records, summarize impact scope, extract the timeline, and generate the document structure. The decomposition result should be a structured plan rather than text that only humans can read, so that subsequent steps can be executed and validated programmatically.
3.2 Tool Invocation
Tools are the only channel through which an agent interacts with external systems. Each tool should have a clear name, input parameter definitions, return structure, and error codes. Common tool categories include:
- Data query: access databases, data warehouses, or metric platforms
- File processing: read and write documents and spreadsheets in object storage
- Messaging: send in-app messages, email, or instant messaging messages
- Ticket workflow: create, update, and close business tickets
- Code execution: run scripts in an isolated sandbox and return results
Tool definitions must follow the principle of least privilege. An agent that only needs to read order status should not have a tool for modifying orders.
3.3 Permissions and Data Boundaries
Permission control is the baseline for enterprise scenarios. Recommended practices include:
- Assign an independent service identity to the agent instead of reusing a personal account
- Configure permissions at tool granularity and align them with the existing role system
- Mask sensitive fields before placing them into the model context
- Apply secondary validation to write operations, such as amount limits and approver allowlists
3.4 Approval and Human Intervention
High-risk actions should have approval nodes. When the execution layer encounters operations such as deletion, transfers, or external sending, it should pause and generate a confirmation request, then continue only after the responsible person confirms. Approval nodes also serve another purpose: they provide a final interception point for model errors.
3.5 State Management and Recoverable Execution
Multi-step tasks may be interrupted by network timeouts, API rate limits, or abnormal intermediate results. The execution layer should persist the input, output, and state of each step so that a task can resume from the failed step instead of starting over. Idempotency design is critical here: when the same write operation is executed repeatedly, the result should remain consistent.
3.6 Observability and Auditing
Every tool call should record the caller, parameter summary, return status, latency, and associated task ID. Audit logs are used both for troubleshooting and as a basis for compliance checks. Logs must not store plaintext secrets or full sensitive personal information.
4. Reference Architecture and Example Code
Below is a simplified execution loop example to illustrate the basic flow of “plan, validate, execute, record.” The example is only for understanding the structure; production implementations also need error handling, timeout control, and concurrency management.
from dataclasses import dataclass, fieldfrom typing import Callable, Any@dataclassclass Tool:name: strhandler: Callable[..., Any]requires_approval: bool = False@dataclassclass Step:tool: strargs: dictstatus: str = "pending"result: Any = None@dataclassclass Task:goal: strsteps: list[Step] = field(default_factory=list)class AgentRuntime:def __init__(self, tools: dict[str, Tool], approver: Callable[[Step], bool]):self.tools = toolsself.approver = approverdef run(self, task: Task) -> Task:for step in task.steps:if step.status == "done":continue # supports resuming from checkpointstool = self.tools.get(step.tool)if tool is None:step.status = "rejected"continueif tool.requires_approval and not self.approver(step):step.status = "awaiting_approval"breaktry:step.result = tool.handler(**step.args)step.status = "done"except Exception as exc:step.status = "failed"step.result = str(exc)breakself._audit(task, step)return taskdef _audit(self, task: Task, step: Step) -> None:print(f"[audit] goal={task.goal!r} tool={step.tool} status={step.status}")
This code reflects several design points: tools are centrally managed as a registry, and calls to unregistered tools are rejected; write operations can be marked as requiring approval; each step has an explicit status to support resuming from checkpoints; and each execution writes an audit record.
5. Implementation Steps
It is advisable to proceed in the following order and avoid pursuing full automation from the outset.
- Choose a high-frequency, clearly rule-based, reversible business scenario, such as weekly report generation or ticket assignment.
- Map the system interfaces involved in the scenario, organize available operations into a tool list, and retain only the necessary permissions.
- Start in a “read-only plus draft” mode, letting the agent generate results for human review.
- After accumulating data, enable automatic execution for low-risk write operations while keeping approvals for high-risk operations.
- Build an evaluation set and periodically retest accuracy, failure reasons, and the human modification rate.
6. Evaluation Metrics
Implementation effectiveness should not be measured only by traffic. More useful metrics include:
- Task completion rate: the share of tasks completed end-to-end without human takeover
- First-pass acceptance rate: the share of results adopted without modification during human review
- Average processing time: the change in time from task submission to result delivery
- Tool error rate: the proportion of failed calls, counted by tool category
- Permission-violation interception count: the number of abnormal requests blocked by permission checks or approval nodes
These metrics map directly to business cost and quality, making it easier to judge whether continued investment is worthwhile.
7. Risks and Considerations
- Model output is uncertain. Critical fields must pass rule validation and cannot be trusted directly.
- Prompt injection risk: external documents or web content may contain embedded instructions. The execution layer should treat such content as data rather than commands and restrict the scope of tools it can trigger.
- Responsibility must be agreed in advance: agent operations should be attributed to a specific business owner, not to a vague “system.”
- Cost must be designed in: multi-turn reasoning and long context increase invocation cost, so consumption should be controlled through caching, summarization, and step trimming.
8. Conclusion
The competitive focus of AI applications is shifting from “making people willing to open” to “making systems able to deliver.” For developers, the key work is to build clear tool boundaries, reliable state management, strict permission control, and traceable audit trails. Only when this infrastructure is in place can agents move from demo scenarios into real business workflows.
