From Answering Questions to Getting Work Done: Engineering Practices for LLM Agents

In recent years, large language models (LLMs) have undergone a profound paradigm shift: from an early “passive Q&A” stage toward an agentic stage of actively executing tasks. In the past, developers and enterprise users focused on whether a model could accurately answer general knowledge questions or generate a piece of text. Today, the core criterion for judging a model’s strength has become whether it can truly “get work done” — that is, understand complex business intent, orchestrate external tools, and complete automated workflows with real business value end to end.

The Technical Gap from Answer Machine to Executor

In the traditional “answering” mode, an LLM is essentially a probability-based next-token prediction machine. It excels at information summarization and text creation, but it faces two critical limitations: stale information and the inability to act on the physical or digital world. For example, when a user asks to “check yesterday’s departmental cloud resource spending and generate a report,” a pure generative model can neither fetch real-time data nor invoke a report-generation API to complete the task.

To cross this gap and become capable of doing work, a model must have three core foundational capabilities: intent understanding and task decomposition, tool calling, and contextual state management. This requires building a complete Agent engineering architecture on top of a general-purpose LLM.

Core Capabilities: How LLMs Grow “Hands”

1. Structured Output and Function Calling

Getting work done starts with the model understanding instructions and translating them into machine-executable formats. Structured output is the bridge between natural language and code logic. Through function calling, the LLM no longer attempts to answer questions in natural language; instead, based on predefined tool descriptions, it outputs a standardized set of JSON parameters.

The mainstream industry approach is to inject tool schema definitions into the system prompt, guiding the model to emit instructions in a specific format when needed. Below is a generic tool definition structure:

// A tool for retrieving cloud resource monitoring data
{
  "name": "get_cloud_metric_data",
  "description": "Query monitoring metrics for a specified cloud product instance, such as CPU utilization or memory usage",
  "parameters": {
    "type": "object",
    "properties": {
      "instance_id": {
        "type": "string",
        "description": "The unique identifier of the cloud server instance"
      },
      "metric_name": {
        "type": "string",
        "description": "The metric name, e.g. 'CPUUtilization'"
      },
      "start_time": {
        "type": "string",
        "description": "Query start time in ISO 8601 format"
      }
    },
    "required": ["instance_id", "metric_name", "start_time"]
  }
}

When the model receives a request like “show me the CPU load of the web-01 server from yesterday afternoon,” it can identify that the tool above should be called and accurately extract parameters such as instance_id and metric_name. This is the first step from answering questions to getting work done.

2. Multi-Agent Collaboration for Complex Workflows

A single agent with tool calling can only handle simple linear tasks. In complex enterprise scenarios (such as automated operations, code review and fixing, and supply chain scheduling), multiple agents often need to collaborate following a specific workflow. A typical multi-agent architecture includes the following roles:

• Planner: Breaks down a complex high-level goal into a sequence of executable subtasks.
• Executor: The agent that actually calls underlying APIs, executes code, or operates on databases.
• Critic/Reviewer: Validates execution results; if they do not meet expectations, it feeds error information back to the Planner for retry.

For example, in an automated code-fixing scenario, the Reviewer agent analyzes error logs, the Planner agent creates a plan such as “locate the code file -> modify the logic -> run unit tests,” and the Executor agent calls AI coding tools or executes command-line instructions to complete the changes. This divide-and-conquer architecture significantly improves fault tolerance and the completion rate of complex tasks.

3. Memory Mechanisms and Long-Term State Management

Real business tasks are rarely completed in one shot; they may span multiple conversation turns or even several days. The default context window of an LLM cannot support unlimited interaction history. To give the model the ability to “remember what it did yesterday,” memory mechanisms must be introduced.

In engineering practice, a hybrid “short-term memory + long-term memory” architecture is commonly used:

• Short-term memory: Relies on the current conversation’s context window to retain recent interaction details.
• Long-term memory: Summaries of past conversations, key entity information, and intermediate execution results are vectorized and persistently stored. Before each conversation turn begins, relevant memories are retrieved via vector search from the underlying object storage or vector database and injected into the prompt. This allows the model to maintain state consistency when executing tasks that span long periods.

Deployment Challenges and Best Practices for Developers and Enterprises

Although the vision of turning LLMs into productivity tools is compelling, enterprise deployment still faces many challenges.

1. Hallucination Control and Execution Reliability

When a model hallucinates while answering, the result may just be an inaccurate piece of text. But when it hallucinates while doing work, it may call the wrong API and trigger a production incident. To improve reliability, enterprise applications should strictly limit the model’s scope of action:

• Whitelist mechanism: Only register a strictly tested, safe set of tools with the model.
• Human confirmation step: For high-risk operations (such as deleting a database or modifying infrastructure configuration), a human review and approval process must be introduced after the model outputs execution parameters; only after confirmation should the underlying executor run them.
• Observability and trace logging: Use industry-standard logging services and monitoring/alerting platforms to record the input prompt, output parameters, tool execution time, and returned results of every model call, forming a complete trace chain so developers can quickly locate the cause of engineering failures.

2. Integrating Enterprise Private Data with Cloud-Native Architecture

For an LLM to get work done, it often needs to interact deeply with internal enterprise data sources (such as ERP systems, internal knowledge bases, and DevOps platforms). Directly exposing sensitive data to public models poses compliance risks.

The best practice is a hybrid deployment model built on cloud-native architecture: deploy business microservices on a general-purpose container platform to serve as the execution gateway for agents, with these microservices connecting to the private APIs of internal systems. Meanwhile, use standard message queues to asynchronously handle high-concurrency tool invocation requests, ensuring high availability. Under this architecture, the LLM itself is only responsible for “thinking” and “decision-making,” while all actual “doing” happens within the enterprise’s internal security boundary.

Conclusion

The evolution of LLMs from answering questions to getting work done marks the point at which AI moves beyond conversational systems into the far more demanding territory of general task automation. For developers and enterprises, understanding and mastering the core technology stack — structured output, multi-agent orchestration, and memory management — is the key to bridging the gap between concept and production. As model reasoning capabilities continue to strengthen and engineering practices mature, AI will become a tireless digital worker in the digital world, truly delivering the business value of lower costs and higher efficiency.