From Answering Questions to Doing Work: Technical Insights and Engineering Practices for LLM Agents

I. The Essential Difference Between Answering and Doing

Over the past two years, the most intuitive impression of Large Language Models (LLMs) has been their ability to “answer questions”: a user inputs a prompt, and the model generates a response. This process is essentially stateless text generation—the model does not perceive the external environment, produces no side effects, and is not accountable for the results.

“Doing work,” however, means the model must enter real business workflows: it needs to understand goals, break down steps, call tools, observe results, revise plans, and ultimately deliver a verifiable outcome. For example, instead of merely asking “help me write a weekly report,” a user might say, “summarize this week’s project progress, draft it into a weekly report, and send it to relevant team members for review.”

The difference between these two paradigms can be abstracted into four dimensions:

Dimension	Answering Questions	Doing Work
Task Granularity	Single-turn dialogue	Multi-step, long-cycle
External Interaction	None	Call APIs, read/write data, operate files
State Management	Stateless	Requires maintaining execution context
Result Verification	Manual judgment	Executable, verifiable, rollback-capable

Therefore, when discussing “how powerful a new model is,” the real focus shouldn’t be parameter scale or benchmark scores, but rather its comprehensive execution capability within an agentic framework. This capability is determined by multiple engineering modules, not just the model itself.

II. The Agent Capability Stack: From Model to Execution Loop

An agent capable of “doing work” typically consists of five layers of capability:

1. Instruction Understanding and Intent Decomposition

The model needs to translate natural language goals into structured execution plans. This requires strong instruction following and task decomposition capabilities. For instance, given the prompt “analyze the error logs from the past week and provide fix recommendations,” the model must identify four key elements: time range, data source, analysis objective, and output format. It then maps these to a subsequent tool-calling sequence.

In practice, it is highly recommended to use structured prompts to constrain the decomposition process:

You are a task planner. Please decompose the user's goal into executable steps.
Each step must include:
- step_id: Step number
- action: Action type (query / read / write / notify)
- target: Operation target (e.g., logging service, document library, messaging channel)
- expected_output: Expected output
Only output a JSON array, do not output anything else.

This constraint significantly improves the determinism of subsequent tool calls.

2. Tool Calling and Interface Abstraction

The core of “doing work” is interacting with external systems. Whether querying a database, calling a search API, manipulating object storage, or sending messages, the model needs stable and reliable tool-calling capabilities.

From an engineering perspective, it is recommended to abstract all external capabilities into a unified tool description structure, containing the tool name, functional description, parameter schema, and return format specification. The model only needs to decide “which tool to call and what parameters to pass,” while the actual execution is handled by the runtime. This reduces the probability of model errors and facilitates permission control and auditing.

{
  "name": "query_error_logs",
  "description": "Query error logs within a specified time range",
  "parameters": {
    "type": "object",
    "properties": {
      "start_time": {"type": "string", "format": "date-time"},
      "end_time": {"type": "string", "format": "date-time"},
      "keyword": {"type": "string"},
      "limit": {"type": "integer", "default": 50}
    },
    "required": ["start_time", "end_time"]
  }
}

Once the model returns a tool call request, the runtime executes it and passes the result back to the model, forming a “model decision -> tool execution -> result observation” loop.

3. Context Engineering and Memory Management

In long-cycle tasks, the context expands rapidly. Indiscriminately feeding all historical information to the model not only increases costs but also dilutes critical information and degrades reasoning quality.

A practical approach is hierarchical context management:

• Working Memory: Inputs, tool returns, and intermediate conclusions directly related to the current step.
• Task State: Overall goal, completed steps, current step, and pending to-dos.
• Long-term Memory: User preferences, historical decisions, and domain knowledge that can be reused in future tasks.

Take the “weekly report generation” task as an example. When executing the “summarize project progress” step, the model only needs to see the current step’s query results and the task objective, rather than bringing in the entire week’s conversation history. The essence of context management is: providing the model with the right information at the right time.

4. Execution Control and Error Recovery

Real business environments are full of uncertainties: API timeouts, missing data, insufficient permissions, and abnormally formatted tool responses. The ability to gracefully handle these scenarios is what separates a “demo agent” from a “production-grade agent.”

It is recommended to introduce the following control mechanisms into the agent runtime:

• Timeout and Retry: Set a timeout threshold for each tool call; allow a limited number of retries for idempotent operations.
• Conditional Branching: Determine the next step based on the status code or field values returned by the tool.
• Human-in-the-loop: Pause execution and notify a human for confirmation when consecutive failures occur or confidence falls below a threshold.
• Rollback-capable Operations: For write operations, design them to be reversible or accompanied by audit logs.

Execution strategy example:
1. Call the query API with a 5-second timeout;
2. If it returns 200 and data is non-empty, proceed to the analysis step;
3. If it returns empty data, expand the time range and retry once;
4. If it fails twice in a row, mark as an exception and notify the user.

This strategy needs to be tailored to specific business scenarios, but it outlines the necessary deterministic guarantees required for an agent to “do work.”

5. Result Delivery and Verifiability

The final stage of “doing work” is delivering the result in a verifiable and consumable format. This requires the agent to output not just text, but also an execution summary, key data sources, exception descriptions, and follow-up recommendations.

For example, the final delivery of a log analysis task could include:

• Error type distribution and trend overview;
• Key error samples and tracing clues;
• Executed remediation actions or suggested fix plans;
• Exceptions encountered and unfinished items during execution.

This structured delivery elevates the agent’s output from “looking reasonable” to “being verifiable by the business.”

III. Key Practices for Engineering Implementation

1. Go “Narrow and Deep” Before “Broad and Generic”

Do not attempt to build an omnipotent general-purpose agent from day one. A more pragmatic path is to select one or two high-value business scenarios with clear boundaries, such as “log anomaly summarization and notification” or “ticket auto-classification and routing,” and get the execution loop working first. Clearer boundaries and more controllable toolsets lead to a higher probability of successful task completion by the model.

2. Treat the Model as a “Decision Maker,” Not an “Executor”

In an agent architecture, the model is responsible for judgment and planning, while the runtime handles actual execution. Instead of letting the model concatenate raw SQL strings to pass to a database, have the model output structured query intents, which are then executed via predefined parameterized queries. This prevents injection risks and enables fine-grained permission validation.

3. Build Evaluation Sets, Don’t Just Settle for “It Runs”

“Running successfully” is merely the baseline. You need to build evaluation sets tailored to specific scenarios, covering normal flows, boundary conditions, and exceptions. Evaluation metrics can include task completion rate, average execution steps, tool call accuracy, human intervention ratio, and end-to-end latency. Only what is quantifiable can be continuously optimized.

4. Retain Trajectories for Every Execution

Production-grade agents must log every tool call, model decision, and result feedback. These trajectory logs are used not only for troubleshooting but also as core assets for subsequent prompt optimization, tool description adjustments, and edge case discovery.

IV. The Relationship Between Model Evolution and the Technical Foundation

The inherent capability of the model dictates the ceiling of the agent’s performance, while the engineering capabilities provided by the cloud infrastructure determine how much of that ceiling can be realized. Current mainstream agent development paths typically rely on the following general cloud capabilities:

• Model Services: Providing inference APIs for high-quality foundation models;
• Container Platforms: Hosting the agent runtime to ensure isolation and elastic scaling;
• Object Storage: Storing execution artifacts, evaluation sets, and trajectory logs;
• Message Queues: Decoupling asynchronous steps in long-chain tasks;
• Logging and Monitoring: Tracing execution chains to detect anomalies and performance bottlenecks promptly.

This combination is not tied to any specific vendor; developers can build it using general-purpose services. The key is engineering the model’s capabilities rather than expecting the model itself to solve everything.

V. Conclusion

The transition from “answering questions” to “doing work” is fundamentally the process of an agent evolving from single-point generation capability to end-to-end execution capability. There is no single technological singularity in this process; it is jointly driven by multiple engineering modules, including prompt design, tool abstraction, context management, execution control, and result verification.

For developers, the more pragmatic focus is not on “how powerful” a single model update is, but rather: in your business scenario, can you build a stable, observable, and optimizable execution pipeline using currently available models and services? That is the true hallmark of an agent starting to “do work.”