AI Agents: The Engineering Leap from Answering Questions to Getting Work Done

1 hr ago0 views

Large language models have become remarkably good at answering questions, but answering is not the same as delivering. This article examines the engineering leap required to turn LLMs from single-turn text generators into agents that can actually complete multi-step work. It breaks down five technical pillars — task planning, tool calling, context engineering, execution loops, and observability — and highlights three common pitfalls teams encounter when moving agent demos to production. Readers will come away with practical design patterns for building AI agents that hold up in real business environments.

1. The Problem: Answering Questions Is Not the Same as Delivering Results

Over the past two years, large language models have improved rapidly at “answering questions.” Knowledge Q&A, text summarization, code completion, logical reasoning — these tasks share common characteristics: clear input, well-defined boundaries, and output that can be verified offline. The user provides a prompt, the model returns text, and the task is done.

But real business scenarios are rarely that clean. What enterprises actually need is AI that “finishes the job” — querying data, calling APIs, writing files, sending notifications, running tests, fixing bugs — with multi-turn feedback and exception handling along the way. This turns a single-turn generation problem into a multi-step execution problem.

The gap between the two is not about model parameter counts; it is about systems engineering. A model that answers questions is essentially a conditional probability distribution sampler. An agent that gets work done must be an execution system with state, tools, and feedback loops.

This article breaks down the core elements of this capability leap from an engineering perspective and offers practical design approaches.

2. Five Technical Pillars of the Capability Leap

2.1 Task Planning: From Single-Step Generation to Multi-Step Decomposition

In Q&A mode, the model’s output length roughly equals the task length. In execution mode, a single user request may need to be broken down into dozens of sub-steps.

The core of task planning is translating a natural language goal into an executable directed acyclic graph (DAG). A typical approach:

# Pseudocode: basic structure of task decomposition
task_graph = {
    "goal": "Generate quarterly sales report and send it to stakeholders",
    "nodes": [
        {"id": "n1", "action": "query_database", "deps": []},
        {"id": "n2", "action": "aggregate_metrics", "deps": ["n1"]},
        {"id": "n3", "action": "generate_chart", "deps": ["n2"]},
        {"id": "n4", "action": "compose_document", "deps": ["n2", "n3"]},
        {"id": "n5", "action": "send_notification", "deps": ["n4"]},
    ]
}

Three key engineering problems:

Granularity control: Decompose too coarsely and individual steps fail often; decompose too finely and scheduling overhead plus context bloat drag the system down. A useful rule of thumb is that each sub-task corresponds to one tool call or one model inference.
Dependency management: Which steps can run in parallel and which must be sequential should be modeled explicitly at planning time; otherwise, race conditions or deadlocks appear during execution.
Dynamic replanning: When a step fails or returns an unexpected result mid-execution, the system must be able to locally modify the DAG rather than restart from scratch.

2.2 Tool Calling: The Bridge from “Knowing” to “Doing”

A model can only generate tokens. To make it “do things,” you have to give it hands. Tool calling (Function Calling / Tool Use) is those hands.

A mature tool-calling layer needs to address:

Tool description and discovery. Every tool needs a structured description: name, purpose, parameter schema, return format, and error codes. As the number of tools grows, dynamic filtering by scenario becomes necessary to avoid stuffing dozens of tool descriptions into the context.

Parameter validation and correction. Model-generated parameters are not always valid. Validate against the schema before invocation, and return errors in a structured form after invocation so the model gets a chance to self-correct.

Idempotency and safety. Write-operation tools (sending messages, modifying data, placing orders) must use idempotency keys and permission checks to prevent model retries from creating duplicate side effects.

# Pseudocode: the basic tool-calling loop
def execute_tool_call(tool_name, params):
    tool = registry.get(tool_name)
    if not tool:
        return {"error": "TOOL_NOT_FOUND", "hint": "See context for the list of available tools"}
    if not validate_schema(params, tool.schema):
        return {"error": "INVALID_PARAMS", "schema": tool.schema}
    if tool.side_effect and not check_permission(tool, params):
        return {"error": "PERMISSION_DENIED"}
    try:
        result = tool.invoke(params)
        return {"ok": True, "data": result}
    except Exception as e:
        return {"error": "EXECUTION_FAILED", "message": str(e)}

2.3 Context Engineering: Managing Information in Long Tasks

Multi-step execution quickly fills the context window. If you stuff the full history into every round, cost and latency rise sharply, and the model’s attention gets diluted.

Core context engineering strategies:

Layered memory: Split information into working memory (relevant to the current step), short-term memory (relevant to the current task), and long-term memory (accumulated across tasks). Apply different compression and retrieval strategies to each layer.
Structured summarization: After each sub-task completes, compress the raw interaction into a structured record that keeps only conclusions, artifact references, and key decisions.
On-demand retrieval: When historical information is needed, inject it via retrieval rather than full concatenation to reduce token waste.

A practical rule of thumb: always keep the task goal, current progress, to-do list, and the most recent tool return in context, and recall everything else on demand.

2.4 The Execution Loop: Perceive — Decide — Act — Feedback

A system that gets work done must form a closed loop. The problem with open-loop execution: once a step deviates from expectations, every subsequent step builds on that error.

A closed loop consists of four stages:

Perceive: Collect tool returns, environment state, and user feedback.
Decide: Judge whether the current step succeeded; decide whether to retry, reroute, or terminate.
Act: Execute the next operation.
Feedback: Write execution results back into the context in structured form, as the basis for the next decision.

The most error-prone part is success determination. A tool returning “200 OK” does not mean the task succeeded in a semantic sense. You need a result validator layer above the tool layer that uses rules or a model to judge actual outcomes.

2.5 Observability: Making Execution Debuggable and Measurable

One of the biggest engineering challenges of agent systems is unpredictability. The same request may take different paths across two runs. Without observability, production issues are nearly impossible to diagnose.

Core metrics to record:

Task level: success rate, average number of steps, average duration, human intervention rate.
Step level: call count, failure rate, retry rate, and average latency per tool.
Model level: token consumption, inference latency, tool selection accuracy, parameter generation error rate.

Logs should retain the full execution trace, including the input context summary, model output, and tool calls and returns for each round. This serves both as a debugging reference and as training data for future fine-tuning or prompt optimization.

3. From Demo to Production: Three Common Pitfalls

Pitfall 1: Treating an agent as a longer prompt. Many people build agents with instructions like “You are an XX assistant, please complete the following task step by step.” This works in demos, but breaks in production as soon as unusual branches appear. The right approach is to split planning, execution, and validation into independent modules, each with a clear input/output contract.

Pitfall 2: Ignoring failure paths. Tools time out, return dirty data, or hit permission errors. If the system only designs for the happy path, production failure rates will be very high. Every tool call site needs timeout, retry, degradation, and fallback strategies.

Pitfall 3: Pursuing full automation too early. In high-risk scenarios (modifying production data, sending external messages, anything involving money), design human confirmation checkpoints. The value of agents is not fully replacing humans — it is freeing people from repetitive work so they can focus on exception handling.

4. Conclusion

Going from “answering questions” to “getting work done” essentially upgrades the LLM from a text generation component to a task execution system. This upgrade cannot be achieved by simply swapping in a stronger model — it requires solid engineering across five layers: task planning, tool calling, context management, execution loops, and observability.

For developers, this means an expanded skill stack: beyond prompt engineering, you need to understand task orchestration, state management, error handling, permission control, and logging and monitoring — core topics of traditional backend engineering. For technology selection, evaluate an agent solution not only by its scores on standard Q&A benchmarks, but by its actual performance on multi-step tasks, exception branches, and long contexts.

Answering questions is a capability; getting work done is engineering. The latter is the dividing line that determines whether AI can truly enter production systems.