From Answering Questions to Getting Work Done: The Agentic Evolution of Large Language Models

1. The Problem: Answering Capability vs. Execution Capability

For a long time, the evaluation of large language models centered on “answering questions”: knowledge Q&A, text summarization, code completion, and mathematical reasoning. These tasks share common traits—well-defined inputs, clear boundaries, single-turn completion, and results that can be evaluated offline. The model is essentially a conditional probability distribution: given a context, it generates text, and the task is complete.

But when we place models into real production workflows, the problem immediately becomes more complex. What users describe is often not a single question but a goal. “Pull the anomalies out of this batch of logs, categorize them, and write them into a weekly report.” “Generate interface drafts from this requirements document and run the tests.” These requests have no standard answers. They require decomposition, information retrieval, calls to external systems, strategy adjustment based on intermediate results, and self-correction upon failure.

This is the divide between “getting work done” and “answering questions.” Answering is a single mapping; getting work done is a multi-step closed loop. From an engineering perspective, the latter requires four core capabilities: task planning, tool invocation, state memory, and execution verification. This article is organized around these four pillars and discusses how to transform a generative model into a usable execution agent.

2. Pillar One: Task Planning—Breaking Goals into Executable Step Sequences

Planning is the first threshold of agentic capability. A usable planner must accomplish three things: goal decomposition, dependency ordering, and dynamic re-planning.

Goal decomposition hinges on granularity control. Too coarse, and each step remains an unexecutable natural language description. Too fine, and the step count explodes, causing context bloat and error accumulation. A common engineering approach is “two-level decomposition”: first produce a coarse-grained phase list (typically 3–7 steps), then expand each phase into concrete actions as needed. This controls context length while preserving room for on-demand refinement.

Dependency ordering requires the planning output to be a directed structure rather than a linear list. Many tasks have parallel sub-goals (e.g., querying multiple data sources simultaneously) and hard prerequisites (e.g., credentials must be obtained before calling an API). A JSON-described plan should explicitly carry a depends_on field, letting the executor perform topological sorting rather than relying on the model to “remember the order” in natural language.

Dynamic re-planning is the most easily underestimated component. Static plans have a high failure rate in real environments: API timeouts, unexpected response formats, insufficient permissions. A reliable execution agent must evaluate after each step whether “the current state still supports the original plan.” If not, it should trigger local re-planning rather than stubbornly running the full course.

A simplified plan structure:

{
  "goal": "Summarize anomaly logs and generate a weekly report",
  "steps": [
    {"id": "s1", "action": "query_logs", "depends_on": []},
    {"id": "s2", "action": "classify_anomalies", "depends_on": ["s1"]},
    {"id": "s3", "action": "fetch_owner_info", "depends_on": ["s1"]},
    {"id": "s4", "action": "compose_report", "depends_on": ["s2", "s3"]}
  ]
}

The executor schedules based on this. The model is only responsible for semantic decisions at the action level, keeping responsibilities clearly separated.

3. Pillar Two: Tool Invocation—Connecting the Model to the Real World

No matter how elegant the plan, it is empty talk if it cannot be executed. Tool invocation is the sole channel between the execution agent and external systems, and its engineering quality directly determines the system’s ceiling.

The quality of tool descriptions matters more than quantity. A common mistake is piling up dozens of tools with vague descriptions and overlapping parameter semantics. This significantly increases the model’s mis-selection rate. The right approach: each tool’s name, purpose, parameter constraints, and failure semantics should be written clearly in a structured schema, and the description should explicitly state “when not to use it.” The tool set should converge—better to let the model compose multiple steps than to provide semantically ambiguous near-synonym tools.

The invocation protocol should distinguish “decision” from “execution.” The model should output a structured invocation intent (tool name + parameters), not execute directly. The middleware layer handles parameter validation, permission checks, timeout control, and retry policies. The benefit: model-side logic is testable and replayable, while execution-side policies can evolve independently.

def dispatch(intent, registry, context):
    tool = registry.get(intent.name)          # whitelist check
    if tool is None:
        return Err("unknown_tool")
    args = tool.schema.validate(intent.args)  # parameter validation
    if not context.acl.allow(tool, args):     # permission check
        return Err("forbidden")
    return tool.invoke(args, timeout=context.timeout, retries=2)

Return values must be trimmed for model consumption. Raw API responses are often verbose (large JSON blobs, HTML, logs). Dumping them directly into context wastes tokens and dilutes signal. The middleware layer should summarize, filter fields, and normalize errors, compressing results into short structures the model can consume efficiently. This step has an extremely high return on investment but is often overlooked.

4. Pillar Three: Memory Management—Long Tasks in Short Contexts

Multi-step tasks inevitably span multiple model calls, and each call has a limited context window. The essence of memory management is: within a limited window, retain the information most useful for the current decision.

Engineering practice typically divides this into three layers:

• Working memory: the current step’s input, the previous step’s result, and the current plan fragment. This layer must be concise—only include what will be used immediately.
• Task memory: conclusion summaries of completed steps and key intermediate artifacts (e.g., file paths, IDs, status codes). Stored as structured key-value pairs, injected on demand rather than dumping full text back.
• Long-term memory: cross-task preferences, historical experience, and domain knowledge. This layer requires a retrieval mechanism, recalled by relevance, with confidence scores and expiration policies to prevent stale information from polluting decisions.

A common pitfall is concatenating all historical conversation as-is. It seems convenient short-term, but long-term it causes rapid context bloat, drowns key information, and drives up costs linearly. The correct approach: after each step, the middleware generates a “step summary record” that retains only conclusions and downstream dependencies, with the original text archived for reference.

5. Pillar Four: Execution Verification—Letting the System Know When It’s Wrong

The most problematic aspect of an execution agent is not that it can’t do something—it’s that it does something wrong without knowing. Verification mechanisms must exist simultaneously at three levels:

Step-level verification: Does the tool return match the expected schema? Is it empty? Did it hit an error code? These checks are deterministic and should be done by code, not delegated to the model.

Phase-level verification: After a phase completes, check whether its output satisfies the preconditions for the next phase. For example, “Does the classification result cover all anomaly entries?” These checks require some semantic judgment and can be implemented with a combination of lightweight rules and model review.

Terminal verification: Before the task ends, perform a consistency check against the original goal. This step should ideally introduce an independent verification role (which can be another model call or a rule engine) to avoid the blind spot of “the system judging itself.”

Failure handling strategies also need tiering: retryable errors (timeouts, rate limits) go through backoff retries; non-retryable errors (invalid parameters, insufficient permissions) trigger re-planning; semantic errors (results not matching expectations) roll back to the previous checkpoint. Systems without tiered strategies often collapse entirely upon the first exception.

6. Architecture in Practice: A Minimal Viable Execution Skeleton

Stringing the four pillars together yields a minimal closed loop:

User goal
  → Planner (decomposition + ordering)
  → Execution loop:
      Fetch next step → Assemble context (working memory + task memory)
      → Model decision (tool intent)
      → Middleware validation and dispatch
      → Tool execution
      → Result trimming → Write to task memory
      → Step/phase verification → Decide: continue / retry / re-plan
  → Terminal verification → Output

The engineering value of this skeleton: the model appears in only two places—planning and per-step decision-making. Everything else is deterministic code. This leverages the model’s semantic capabilities while building reliability on testable engineering logic. For production systems that need to run long-term, support multi-person collaboration, and be auditable, this separation of responsibilities is essential.

7. Engineering Trade-offs and Best Practices

Trade-off 1: Autonomy vs. controllability. Fully autonomous execution agents are uncontrollable in open environments. Set autonomy levels by task risk: low-risk operations (read-only queries) can be fully automated; medium-risk operations (writing files, sending messages) require receipt confirmation; high-risk operations (deletion, payment, configuration changes) must involve human intervention. Autonomy should be a configuration item, not hard-coded.

Trade-off 2: Step limits vs. task completeness. Maximum step counts and maximum duration must be set to prevent infinite loops and runaway costs. At the same time, design graceful termination: when limits are reached, output “completed portion + reason for incompletion + suggested next steps” rather than silently failing.

Trade-off 3: Model capability vs. engineering compensation. Don’t expect the model to be reliable at every stage. What can be solved with rules (format validation, permission checks, sorting) should never use the model. What must use the model (semantic decomposition, intent recognition) should be stabilized through structured output constraints, few-shot examples, and result review. The more thorough the engineering compensation, the lower the dependence on any single model capability.

Practical advice: Record every execution trace in full (plan, per-step input/output, verification results, timing). This serves both as a debugging reference and as the data source for optimizing planning strategies and tool descriptions. Observability is not an add-on—it is the prerequisite for an execution agent to iterate continuously.

8. Conclusion

From “answering questions” to “getting work done” is fundamentally about embedding a generative model into a controlled execution system. The model provides semantic understanding and decision-making; engineering provides planning and scheduling, tool channels, state management, and verification loops. Only when the boundaries are clear and each part fulfills its role can the system be both flexible and reliable. For developers, the real challenge is not calling the model—it is designing this closed loop rigorously enough. This is precisely the engineering work most worth investing in during the current stage of agentic deployment.