From Answering Questions to Doing Work: The Technical Reality Behind LLM Capability Leaps

1. Why “Getting Work Done” Has Become the New Watershed for Model Updates

Over the past two years, the race to improve large language models centered on answering questions: broader knowledge, more accurate reasoning, better writing style. A user asked a question, the model returned a block of text, and a human judged whether it was right—the classic single-turn Q&A paradigm. Recently, however, the industry narrative has shifted. New model releases emphasize less on static benchmark scores and more on whether the model can “independently complete a task”: write a runnable data processing script, troubleshoot an online alert, or generate an analysis report with consistent data definitions.

The driving force behind this shift is pragmatic. Enterprises are not adopting large models for conversation; they want to augment or replace specific steps in business processes. A model that only answers questions is essentially a knowledge retrieval and text generation tool. A model that can get work done is a digital worker that can be inserted into a workflow. Understanding the technical composition of this leap from answering questions to doing work is therefore the most valuable lens through which developers can evaluate any model update. That is the reality this article aims to unpack.

2. The Fundamental Difference Between Answering Questions and Getting Work Done

The two capabilities may seem like a difference of degree—answering more accurately versus answering more comprehensively—but they actually represent different paradigms, which can be broken down across four dimensions:

Dimension	Answering Questions	Getting Work Done
Interaction Mode	Single-turn or few-turn, one-shot output	Multi-turn, long-horizon interaction with continuous environment feedback
Output Format	Natural language text	Action sequences: tool calls, code, file changes
Evaluation Criteria	Semantic correctness of what is said	Task-level success, machine-verifiable
Cost of Errors	Low; wrong answers can be redone	Errors propagate and amplify at each step; recovery mechanisms required

Three key insights follow. First, the knowledge and reasoning required for answering questions are the foundation for doing work, but they are not the whole picture: doing work also requires the ability to accurately translate reasoning results into structured actions, such as tool parameters. Second, the evaluation criteria change. Success in doing work is often judged not by subjective human assessment but by the environment: whether a query returns data, whether a script executes successfully, whether an API returns the expected result. Third, the weight of error handling rises sharply. In a 20-step task chain, even if each step has a 95% success rate, the overall success rate is only about 36% (0.95 to the power of 20). The real engineering challenge lies in error recovery, not per-step intelligence.

3. Five Technical Pillars Behind Getting Work Done

3.1 Task Planning and Decomposition

Faced with a vague goal like “complete the monthly business analysis,” the model must first decompose it into subtasks: data retrieval, cleaning, calculation, chart generation, and report writing. The quality of planning determines the input to every subsequent step. If the granularity is too coarse, individual steps become too difficult. If too fine, the number of steps explodes and errors accumulate. A common industry practice is to have the model output an explicit plan before executing step by step, while allowing the plan to be dynamically revised during execution.

3.2 Tool Calling (Function Calling / Tool Use)

This is the most direct signal that a model can get work done: the model no longer outputs text for human consumption, but rather a structured call intent—tool name plus parameters—which an external system executes before returning results to the model. Three engineering details determine success. Tool descriptions should be written for the model, with clear semantics, explicit boundaries, and examples. Parameter filling must be accurate, especially for enum values and format constraints. Failed calls need a fallback path so the model does not repeatedly hit the same wall.

3.3 Long Context and Memory Management

Long context windows solve the problem of seeing more at once: reading an entire code repository or a batch of documents before acting. Cross-session memory, however, requires a memory mechanism. A common layering is working memory—the current conversation context—plus episodic memory—historical task trajectories—and long-term knowledge—business rules that have been distilled over time. Retrieval-augmented generation (RAG) is used to recall information on demand, avoiding the cost and noise of stuffing everything into the context window.

3.4 Code Generation and Sandboxed Execution

Code is the model’s universal action language. Data processing, batch operations, and system operations ultimately become scripts or programs. Models that can get work done typically have a closed-loop capability: generate, execute, read errors, and fix. Engineering-wise, a sandbox is essential: network isolation, filesystem isolation, and resource limits. Execution results—standard output, error messages, artifact paths—must be fed back to the model in a structured way for iteration.

3.5 Reflection and Self-Correction

When a single output is unreliable, an observe-reason-act loop can improve quality. The model observes whether each step’s result matches expectations. If not, it analyzes the cause, adjusts its strategy, and tries again. Explicit retry limits and failure escalation prevent infinite loops. This capability is implemented jointly by the model and the orchestration layer, not by the model alone.

The following framework-agnostic pseudocode illustrates the minimal loop of model decision, tool execution, and result feedback:

# Pseudocode: a generic model-tool execution loop, independent of any specific framework
TOOLS = [
    {
        "name": "sql_query",
        "description": "Query the business data warehouse. Read-only SELECT statements only.",
        "parameters": {
            "type": "object",
            "properties": {
                "sql": {"type": "string", "description": "A valid read-only query statement"}
            },
            "required": ["sql"]
        }
    },
]
def run_task(goal: str, max_steps: int = 15):
    messages = [{"role": "user", "content": goal}]
    for _ in range(max_steps):
        resp = model.chat(messages, tools=TOOLS)          # Model decides the next action
        if not resp.tool_calls:
            return resp.text                               # Task complete
        for call in resp.tool_calls:
            try:
                result = execute_tool(call.name, call.args) # External execution
                messages.append(observation(call, result))  # Feed result back
            except ToolError as e:
                messages.append(observation(call, f"Execution failed: {e}"))
    raise RuntimeError("Step limit reached; escalate to human")

This code reveals three engineering points. Execution results must be fed back to the model; otherwise it has no factual basis for further reasoning. Failure messages are equally valid feedback and serve as input for correction. Step limits and human handoff points are essential circuit breakers.

4. Evolving Evaluation: How to Verify a Model Can Really Get Work Done

This is where the gap between marketing and real-world experience is most likely to appear, and where the reality is worth clarifying. Many model updates report improvements on static Q&A benchmarks, but static benchmarks cannot cover multi-step interaction, environment feedback, and error recovery. Higher scores do not necessarily mean higher task completion rates. Developers should build their own evaluations from an agent perspective:

• Source real business tasks—50 to 200 is a reasonable range—not public Q&A datasets.
• Define machine-verifiable success criteria for each task: whether the artifact file is generated with correct content, whether API responses match expectations, whether data is correctly written to the database.
• Fix the tool set and execution environment, then run A/B comparisons between model versions under identical conditions.
• Track four core metrics: task completion rate, average number of steps (efficiency), tool call parameter accuracy, and token cost.
• Maintain a separate set of boundary and adversarial cases: attempts to induce unauthorized tool calls, ambiguous goals, dirty data inputs. Observe failure modes rather than relying only on average scores.

This approach is not costly, yet it directly answers the question that matters most: whether a given update is worth it for your business. It gets closer to the truth than any public leaderboard.

5. Engineering Architecture: Putting a Work-Capable Model into Production

A model alone cannot be production-ready. A layered architecture is required:

1. Model layer: a unified inference entry point that abstracts away model-specific differences and enables version switching and canary releases.
2. Orchestration layer: responsible for task decomposition, flow control, retries, and degradation. This is the agent’s scheduling center.
3. Tool layer: internal APIs, databases, and scripts are wrapped as permission-controlled tools, with parameter validation and allowlists enforced first.
4. Memory layer: short-term context and long-term memory are stored separately, with retrieval-based injection on demand.
5. Observability layer: full-trajectory logs, cost tracking, and failure attribution, with replay and audit support for any task.

For cloud deployment, common platform capabilities can be combined: container platforms to host the orchestrator and tool executors with elastic scaling; object storage to persist task artifacts and memory snapshots; message queues to decouple asynchronous execution of long-running tasks; logging and monitoring services for end-to-end auditability. The key selection criterion is not which individual component is more powerful, but whether the permission model is unified and whether the audit trail is end-to-end. The security boundary of an agent application is defined by these infrastructure choices together.

Human-in-the-loop design is equally critical. High-risk actions—write operations, fund transfers, external communications—should retain manual confirmation nodes. Set confidence thresholds and sampling reviews for automated execution. Follow the principle of least privilege: each agent should receive only the subset of tools necessary to complete its task.

6. Enterprise Rollout Risk Checklist

Before going live, review each item:

• Amplified hallucination: in agent scenarios, a model may fabricate parameters, call nonexistent tools, or invent execution results. Dual validation of parameters and results at the tool layer is essential.
• Permission abuse: once a tool is mounted, the model has effectively been granted the corresponding capability. Permission boundaries must be explicitly declared and audited regularly.
• Cost escalation: multi-step loops inherently amplify token consumption. Set budgets and circuit breakers at the task level.
• Evaluation drift: changes in business definitions can invalidate old evaluation sets. Evaluation assets must be maintained in sync with the business.
• Accountability boundaries: automated actions must be traceable and reversible. When something goes wrong, you need to locate the specific step and decision responsible.

7. Conclusion

The shift from answering questions to doing work is not an ordinary version upgrade. It is a paradigm migration: the object of evaluation changes from a piece of text to a completed outcome, and the engineering focus expands from prompt design to tool ecosystems, orchestration logic, and security governance. For developers, when evaluating any model update, the most valuable approach is not to look at benchmark scores but to run your own business task set through a full task loop. That is the closest you can get to the truth.