From Answering Questions to Getting Work Done: A Deep Dive into LLM Capability Shifts

1. Shifting the Evaluation Frame: From “What Does It Know?” to “Can It Get the Job Done?”

For the past few years, the default way to evaluate a large language model was to ask what it knew: How broad is its knowledge coverage? How accurate are its answers? How fluent is its generated text? The underlying paradigm was text in, text out. Humans stayed in the loop, breaking tasks into pieces, checking results at each step, while the model only handled the question-and-answer portion of each step.

Recent iterations of flagship models have shifted the evaluation frame in a meaningful way: from “what does it know” to “can it actually get things done.” Typical scenarios now include: autonomously searching for information, cross-validating sources, and producing a research report with citations; reading a multi-module code repository, locating defects, modifying code, running tests, and submitting patches; or navigating a business backend across multiple pages to verify data and export reports.

This is the shift from answering questions to getting work done. This article discusses general industry trends and engineering methods, and does not focus on any specific vendor or model version.

2. Three Key Differences Between Answering Questions and Getting Work Done

1. Interaction Paradigm: From Human-Driven to Goal-Driven

Answering questions is a one-question-one-answer pattern. The human breaks the task into steps and verifies each step in the loop. Getting work done means the human provides only the goal and constraints, while the model plans its own steps, selects tools, checks intermediate outputs, and corrects itself when things fail. The human’s role shifts from operator to reviewer.

2. Output Format: From Natural Language to Structured Actions

When answering questions, the output is text. When getting work done, the output sequence is full of actions: function calls, code execution, API requests, file reads and writes. Natural language appears only during planning and summarization. This places far higher demands on instruction following and the stability of structured outputs.

3. Failure Modes: From Visible Errors to Process Drift

When a model answers incorrectly, the user spots it immediately and can simply ask again. When a model fails at getting work done, the failure often happens silently at some intermediate step: a wrong parameter field, an unnecessarily long path, or citing incorrect data without verification. Supporting real task execution therefore requires self-checking on the model side, plus observability and rollback mechanisms on the engineering side.

3. Four Technical Foundations for Getting Work Done

1. Outcome-Oriented Reinforcement Learning: Teaching Models to Think

Instruction fine-tuning teaches models to speak. Reinforcement learning with task success as the reward signal teaches models to think: positive feedback only comes when the final answer is correct and the task is actually completed. Through extensive trial and error, models learn to look ahead multiple steps, backtrack mid-process, and plan before acting. The visible result is more structured reasoning: models produce a plan before execution, and when deviations appear during execution, they actively correct rather than plowing ahead blindly.

2. Tool Calling: From “You Should Do This” to “I’ll Do It Directly”

Modern models can issue structured calls with parameters, commonly known as function calling or tool calling. The real gap is not whether a model can call tools, but how accurately it does so:

• Parameter accuracy: Can it extract fields correctly from context and format them properly?
• Multi-tool orchestration: Can it chain multiple tools in a sensible order and handle dependencies between them?
• Error handling: When a call fails, can it read the error message and switch to an alternative approach, rather than retrying the same call in a loop?

3. Long Context and External Memory: Fitting the Workspace into the Model

Getting work done often means reading long documents, reviewing large repositories, and tracking long processes. As context windows expand to hundreds of thousands of tokens or more, capabilities like “understanding an entire repository” and “maintaining state across multi-turn tasks” become possible. But engineering teams should stay clear-eyed: long context does not mean unlimited memory, and attention still dilutes with very long inputs. The mature practice is to combine long context, retrieval-augmented generation, and external state storage. Context holds what is currently relevant, external storage holds history and global state, and retrieval recalls information on demand.

4. Code Execution and Sandboxing: Letting Models Think with Their Hands

For tasks like precise calculation, batch data processing, and format conversion, pure text generation is error-prone, while generating code and running it can push accuracy to a level acceptable for engineering use. The required companion is sandbox isolation: restricting network access, filesystem scope, and resource quotas so that execution does not become a security risk.

4. How to Evaluate Whether a Model Can Get Work Done

1. From Static Benchmarks to Dynamic Task Benchmarks

Knowledge benchmarks measure how much a model knows. Agent benchmarks measure how much a model gets done: placing the model in a real or simulated environment such as a browser, terminal, filesystem, or business API, then measuring end-to-end task completion rate, average step count, and failure recovery rate.

2. Three Fine-Grained Metrics Worth Watching

• Planning quality: Is the task decomposition reasonable? Does it involve unnecessary detours or endless loops?
• Tool use accuracy: Parameter correctness, sensible call ordering, and the retry and fallback strategy after failures.
• Robustness: Performance under API errors, missing information, and environmental noise. This is what separates demo-ready from production-ready.

3. Build Your Own Business Benchmark

Public leaderboards are only useful for initial screening. A more reliable approach is to extract 20 to 50 tasks from your actual business, define completion criteria and scoring rules manually, run blind tests across candidate models on the same tasks, rank them by completion rate times per-task cost, and continuously refresh the task set to prevent overfitting to fixed test cases.

5. Developer Practice: Building an Engineering System Around Task-Executing Models

The model provides execution capability. Whether work actually gets done depends mostly on engineering.

1. Adopt a Plan-Execute-Reflect loop so tasks progress through a closed loop rather than being generated in one shot.

2. Write good tool documentation. Tool interface descriptions are the model’s manual: functions, parameters, return structures, error codes, applicable boundaries, and common pitfalls, written like onboarding documentation for a new colleague. Tool granularity should lean toward smaller and more numerous rather than large and all-encompassing. Write operations should be designed to be idempotent where possible to make retries safe.

3. Manage state and memory properly. Short-term state lives in context, mid-term state in compressed task summaries, and long-term state in external storage with retrieval.

4. Four safety guardrails. Enforce least-privilege permissions. Require human approval for high-risk operations like deleting data, sending external messages, or making payments. Set limits on step count, call count, and token consumption to prevent runaway behavior. Maintain full audit logs so every step can be replayed and attributed.

5. Treat agent observability like distributed systems observability. Aggregate task completion rate, average step count, and per-step success rate into dashboards. For failed tasks, replay step by step to locate root causes.

Below is an illustrative generic agent main loop in Python-style pseudocode:

tools = [search, read_doc, run_code, write_file, notify]
state = TaskState(goal=goal, records=[], max_steps=20)
while not state.done and len(state.records) < state.max_steps:
    plan = model.plan(state.goal, state.summary())        # 1. Plan
    action = model.select_tool(plan, tools)               # 2. Select tool and parameters
    result = action.execute(sandbox=True)                # 3. Execute in sandbox
    state.record(action, result)                          # 4. Record and observe
    if result.failed:
        state.record(model.reflect(result.error))         # 5. Reflect and correct
    else:
        state.done = model.verify(state.goal, state.summary())  # 6. Final verification

6. Deployment and Cost Estimation. Task-executing models consume significantly more inference resources: long reasoning chains, multiple rounds of tool calls, and long context inputs can push per-task token costs several times to tens of times higher than single-turn Q&A. Inference compute can be self-hosted or sourced from public cloud model inference services. Key considerations include throughput, latency, concurrency quotas, and pricing transparency, with ROI calculated per task type.

6. Three Hurdles from Demo to Production

1. The Accuracy Hurdle: Demos cherry-pick the easiest cases. Production demands stability across the long tail. Evaluation needs to use stratified sampling of real tasks.

2. The Cost Hurdle: Account for costs per task, not per conversation turn. Deploy agents first on high-frequency, high-labor-cost tasks.

3. The Accountability Hurdle: Task-executing models genuinely modify data, send requests, and reach users. Approval chains and rollback plans must be designed up front, with clear boundaries between model decisions and human confirmation.

7. Conclusion

The shift from answering questions to getting work done is fundamentally about upgrading large language models from knowledge containers to execution agents. For developers, the real barrier is no longer wiring up a chat API, but building an entire execution environment around the model: planning, tools, memory, safety, and observability. For enterprises, model selection should look beyond leaderboards to completion rates and unit costs on their own task sets. Only when both are done well can task-executing models move from launch events into production systems and become accountable, auditable, and scalable digital labor.