How LLMs Learn to Get Work Done: An Engineering Roadmap from Correct Answers to Completed Tasks
Single-turn QA benchmarks tell you whether a model knows the answer, but not whether it can finish the job. This article explains why task completion does not scale linearly from per-step accuracy, then walks through an engineering roadmap for building LLMs that carry multi-step work to a termination condition: task-level verifiable rewards, domain-segmented data cleaning, interleaved planning and execution, agent framework engineering, verifiable UI aesthetics, layered evaluation, and safety boundaries. Developers will learn how to define what "done" means, measure stability, and treat the model as one swappable component in a larger pipeline rather than the whole system.
How LLMs Learn to Get Work Done: An Engineering Roadmap from Correct Answers to Completed Tasks
1. Why “getting the answer right” is no longer enough
For the past two years, the mainstream way to measure model capability has been single-turn QA-style evaluation: give the model a prompt, compare the output against a reference answer, and check whether the knowledge coverage is complete and the phrasing fluent. This kind of evaluation rewards “breadth of knowledge” and “language quality.”
Real engineering tasks have a completely different structure. Requirements are usually vague, acceptance criteria are hidden in code and tests, and the process involves dozens of rounds of reading and writing files, retrieval, compiling, running tests, reading errors, and revising. Any deviation at any stage propagates downstream, and it often cannot be detected in the final answer.
This produces a mathematical fact that is easy to overlook: single-turn accuracy cannot be extrapolated linearly to task completion rate. Suppose a task requires 8 consecutive steps, each with an independent per-step accuracy of 95%. Then the probability of the whole chain passing on the first try is roughly 0.95⁸ ≈ 66%; drop each step to 90% and the completion rate falls to just 43%. This explains why models that “look stronger” still fail on long tasks — the weakness is not in any single point of knowledge, but in error accumulation and state retention.
So the center of gravity must shift: from “given an input, return a high-quality piece of text” to “under environmental constraints and changing state, advance the task to its termination condition.” This shift simultaneously changes the training objective, the data strategy, the inference structure, and the surrounding engineering.
2. Restructuring the training objective: writing “completion rate” into the objective function
The first-principles metric for task capability is the task-level reward, not per-token likelihood. Useful high-value signals share three characteristics: they are verifiable, reproducible, and hard to fake.
- Whether compilation and build succeed
- The pass rate of unit and integration tests
- The number of static checks, type checks, and formatting/style warnings
- When human review is required, a structured score from the review verdict
Two points are critical. First, prefer continuous rewards (pass rate) over binary rewards (pass/fail); otherwise the gradient signal is extremely sparse and training struggles to converge. Second, you must add process penalties; otherwise the model learns to “game the reward” — deleting assertions, modifying tests, or marking failing cases as skipped.
# Sketch of a task-level reward: replace subjective scoring with verifiable signalsdef task_reward(traj, sandbox):result = sandbox.run_suite(traj.final_patch) # build + tests + static analysisif not result.compiles:return -0.5 # build failure, strong penaltyscore = result.pass_ratio # continuous reward, eases sparsity# Process penalties: suppress wasted turns, duplicate calls, over-budget trajectoriespenalty = (0.02 * traj.invalid_calls+ 0.01 * traj.duplicate_calls+ 0.01 * max(0, len(traj.steps) - traj.step_budget))# Anti-gaming guard: tampering with tests counts as cheatingif traj.tampered_with(protected_paths=result.frozen_tests):penalty += 1.0return clamp(score - penalty, -1.0, 1.0)
The corresponding engineering discipline is: the test cases used for acceptance must be isolated from the data visible during training. Freeze the evaluation set, hold back some cases, and apply a time-based split to prevent data contamination from inflating the metrics.
3. Data engineering: validity over scale
“More data is always better” does not hold in task-oriented training, because invalid or low-quality samples not only fail to contribute gains but also consume context budget and dilute attention. A more reliable approach is domain heterogeneity + low-cost validation.
1) Per-domain cleaning rules. The definition of quality differs across domains, so a single set of thresholds cannot be shared:
- Code: require parseable syntax, complete dependency context, and removal of auto-generated noise and repeated templates; preserve the correspondence between issues/commits and code changes so there is a supervision signal for “instruction → modification.”
- Math and reasoning: require complete reasoning chains, verifiable intermediate steps, and verifiable final answers; discard samples that have only a conclusion without a process.
- Web and text: emphasize DOM/structural integrity, runnable styles and scripts, and coherent content semantics.
2) Run ablations on a small model first. Fix the architecture and training budget, replace only the data recipe, and observe the marginal gain:
| Recipe | Code tasks | General QA | Conclusion |
|---|---|---|---|
| A: full mix | +1.2 | -0.4 | general capability regresses |
| B: cleaned code + general replay | +3.5 | +0.1 | adopted |
| C: high-quality code only | +3.8 | -2.1 | noticeable forgetting |
This table reveals two common traps: diminishing marginal returns (continuing to add the same kind of data no longer improves things) and capability trade-offs (specialized gains accompanied by general-capability regression, i.e., catastrophic forgetting). The engineering countermeasures are to keep a general-data replay ratio and to cap the weight of specialized data.
3) Treat the process like software engineering. Ship in phases — a small gray-release traffic slice first, then a full rollout — is the only reliable way to advance a data recipe. Every data-strategy change should leave a traceable record: recipe version, source distribution, cleaning-rule version, and ablation conclusions.
4. Interleaved thinking: let planning and execution alternate
“Think it all through, then answer” works for knowledge QA, but not for multi-step tasks. Environmental feedback (compile errors, test failures, abnormal API responses) only appears after execution, so the plan must be open to being overturned by later observations.
There are three usable thinking modes, combined as needed:
- Persistent thinking: the reasoning block is retained across turns, forming a continuous reasoning context and avoiding re-understanding the task from scratch each turn.
- Turn-level thinking: re-evaluate the plan after each tool return, allowing explicit plan rewrites rather than stubbornly sticking to the original plan.
- No-think direct answer: skip reasoning on low-complexity tasks in exchange for latency and cost advantages.
A qualified multi-step trajectory should take this shape:
[Turn 3]think : The test failure comes from mismatched config key names; the original plan "add an adapter layer" is ineffective; switch to unifying config readsplan v2: 1) locate the config loading entry 2) unify key names with backward compatibility for old keys 3) add one regression testaction: read(path="src/config/loader")expect: confirm where the actual key-name mapping table is defined
Two implementation details decide success or failure. First, thinking content is internal state and should not leak to the user as the final answer; the output layer must distinguish a “reasoning channel” from a “delivery channel.” Second, historical thinking quickly devours the context budget, so compression is mandatory: summarize early steps as “approaches tried + failure reasons” and keep only recent reasoning in full.
5. The model isn’t everything: the weight of framework engineering
An often-underestimated fact is that the final success rate is not determined by the model alone. The agent framework itself (prompt organization, tool design, orchestration strategy) can account for a substantial share of the outcome — commonly estimated in industry practice at around 30%. Three areas deserve the most investment:
1) Layered system prompts. A recommended fixed order: role and task goal → hard constraints (explicit prohibitions) → tool contracts → output format → a few high-quality examples → termination conditions. The principles are constraints up front, a single source of truth, and rules that do not contradict each other. Examples must cover edge cases, otherwise the model treats the examples as the only paradigm.
2) Hierarchical tool-call design. Group by risk and semantics (read / query / write / execute), and isolate execution tools separately with reduced permissions. Write tools should support idempotency keys, and destructive operations should require a second confirmation. Most importantly: write errors as structured information that can guide the next step, not as a message only a human can understand.
{"ok": false, "code": "FILE_LOCKED","message": "The file is locked by another task","retryable": true, "backoff_ms": 500,"next_hint": "Wait and retry, or switch to append mode to avoid overwriting the whole file"}
In practice, improving error messages often yields more than switching models.
3) Stability of multi-agent parallelism. Parallelism speeds things up, but it is also a source of conflicts. Workable rules: converge write operations to a single owner; do not share writable state across task shards; introduce version numbers or optimistic locking on shared resources; make retries idempotent; and define the result-merging strategy before the task begins, to avoid “two branches each finishing half the job.”
6. UI aesthetics is a trainable, verifiable engineering capability
“The pages this generation of models produces suddenly look better” is not magic. Aesthetics can be trained because they decompose into observable dimensions:
- Layout: grid alignment, visual focal point, rhythm of whitespace
- Scale: whether spacing falls on a unified scale system (e.g., a consistent spacing base)
- Hierarchy: font-size gradient, font-weight contrast, clarity of primary vs. secondary relationships
- Color: readability thresholds (a common industry practice is body-text contrast of at least 4.5:1)
- Responsiveness: whether breakpoint behavior is sensible, and whether overflow or misalignment occurs
- Motion: whether durations and easing are consistent, and whether the “reduced motion” preference is respected
- Completeness: whether empty, loading, error, and over-long-text states are all handled
The training-side approach has three steps: collect high-quality UI samples; filter through a dual channel (human spot-checks + automated scoring by a vision-language model, outputting a structured score rather than vague praise); and strengthen “non-semantic capability” as a separate training objective. The generation side adds a self-check layer that turns subjective judgment into an executable check:
def ui_lint(node, theme):issues = []if contrast_ratio(node.fg, node.bg) < 4.5:issues.append(("contrast_low", node.id))if not on_scale(node.padding, theme.spacing_scale):issues.append(("spacing_offscale", node.id))if node.hit_area < 44: # minimum touch target sizeissues.append(("tap_target_too_small", node.id))if overflow_mode(node) == "clip": # content is truncatedissues.append(("content_clipped", node.id))return issues
Once aesthetics is verifiable, it enters a “generate → check → fix” loop, and quality no longer depends on luck.
7. How to measure “can it run the full flow”
Evaluation must be layered; otherwise the score cannot map to real gains:
- L1 single function / single file: algorithmic correctness
- L2 multi-file modules: cross-file references and interface consistency
- L3 repository level: requirement understanding and minimal-change capability
- L4 multi-step tasks with tools and environment interaction: state management, error recovery, termination judgment
Recommended core metrics: end-to-end completion rate (all acceptance scripts green), first-pass rate, average number of rework rounds, human-deliverable rate, cost per task (tokens, number of tool calls, wall-clock time), and the multi-sample consistency rate on the same task — the last one directly reflects stability, and stability is the threshold for whether something can go to production.
8. Implementation recommendations and safety boundaries
- Build your own task set before choosing a model. Start from 20–50 real tickets, each with an executable acceptance script. This is the only evaluation baseline that leaderboards cannot mislead.
- Isolate untrusted artifacts in a sandbox. Model-generated code and commands are untrusted input and should run in a disposable, least-privilege, isolated environment with restricted outbound network access and no mounted production credentials; for irreversible operations such as deletion, release, billing, or permission changes, enforce manual confirmation.
- Layer roles in orchestration. Separate planning, execution, and review; use a different sampling configuration for the review step to reduce self-confirmation bias.
- Observability first. Log the full trajectory (prompts, tool inputs/outputs, latency, token consumption), wire it into a logging service and monitoring/alerting, and build completion-rate and failure-code dashboards by task type.
- Cost governance. Combining four techniques — result caching, parallel fan-out, timeout with backoff retries, and context-summary compression — usually brings long-task costs down significantly.
- Assemble the foundation from general-purpose capabilities. General cloud capabilities such as model serving, container platforms, object storage, message queues, logging services, and monitoring/alerting are enough to support the pipeline above; in public cloud environments such as Baidu AI Cloud, these general capability categories can serve as the foundation for engineering implementation, with specific specs and quotas per the official documentation.
9. Three common misconceptions
- Swapping the model without changing the framework. If the framework stays the same, capability gains get eaten by tool design and error messages.
- Using general leaderboards in place of your own scenarios. Leaderboards have nothing to do with your codebase distribution, dependency complexity, or team conventions.
- Treating aesthetics, documentation, and naming as untrainable. As long as a metric can be defined and verified, it is an engineering problem.
Conclusion
“Getting work done” is not a mysterious attribute of any one generation of models; it is an engineering chain that can be decomposed, measured, and reproduced: at the objective level, replace subjective preference with task-level verifiable rewards; at the data level, replace blind accumulation with domain-segmented cleaning and small-scale ablation; at the inference level, use interleaved thinking to let the plan keep correcting against observations; at the framework level, compensate for what lies beyond the model with layered prompts, structured tool errors, and idempotent orchestration; at the evaluation level, replace single-point leaderboards with layered task sets and stability metrics.
For developers and teams, the most valuable order is fixed: define the standard for “done” first, then talk about model capability. When the acceptance scripts, metric dashboards, and safety boundaries are all in place, switching to any new generation of models means you are just replacing one component in the chain — not redoing the entire engineering effort.
