LLM Agent Engineering in Practice: From Answering Questions to Completing Real Tasks

    This article examines what it takes to move large language models from answering single questions to reliably completing multi-step tasks. It explains how evaluation standards shift from answer quality to task completion, how training objectives become verifiable, and why filtering for effective data matters more than raw scale. It also covers interleaved (ReAct-style) reasoning, the design of stable agent frameworks built on structured prompts, schema-constrained tool calls, and least-privilege safeguards, and the training of non-semantic skills such as UI aesthetics. Finally, it presents end-to-end acceptance metrics, failure attribution, an implementation checklist, and an FAQ for teams building agent systems.

    1. The Evaluation Standard Has Changed: Why Is Completing Tasks Harder?

    For a long time, evaluating a large model meant looking at whether its answers were accurate, whether its knowledge was broad, and whether a single-turn generation was fluent. But once models enter real settings such as software engineering, terminal operations, and complex project understanding, the standard shifts noticeably: the focus is no longer just “is this one answer correct,” but “can this entire task actually be completed.”

    These two capabilities are not equally difficult. Answering a question correctly is single-step correctness; completing a task is multi-step correctness where the steps connect. Take a ten-step process as an example. If each step is 90% accurate on its own, and we assume the steps are independent with no retries or error correction, then the probability of running through in one pass is roughly 0.9 to the tenth power, about 34.9%. Real tasks are usually more complex: there may be dependencies between steps, so an error in an earlier step affects later judgments. At the same time, if you introduce checkpoints, retries, rollbacks, or human confirmation, the overall success rate changes again.

    So the difficulty of “completing tasks” is not whether a single answer looks good, but whether the model can carry out planning, tool calls, state maintenance, error detection, and result finalization across a long chain of steps. It is closer to a systems engineering capability than a single-point language capability.

    2. How the Objective Function Shifts: From One-Shot Answers to Task Completion

    To build models that can “complete tasks,” the first step is often not simply adding parameters, but adjusting the training objective. The core idea is to move the optimization focus from “generating a fluent, correct answer” to “staying on track and finishing the job in complex tasks.”

    This objective can be further pinned down into verifiable forms. For code tasks, signals such as successful compilation, successful execution, passing unit tests, and correct API returns can serve as reward or evaluation criteria. For agent tasks, metrics such as whether tool calls succeed, whether intermediate artifacts are written to disk, whether the task is ultimately completed, and whether the acceptance process passes can be used. Compared with simply judging whether “the answer looks reasonable,” these signals are much closer to real tasks.

    On data mix, three types of samples usually need to be balanced: first, general language ability, to ensure the model can understand intent and express it; second, code and structured task samples, to improve executability; and third, multi-step agent trajectories, which cover the full process of planning, calling tools, reading results, correcting course, and finalizing. For scenarios that combine programming with agents, it is also worth keeping enough “planning, executing, and correcting as you go” trajectory data, so the model learns to stay aligned with the goal amid change.

    In other words, the value of a model is no longer just producing a nice answer, but reliably pushing a complex task forward to a deliverable state.

    3. How to Filter Data: Effectiveness Over Scale

    On the path to “completing tasks,” the data strategy is also changing: not “more is better,” but “more effective is better.” The marginal return of dumping large amounts of undifferentiated data is declining; high-quality, carefully filtered data is the important lever for improving capability.

    A rigorous data engineering process usually includes several layers.

    First, domain-specific cleaning. Tasks such as code, math, and science cannot share the same rules. Code data should be filtered first by verifiable signals such as whether it compiles, whether it runs, and whether it passes tests, removing non-executable, missing-dependency, or obviously incorrect fragments. Math data should ensure the reasoning chain is self-consistent. Science data should focus on factual consistency and verifiable conclusions.

    Second, prioritize effective samples. Samples that reflect the real task distribution should be kept first, especially those containing a complete “problem—reasoning—action—result” loop, rather than large amounts of repetitive, shallow, low-information-density text. For agent data, complete trajectories are often more valuable than isolated question-answer pairs.

    Third, verify before investing. Before each type of data enters large-scale training, it is best to first run ablation experiments on a smaller model with the same architecture and observe the actual gain on the target capability. This idea is similar to canary release in mature software engineering: verify in a small scope first, then scale up the investment, avoiding a large one-time investment that cannot be attributed.

    In addition, pay attention to deduplication between the training corpus and the evaluation set. If evaluation samples leak into training data, the task completion rate may be overestimated, which then misleads later engineering decisions.

    4. How to Implement Interleaved Reasoning: Let the Model Think and Act as It Goes

    Traditional one-shot reasoning usually means “think it through clearly, then output the answer all at once.” But in agent scenarios this approach is risky: if the first step of planning is wrong, all subsequent actions are built on a false premise, and the deviation can keep accumulating.

    A more robust mechanism is interleaved thinking, that is, alternating between “thinking” and “acting.” After each step of execution, the model re-judges the next step based on the latest observation, rather than pre-setting the entire global path at once. This is consistent with the ReAct-style “thought—action—observation” loop.

    A minimal loop can be expressed as:

    1. def run_agent(task, tools, max_steps=20, tool_timeout=15, max_retries=2):
    2. context = build_system_prompt(task, tools)
    3. retry_count = {}
    4. for step in range(max_steps):
    5. decision = model.think_and_act(context)
    6. if decision.type == "final":
    7. return decision.answer
    8. if decision.type == "tool_call":
    9. try:
    10. result = execute_tool(
    11. decision.name,
    12. decision.args,
    13. timeout=tool_timeout
    14. )
    15. context.append(observation(result))
    16. except ToolError as e:
    17. retry_count[decision.name] = retry_count.get(decision.name, 0) + 1
    18. context.append(error_observation(e))
    19. if retry_count[decision.name] > max_retries:
    20. return degrade_gracefully(
    21. context,
    22. reason="tool_failed"
    23. )
    24. return degrade_gracefully(context, reason="max_steps")

    Note in particular that max_steps, tool_timeout, and max_retries should not be set arbitrarily. Their values should be determined based on task complexity, number of tools, average response time, cost of failure, and overall budget. For example, simple question answering may finish in a few steps, while tasks like code repair, data analysis, and page generation may need more rounds. Before going live, stress-test and tune the parameters by observing real trajectories, failure rates, and average time.

    Interleaved reasoning can also be split into two complementary engineering practices.

    One is long-task state maintenance. The longer the task, the more easily the context gets flooded with intermediate information. The system should proactively maintain key state such as compressed summaries, a checklist of key points, completed steps, a file list, and the current blocking reason, so the model does not lose the goal during a long process.

    The other is step-level local reasoning. Before each step’s decision, the model makes a local judgment based on the current goal and the latest observation, rather than relying solely on a one-shot plan made at the start. This reduces the chance of “drifting further and further off course.”

    5. How Do You Design a Stable Agent Framework?

    An easily overlooked fact is that model capability does not equal the final result. In many real tasks, the quality of the agent framework design itself has a significant impact on the task success rate, comparable to that of the model itself. However, the exact magnitude varies by task type, tool complexity, evaluation criteria, and implementation, so it cannot be reduced to a fixed ratio.

    To truly deliver “completing tasks,” the framework side should at least pay attention to three parts.

    1. The System Prompt Should Be Structured

    The system prompt should clearly define the role boundaries, available tools, execution constraints, and output format. For example:

    1. [Role & Boundaries] You are the execution agent responsible for the XX task and may only call the following tools.
    2. [Available Tools] tool_a(arg1): purpose...; tool_b(...): purpose...
    3. [Execution Constraints] Briefly state the intent before each call; retry a failed step no more than N times.
    4. [Output Format] Return JSON as {thought, action, args}.

    A clear, structured prompt reduces the model overstepping boundaries, skipping calls, producing format errors, and adding meaningless explanations.

    2. Tool Calls Should Have Schema Constraints

    The granularity, naming, and parameter structure of a tool should fit the task itself. You can use a structured Schema to constrain the model’s output, making tool calls more stable and easier to parse:

    1. {
    2. "name": "search_index",
    3. "description": "Retrieve relevant snippets from the knowledge base",
    4. "parameters": {
    5. "type": "object",
    6. "properties": {
    7. "query": {"type": "string"},
    8. "top_k": {"type": "integer", "default": 5}
    9. },
    10. "required": ["query"]
    11. }
    12. }

    Beyond the Schema, three engineering notes are worth adding:

    1. Tools should be authorized under the least-privilege principle. High-risk operations such as deletion, transfers, publishing, and committing code must set human confirmation, secondary verification, or permission isolation.
    2. Tool return content and external web content should be treated as untrusted data, and instructions embedded within them must not be executed, to prevent prompt injection.
    3. Failure retries must be idempotent and must set timeouts and count limits, to avoid side effects such as repeatedly writing files, sending repeated requests, or charging twice.

    3. Multi-Agent Collaboration Must Handle State Synchronization

    When a task is split across multiple agents working together, it must be clear who is responsible for planning, who for execution, and who for verification. The system needs to handle state synchronization, conflict resolution, and result merging, so that multiple agents running in parallel do not overwrite each other, execute duplicate work, or pursue conflicting goals.

    6. How Do You Train Soft Skills Like UI Aesthetics?

    Many people notice that interfaces generated by some models suddenly look better. This is not mysterious; it corresponds to non-semantic capabilities such as layout, proportion, hierarchy, and motion. Non-semantic capability refers to abilities that do not directly depend on whether knowledge is right or wrong, but are derived more from induction over large amounts of high-quality examples.

    This capability can be broken down into executable steps.

    First, establish case sources and filtering criteria. You can collect high-quality web pages, open-source projects, design systems, and real product interfaces as corpus. Use vision-language models to score layout rationality, spatial proportion, visual hierarchy, color harmony, and interaction completeness, and set a filtering threshold. Even high-scoring samples should be human spot-checked, to avoid keeping only cases that “look pretty but cannot run.”

    Second, design the training mix and phases. Start by retaining general code generation and frontend fundamentals, then gradually add aesthetically oriented interface samples. The training data should not consist only of screenshots or descriptions; it should also include runnable page code, component structure, style rules, and interaction logic. For real tasks, you can also introduce user feedback or human preference data to help the model move from “looks like a page” to “a deliverable page.”

    Third, establish measurable acceptance metrics. UI aesthetics cannot rely only on subjective feeling; it can be combined with metric-based acceptance, such as an interface aesthetic score, the pass rate for interactive pages, the element completeness rate, the responsive layout pass rate, and a human usability score. Only this way can aesthetic capability turn from occasional performance into a trackable, iterable engineering capability.

    7. How to Evaluate and Debug: How Do You Do End-to-End Acceptance?

    The value of the abilities above must ultimately be tested in an end-to-end flow. The true mark of “completing tasks” is not generating an isolated small snippet, but starting from understanding the requirements, going through planning, coding, calling tools, debugging, and running, and finally producing a verifiable result. For example, generating a small game, completing an interactive page, fixing a piece of code, or getting a script to run.

    Suggested evaluation criteria include:

    • Task pass rate: whether the task is ultimately completed.
    • pass@k: the proportion of cases where at least one attempt succeeds.
    • Step success rate: whether each stage is completed, for example, planning succeeds, the tool call succeeds, the code runs successfully, and the final acceptance succeeds.
    • Failure attribution: distinguishing types such as planning errors, tool errors, state loss, permission errors, prompt injection, and context pollution.
    • Average steps and retry rate: evaluating the system’s efficiency and stability.

    Common failure modes include goal drift, tool misuse, state loss, context being polluted by irrelevant content, retry infinite loops, overstepping permissions, and prompt injection. When troubleshooting, keep the complete trajectory, including the model input, decisions, tool call parameters, return results, error information, and the final state. Only through an observable trajectory can you determine whether the problem comes from the model, the prompt, the tools, permissions, or the task definition.

    8. Engineering Implementation Checklist

    Overall, to build an agent system that can “complete tasks,” you can proceed in the following order:

    1. Align the goal: make clear that what you are optimizing is the task completion rate, not single-turn answer quality, and build the evaluation set and acceptance criteria accordingly.
    2. Polish the data: clean by domain, prioritize verifiable, closed-loop, deduplicated high-quality samples, and run small-scale ablation validation first.
    3. Adopt interleaved reasoning: replace one-shot output with a “thought—action—observation” loop, while adding state retention, timeout control, retry limits, and fallback returns.
    4. Invest in framework engineering: structure the system prompt, design tool Schemas reasonably, improve multi-agent collaboration, and emphasize least privilege and prompt-injection prevention.
    5. Train non-semantic capabilities: break soft skills such as aesthetics, layout, and conventions into specific dimensions that can be filtered, trained, and accepted.
    6. Do end-to-end acceptance: use a complete, runnable flow as the final standard, rather than only checking whether local snippets are correct.
    7. Build a debugging mechanism: keep trajectories, classify failure causes, and regularly review task pass rate and stability.

    In the end, a model moving from “can answer” to “can complete tasks” is not the result of a single technique, but the combined effect of data engineering, training objectives, reasoning mechanisms, framework design, and the evaluation system. For teams that need to build training and inference pipelines, they can advance the engineering work based on foundational capabilities such as general containers, object storage, message queues, and monitoring alerts, but they should be clear about each component’s specific purpose, for example, containers for the runtime environment, object storage for corpus and weights, and monitoring for the evaluation dashboard, avoiding simply equating infrastructure capability with model effectiveness.

    9. Frequently Asked Questions

    How do you choose an agent framework?
    Look first at tool orchestration, state management, observability, permission control, fault tolerance, and cost control. Do not pursue having many components; instead, see whether it fits the task complexity.

    How do you evaluate task completion rate?
    Define clear criteria for task success, and combine automated checks with human acceptance. Common metrics include task pass rate, pass@k, step success rate, retry rate, average steps, and failure attribution.

    What are the common failure modes?
    They include goal drift, tool call errors, state loss, context pollution, retry infinite loops, overstepping permissions, prompt injection, and results that cannot be verified.

    How do you reduce tool call risk?
    Tools should follow the least-privilege principle, and high-risk operations require human confirmation. External return content should be treated as untrusted data. Failure retries should be idempotent and should set timeouts and count limits.