Agent Engineering: Moving LLMs from Answering Questions to Reliably Delivering Tasks

4 hr ago0 views

This article explains the engineering layer required to turn LLM capabilities into reliable task execution, moving beyond chat, fluency, and benchmark metrics. It presents a five-layer reference architecture covering models, context management, execution runtime, tools and integrations, and evaluation/observability. It focuses on runtime loops and termination conditions, context control, tool protocols, stability practices such as layered timeouts, retries, sandboxing, state persistence, and human checkpoints. It also covers trajectory-level evaluation, CI gates, observability, cost governance, organizational coordination, a phased rollout path, and common anti-patterns for production agents.

1. The starting point: model capability has improved, but task completion has not

Over the past two years, the industry’s way of evaluating large models has changed noticeably. Early discussions focused on the fluency of single-turn answers, leaderboard scores, and context length. More and more teams now ask a different question: if given a real objective, can the model complete the entire process on its own and deliver the result reliably?

A whole engineering layer sits between these two questions. When a model answers “how do I troubleshoot this service’s memory leak,” it generates text. But when an agent completes the same goal, it needs to read monitoring metrics, pull logs, identify version differences, form hypotheses, execute verification commands, adjust direction based on the returned results, and stop to ask a human when information is insufficient. The former is a single inference. The latter is a long-running control system with side effects, possible failure, and recovery requirements.

This is the engineering meaning of moving “from model research to a product system”: when model capability is no longer the only bottleneck, the competitive focus shifts to what surrounds the model — how to make AI truly execute tasks, run stably, be measurable, and be governable. The industry has used many names for this surrounding system. At its core, it is a combination of an agent runtime, a tool ecosystem, and an evaluation/observability system.

This article does not discuss any specific vendor implementation. Instead, it breaks down this system into reusable engineering structures.

2. A five-layer reference architecture for agent systems

Layering an agent system helps avoid a common failure mode: stuffing all logic into a prompt. A sustainable agent platform can be roughly divided into five layers:

Layer	Responsibility	Common failure symptoms
L0: Model layer	Reasoning, planning, code/structured output	Unstable instruction following, drifting during long tasks
L1: Context layer	Deciding what the model should see at the moment	Context explosion, key information drowned out
L2: Execution runtime	Loop control, state machines, termination, recovery	Infinite loops, repeated calls, restarting from scratch after failure
L3: Tools and integration layer	Wrapping business systems into reliable, callable capabilities	Vague tool descriptions, error messages that the model cannot use
L4: Evaluation and observability layer	Measuring results, process, and cost; setting regression gates	Tuning prompts by intuition; one change breaks many things

Many teams concentrate too much effort on L0 — switching models or tweaking prompts — while most production incidents occur at L2 and L3. The following sections expand on each layer.

3. L2 execution runtime: the agent “kernel”

The runtime is the most fundamental difference between an agent and a chatbot. It solves problems very similar to those of an operating system kernel: process scheduling, resource quotas, state persistence, and exception recovery.

3.1 Main loop and termination conditions

The core of an agent is a constrained loop, not a loop that can “run for as long as it wants.” Four exit conditions must be explicitly defined: goal achieved, budget exhausted, no progress for several consecutive steps, and human intervention required.

# Illustrative pseudocode: a controlled agent main loop
def run_agent(goal: str, ctx: Context, budget: Budget) -> Result:
    trace = Trace.start(goal)
    while not budget.exhausted():
        # 1. Assemble context (subject to token budget)
        prompt = ctx.render(max_tokens=budget.context_tokens())
        # 2. Reason and decide
        decision = model.act(prompt, tools=ctx.tool_schemas())
        if decision.type == "final":
            return Result.ok(decision.answer, trace)
        if decision.type == "ask_human":
            return Result.need_input(decision.question, trace)
        # 3. Execute tool with timeout and quota
        observation = executor.call(
            decision.tool,
            decision.args,
            timeout=budget.tool_timeout(decision.tool),
            idempotency_key=make_key(goal, decision),
        )
        # 4. No-progress detection: switch strategy if the same call repeats
        if trace.is_repeating(decision, window=3):
            ctx.force_strategy_switch(reason="repeated_action")
        ctx.append(decision, observation)
    return Result.budget_exceeded(trace)

Key design points:

Budgets are first-class citizens: token count, number of tool calls, wall-clock time, and monetary cost should be set at the entry point, not discovered after the fact.
No-progress detection: if the same tool call with the same arguments appears three times in a row, the model is likely stuck in a loop. The most effective intervention at that point is usually not to retry, but to change strategy — switch retrieval sources, switch tools, narrow task granularity, or escalate to a human.
Observations must be returned: error messages from tools must be cleaned up and fed back to the model. “Permission denied” is less useful than The permission needs to be enabled in the project settings; the current role is viewer. The readability of tool errors directly determines an agent’s ability to recover on its own.

3.2 Context window management: more context is not always better

The real killer in long tasks is context rot: irrelevant content accumulates, key information becomes diluted, model attention drifts, and cost rises linearly. Engineering can use three complementary methods:

Structured compression: compress early conversation into a three-part summary — “completed facts + open questions + ruled-out hypotheses” — rather than a free-text summary.
External memory: keep long-text assets such as documents, logs, and code on the storage side. The context should contain only identifiers and retrieval results, with assets pulled on demand. Object storage is suitable for session snapshots and intermediate artifacts, while retrieval indexes are maintained separately.
Layered memory: define separate write and recall rules for short-term memory (current task state), mid-term memory (conclusions from the current session), and long-term memory (user preferences, project conventions). Otherwise, these layers will contaminate each other.

3.3 Tool protocolization: turning “functions” into capabilities the model can understand

The quality of the tool layer determines the ceiling of agent capability. A proper tool definition should include at least: a clear semantic description, a JSON Schema for parameters, an enum of error codes, side-effect labeling, and timeout/idempotency semantics.

{
  "name": "query_service_metrics",
  "description": "Query aggregated metrics for a given service within a specified time window. Read-only, no side effects.",
  "parameters": {
    "type": "object",
    "required": ["service", "metric", "range"],
    "properties": {
      "service": {"type": "string", "description": "Service identifier, such as order-api"},
      "metric":  {"type": "string", "enum": ["qps", "p99_latency", "error_rate"]},
      "range":   {"type": "string", "description": "ISO8601 time range"}
    }
  },
  "side_effect": "none",
  "timeout_ms": 3000,
  "idempotent": true
}

Two practical recommendations: provide enums instead of free-form text (which significantly reduces invalid calls), and label whether a tool has side effects or not, because the runtime needs this information to decide whether automatic retries are safe.

4. Stability engineering: turning “occasional success” into “predictable delivery”

A 70% success rate may look good in a demo, but in production it means 30% of requests need human backup. Stability should be built along four dimensions.

4.1 Layered timeouts and backoff retries

You cannot set a single timeout for the entire agent. Timeouts must be set layer by layer: for a single model call, a single tool, a single loop iteration, and the whole task. Retry strategies should be routed by error type:

RETRYABLE = {"RATE_LIMITED", "TIMEOUT", "SERVICE_UNAVAILABLE"}
def call_tool(tool, args, attempt=0):
    try:
        return tool.invoke(args)
    except ToolError as e:
        if e.code not in RETRYABLE or attempt >= 3:
            # Non-retryable errors must be passed back verbatim so the model can decide what to do next
            raise ObservationError(e.code, e.human_message)
        if not tool.idempotent:
            # Do not blindly retry non-idempotent operations
            raise ObservationError("NON_IDEMPOTENT", "The operation has been submitted; check its status first")
        sleep(backoff(attempt, base=0.5, cap=8, jitter=True))
        return call_tool(tool, args, attempt + 1)

Non-idempotent operations should not be automatically retried. This is one of the easiest places to get into trouble. Write-type tools should support idempotency keys so that retries become safe operations.

4.2 Side-effect isolation and sandboxing

Code execution, file modification, and external requests are the highest-risk operations. They need independent sandboxes, resource quotas, network egress allowlists, and least-privilege credentials. The principle is simple: the permissions an agent receives should always be smaller than the permissions the person operating it already has in the system.

4.3 State persistence and resumable execution

The decision, tool input parameters, and observation result of every step should be persisted (message queues decouple tasks, object storage stores snapshots, and relational storage stores structured state). This brings three benefits: execution can resume after a crash, runs can be replayed offline, and the data can be used as an evaluation dataset. Without complete trace retention, there is no iteration capability for agents.

4.4 Human-in-the-loop checkpoints

Do not try to make the agent fully autonomous. Classify operations by “reversibility × impact scope”: low-risk read-only operations can run automatically; medium-risk operations should generate a plan for human confirmation; high-risk operations should only produce recommendations and not execute. Checkpoints are a product capability, not a compromise.

5. L4 evaluation: no trajectory-level evaluation means no iteration

Agent evaluation is an order of magnitude harder than model evaluation because it involves multi-step decisions, tool uncertainty, and path diversity. A practical approach is to split metrics into three layers:

Result metrics: whether the task was completed, whether the result was correct, and whether human rework was required.
Process metrics: whether the number of steps was redundant, whether tool selection was reasonable, whether repeated calls or infinite loops occurred, and the error self-recovery rate.
Cost metrics: token consumption, number of tool calls, end-to-end latency, and cost per task.

For dataset construction, maintain a regression set: filter typical successful and failed cases from real production traces, then fix them as repeatable test cases. Any change to prompts, model versions, or tool definitions should first run against the regression set, with the pass rate used as a release gate.

# Illustrative CI gate configuration
agent_eval:
  dataset: regression_v7        # 200 real trajectories
  gates:
    task_success_rate: ">= 0.85"
    redundant_step_ratio: "<= 0.20"
    tool_error_self_recovery: ">= 0.60"
    avg_cost_per_task: "<= baseline * 1.05"
  on_fail: block_release

It should be emphasized that gate thresholds must be determined by each team through testing in its own scenario. Any so-called “industry standard” copied from elsewhere is unreliable.

6. Observability and cost governance

The debugging unit of an agent is not “one request,” but “one trajectory.” Therefore, the observability system should be modeled around trajectories:

End-to-end tracing: use a trace_id to connect model calls, tool calls, retrieval, and human intervention. Each step should record input/output summaries, duration, token count, and error codes.
Separate logs and alerts: logging services handle fine-grained traces, while monitoring and alerting should focus only on business-level metrics, such as sudden drops in task success rate, rising tool error rates, and abnormal per-task cost.
Cost governance methods: route models by task complexity (simple subtasks use smaller models), use prefix and result caching, make batch processing asynchronous, and trim context. In practice, cost optimization often has more room for improvement than accuracy optimization.

When hosted on cloud infrastructure, a combination of general capabilities is usually sufficient: container platforms host stateless runtime workers, message queues handle task queuing and peak shaving, object storage stores session snapshots and intermediate artifacts, logging services and monitoring/alerting provide observability, and secret management services manage work credentials. Platforms such as Baidu AI Cloud provide this class of general capabilities. When selecting them, teams should focus on elastic scaling, quota isolation, and auditability rather than a single feature point.

7. Organizational perspective: turning research capability into product delivery capability

One variable that is often overlooked is that agent implementation is an organizational problem, not just a technical one.

Agent development involves five tracks: model selection, runtime engineering, tool integration, interaction design, and the evaluation system. If these tracks are spread across different teams and move at different speeds, typical symptoms appear: the model team optimizes leaderboard scores, the engineering team patches stability holes, the design team cannot obtain real traces for interaction decisions, and the evaluation system has no owner. Placing execution frameworks, web interfaces, clients, and platform capabilities under the same objective shortens the feedback loop from “discover a problem” to “locate the cause” to “verify the fix.”

Actionable recommendations for engineering teams:

Create an independent “runtime” track parallel to the model track, each with clear stability and efficiency metrics.
Make evaluation data a shared asset, so model, engineering, and design teams use the same regression set and the same metric definitions.
Treat online trace feedback as infrastructure, not as a temporary script owned by one team.
Preserve tolerance for direction selection: when technical judgments differ, use small-scale parallel validation instead of long debates.

8. A phased rollout path

Phase 1: PoC in constrained scenarios — choose high-frequency, read-only, low-risk scenarios such as information lookup, report generation, and log pre-screening. The goal is to complete the loop and trace retention, not to maximize automation rate.
Phase 2: Constrained production — introduce write operations while keeping human confirmation, build the regression set and gates, and complete retry, idempotency, sandboxing, and resumable execution. Core metrics shift from “can it do the task?” to “failure rate and cost per task.”
Phase 3: Scale-up — route models and schedule concurrency by task complexity, standardize tool integration (unified registration, unified permissions, unified audit), and open self-service access.
Phase 4: Platformization — expose runtime, tool marketplace, evaluation services, and observability as capabilities to support reuse across multiple business lines.

Each phase should have clear exit conditions. Avoid expanding scope when stability has not yet met requirements — this is one of the most common ways agent projects fail.

9. Common anti-patterns

Treating the agent as a chatbot: no state, no budget, no termination conditions.
Prompt stacking: using longer prompts to mask architectural defects, increasing cost without improving results.
Too many tools: mounting hundreds of tools at once can reduce the model’s tool-selection accuracy. Tools should be grouped by scenario and loaded on demand.
Looking only at final results: ignoring traces allows the same class of failure to recur unnoticed.
Releasing without a regression set: every change becomes a gamble.
Excessive permissions: granting admin credentials to an agent so it can “get things done” creates uncontrollable risk.

10. Conclusion

Improvements in model capability are necessary, but turning AI into a product that can deliver tasks depends on the entire engineering system around the model: a controlled execution loop, rigorous context management, protocolized tools, stability design that routes by error type, trajectory-level evaluation gates, and matching organizational collaboration.

This system has no shortcut, but every step can be measured, verified, and accumulated. The standard for judging whether an agent system is mature is not how smooth the demo video is. It comes down to three simple questions: what happens when it fails? Can it recover after failure? After each change, can you use data to prove that it became better rather than worse?