From Chatting to Doing: Agent Foundation Models and Long-Task Engineering

    The gap between an AI agent that can chat and one that can actually finish the work is what disappoints most teams once the initial hype fades. This article explains why general-purpose conversational models struggle with long-horizon, tool-driven tasks, which capabilities to evaluate in an agent-oriented foundation model, and how to turn model capabilities into a reliable system through five engineering practices: sandboxing, externalized state, context engineering, observability, and evaluation. It includes working code, a table of observability fields, a reference cloud architecture, and a deployment checklist, so you can move an agent from demo to production.

    From Chatting to Doing: Agent Foundation Models and Long-Task Engineering

    1. The Signal: The Gap That Appears Once the Hype Fades

    For a while, “cyber-employee” agents that could autonomously execute tasks drew enormous attention: blanket media coverage, viral demos on social platforms, and a clear bandwagon effect as individual developers and enterprises rushed to deploy them or bought ready-made third-party packages.

    Once the novelty wore off, real-world feedback started to split into three groups:

    • The experience gap: after paying a real deployment cost, the actual interaction felt no different from a general-purpose chat assistant, and some everyday Q&A performed even worse than free lightweight tools.
    • Eroded execution: after attaching a large set of skill plugins, tasks ran slowly, and they often stalled, errored out, or fell into repeated loops midway, requiring frequent manual intervention.
    • Maintenance costs that outweighed the payoff: the time spent supervising and debugging just to get one workflow to run to completion exceeded the time it would have taken to do the task by hand.

    All three point to the same conclusion: “chatting with a model” and “getting a model to do work” are two entirely different capabilities. The former tests language fluency and breadth of knowledge; the latter tests long-horizon decision-making, structured output, and the ability to recover from failures. Judging one system by both standards at once is the source of most of the disappointment.

    2. Root Causes: Why General-Purpose Chat Models Can’t Carry Long-Chain Tasks

    2.1 A mismatch between training objective and execution paradigm

    The core training objective of a general-purpose chat model leans toward “given the context, generate the most likely next span of natural language.” This is a single-step, side-effect-free generation task, and the cost of getting one sentence wrong is very low.

    An agent’s execution paradigm, by contrast, is multi-step, side-effect-heavy, and stateful: observe the environment, decide, call a tool, observe the return, revise the plan, and loop. It demands that the model “take a chain of correct, executable actions under uncertainty,” with an extremely narrow tolerance window. This isn’t a matter of one model being smarter than the other; it’s that they are optimized in different directions.

    2.2 Tool calling is fundamentally a structured-constraint problem

    Tool-calling failures typically look like this: calling an interface name that doesn’t exist, passing an argument of the wrong type (sending a list as a string), omitting a required field, hallucinating a parameter key that doesn’t exist, or ignoring the return value and fabricating a conclusion outright.

    The root cause is that the model samples from a free-text distribution, while the tool protocol demands strict conformance to a schema. Without constrained decoding or structured-output validation, argument errors almost inevitably grow as the number of tools increases; once there are dozens or hundreds of callable tools, simply picking the right one becomes a high-dimensional classification problem.

    2.3 Long-horizon degradation and context pollution

    When a long task fails after running for hours, it usually isn’t that the model “got dumber.” It’s three mechanisms stacking up:

    1. Context bloat: historical reasoning, tool returns, and large blocks of logs keep piling up, and key instructions get diluted in an oversized context.
    2. Error accumulation: one wrong tool return early on gets treated as a factual premise by later steps, so the error amplifies along the chain and becomes less and less self-healing toward the end.
    3. State amnesia: once a task spans days or sessions, the model loses its sense of “which step have I reached,” and either repeats completed steps or skips a critical prerequisite.

    2.4 A lack of engineering guardrails

    Many failures aren’t model problems at all, but engineering defects: no tool-execution timeout or retry, no idempotency design (reruns cause duplicate orders), no permission isolation (the agent deletes data by mistake), and no end-to-end logging (when something breaks, you can’t tell whether the planning, the arguments, or the environment was at fault).

    “Good engineering plus a capable model equals a usable agent.” In this multiplicative relationship, if either factor approaches zero, the result is unusable.

    3. The Capability Side: What an Agent-Oriented Foundation Model Should Have

    A class of models explicitly optimized for agentic workflows has recently emerged. They build on general coding and reasoning ability and add targeted training for long-chain tool use. When evaluating this kind of model, focus on four verifiable capability dimensions rather than a vague notion of “how smart it is.”

    3.1 Stability of tool calls

    Goal: pick the right interface, fill in the right arguments, conform to the format, and parse the result correctly.

    How to evaluate: build a tool pool of 50+ tools, design a task set that requires combining tools across calls, and measure three metrics—tool-selection accuracy, exact-argument accuracy, and schema-conformance rate. Also test the behavior when a tool “returns an error”: a qualified model should read the error message and adjust its arguments before retrying, while an unqualified one repeats the same error or fabricates success.

    3.2 Task planning and decomposition

    Goal: when facing a grand objective, break down the requirements, divide it into phases, identify dependencies, and sensibly interleave sequential and parallel work like a project lead.

    How to evaluate: give a vague goal (for example, “research the technical architecture of three competitors and produce a comparison report”) and observe whether the model proactively seeks clarification, whether it produces a verifiable execution plan, and whether it adjusts the approach mid-way in light of new findings. Strong behavior is plan first, then execute, rather than thinking on the fly and thrashing back and forth.

    3.3 Long-task continuity

    Goal: support batch tasks that run unattended for hours without dropping off, losing memory, or repeating work.

    How to evaluate: design a long task of 200+ steps, inject a few timeouts and network jitters, and check the final completion rate, the number of repeated steps, and adherence to early constraints (for example, whether a rule like “never modify the production configuration” is still remembered late in the run).

    3.4 Multi-agent coordination and parallel scheduling

    Goal: split a task across multiple sub-agents for parallel execution, and handle conflicts and inconsistencies at the aggregation stage.

    How to evaluate: give a set of naturally parallelizable subtasks (modifying several files separately, collecting from several data sources separately) and observe whether execution is genuinely parallel, whether aggregation is complete, and whether conflicts are surfaced explicitly rather than silently overwritten.

    4. The Engineering Side: Five Practices That Turn Model Capability into a Stable System

    Model capability sets the ceiling; engineering guardrails set the floor. These five practices are what carry an agent from demo to production.

    4.1 Sandbox isolation and least privilege

    All tool execution must happen in a restricted environment: containerized runs, a filesystem mounted read-only except for the working directory, a network-egress allowlist, and secrets injected via temporary credentials rather than written into configuration. Destructive operations (delete, commit, publish, pay) must pass through a human-confirmation gate or a two-person review.

    4.2 Externalize state and resume from checkpoints

    Never keep task state only in the context window. After each step, persist structured state to external storage so the process can recover from a checkpoint after a restart or a session interruption:

    1. import json, os, time
    2. STATE_PATH = "runs/{run_id}/state.json"
    3. def load_state(run_id):
    4. path = STATE_PATH.format(run_id=run_id)
    5. if os.path.exists(path):
    6. with open(path) as f:
    7. return json.load(f)
    8. return {"done": [], "plan": [], "budget": {"steps": 0, "tokens": 0}}
    9. def save_state(run_id, state):
    10. tmp = STATE_PATH.format(run_id=run_id) + ".tmp"
    11. os.makedirs(os.path.dirname(tmp), exist_ok=True)
    12. with open(tmp, "w") as f:
    13. json.dump(state, f, ensure_ascii=False, indent=2)
    14. os.replace(tmp, STATE_PATH.format(run_id=run_id)) # atomic write, avoid partial files
    15. def run_agent(run_id, plan, max_attempts=3):
    16. state = load_state(run_id)
    17. for step in plan:
    18. if step.id in state["done"]: # idempotent skip: support resume from checkpoint
    19. continue
    20. for attempt in range(1, max_attempts + 1):
    21. try:
    22. result = execute_with_timeout(step) # must have a timeout
    23. break
    24. except TransientError as e:
    25. if attempt == max_attempts:
    26. state["failed"] = {"step": step.id, "reason": str(e)}
    27. save_state(run_id, state)
    28. raise
    29. time.sleep(2 ** attempt) # exponential backoff
    30. state["done"].append(step.id)
    31. save_state(run_id, state) # persist state each step
    32. return state

    There are three design points: atomic writes (avoid corrupting files when interrupted), idempotency keys (the same subtask produces the same result when repeated), and timeout ceilings (prevent a single stuck step from dragging down an entire overnight run).

    4.3 Context engineering: compression, retrieval, and external memory

    Long tasks can’t be kept alive by “stuffing all the history back in.” A recommended three-tier structure:

    • Short-term workspace: the current step’s inputs and the last tool return, kept within a few thousand tokens.
    • Mid-term task ledger: a structured summary of only “completed subtasks plus key conclusions,” generated by the model or by rules and updated on a rolling basis.
    • Long-term external memory: full logs, intermediate artifacts, and retrievable documents stored in external storage, fetched by reference on demand.

    At the same time, use structured output to keep tool calls within a legal schema and cut format errors at the source:

    1. # pseudocode: constrain tool calls with JSON Schema and validate the return values
    2. tool_schema = {
    3. "name": "search_index",
    4. "parameters": {
    5. "type": "object",
    6. "properties": {
    7. "query": {"type": "string"},
    8. "top_k": {"type": "integer", "minimum": 1, "maximum": 50}
    9. },
    10. "required": ["query"],
    11. "additionalProperties": False
    12. }
    13. }
    14. resp = model.invoke(
    15. messages=window,
    16. tools=[tool_schema],
    17. tool_choice={"type": "auto"}, # allow the model to choose or skip the call
    18. response_format={"type": "json_schema"} # constrained decoding to guarantee schema conformance
    19. )
    20. # defensive validation before use; never trust model output
    21. validate_against_schema(resp.arguments, tool_schema["parameters"])

    4.4 End-to-end observability

    Collect at least the following data for every agent run; otherwise production issues can’t be attributed:

    Layer Key fields Purpose
    Session run_id, goal, start/end time, final status task-level success-rate statistics
    Step step_id, input summary, planning rationale locate planning errors
    Model call prompt/output token counts, latency, model version cost accounting, version rollback
    Tool call tool name, arguments, return code, retry count locate argument and environment errors
    Exception exception type, stack trace, context snapshot at the time retrospective and regression-case accumulation

    Also configure alerting rules: N consecutive steps with no progress, a single task’s token consumption crossing a threshold, or a sudden spike in the tool error rate should all trigger a notification, rather than letting you discover the next morning that the whole night was wasted.

    4.5 Dual-track acceptance: offline evaluation plus online canary

    Don’t accept an agent because it “feels better.” Build a fixed evaluation set (covering four case types: normal paths, boundary inputs, injected tool faults, and privilege-escalation attempts) and run a full regression after every base-model swap or prompt change. Core metrics include:

    • Final task success rate and human-intervention rate
    • Tool-call argument accuracy
    • Average number of steps and token cost
    • Failure self-recovery rate and repeated-step rate

    Roll out with a canary: first run on a small fraction of traffic, compare the new and old version’s metrics, confirm no regression, then switch fully, and guarantee a one-click rollback.

    5. A Reference Architecture and How to Host It in the Cloud

    A typical layering of a production-grade agent system looks like this:

    • Access layer: web console, CLI, and messaging-channel adapters, with unified authentication and rate limiting.
    • Orchestration layer: a task queue plus a scheduler responsible for task decomposition, prioritization, concurrency control, and failed-job re-dispatch; paired with a visual orchestration console that shows the agent topology, real-time status, resource usage, and execution trace, so “which agent is stuck on which step” is visible at a glance.
    • Model layer: a dual channel of a primary model plus a backup model, with automatic failover when the primary errors out or overruns budget.
    • Execution layer: a pool of containerized sandboxes, with tools exposed as independent processes or independent services and permissions granted per task.
    • State and storage layer: object storage for artifacts and snapshots, relational storage for the task ledger, and a message queue to decouple long-running subtasks.
    • Observability layer: a logging service that aggregates end-to-end logs, with monitoring and alerting covering model, tool, and orchestration metrics.

    When deploying in the cloud, these capabilities map directly onto general cloud-product categories: a container platform hosts the sandbox runtime and elastic scaling, a message queue carries asynchronous subtasks and absorbs load spikes, object storage holds large intermediate artifacts and checkpoints, logging and monitoring form the observability base, and key management plus network isolation protect credentials and data. Baidu AI Cloud offers the underlying infrastructure to support such Agent runtimes across general cloud products including containers, message queues, object storage, logging, and monitoring. Teams can build on these general capabilities themselves without depending on any specific proprietary components.

    6. Common Pitfalls and a Deployment Checklist

    Pitfall 1: thinking a stronger model solves everything. When state externalization, idempotency, and timeout/retry are missing, even the strongest model can only execute the wrong thing faster.

    Pitfall 2: piling on dozens of skill plugins. The number of tools correlates positively with the selection-error rate. Trim the tool set per scenario, or add a retrieval-based tool router that exposes only candidate tools to the model.

    Pitfall 3: granting the agent too much privilege. Start read-only and open up gradually; keep a human gate on destructive operations.

    Pitfall 4: tuning prompts frequently without an evaluation set. Without a regression baseline, any “optimization” is a blind change.

    Deployment checklist:

    1. Is the task broken into idempotent substeps, and is each step persisted to disk?
    2. Does each step have a timeout, a retry ceiling, and exponential backoff?
    3. Do destructive operations have permission isolation and human confirmation?
    4. Is there a mechanism for context compression and external memory, and can it recover across days?
    5. Is there end-to-end logging and the three core alert types?
    6. Is there a fixed evaluation set and a canary-rollback plan?
    7. Are the token and tool-call costs of a single task budgetable and circuit-breakable?

    Conclusion

    An agent going from “able to demo” to “able to work” is never the victory of a single point of model capability, but the product of model capability multiplied by engineering guardrails. Understanding the fundamental difference between a general-purpose chat model and the agentic execution paradigm, choosing a base model that has been optimized for long-chain tasks, and then holding it up with sandboxes, externalized state, context engineering, observability, and an evaluation system—that is what lets an agent genuinely take on unattended, all-night long tasks. The hype will recede; what remains are the systems whose engineering details were done solidly.