From Chat Brain to Work Brain: AI Agent Model Selection and Implementation
A common failure mode in AI agents is "process hallucination," where a model claims a task is finished even though the required side effects never actually happened. This article explains why general conversational models behave this way, tracing the problem to training objectives that optimize for response text rather than verifiable outcomes. It then covers reinforcement learning design for agents (environment fidelity, state-based rewards, and reward-hacking prevention), four independent capability dimensions with quantifiable acceptance criteria, and how to build task-level evaluation sets that label expected side effects instead of expected text. It also lays out a layered engineering architecture for running models reliably, along with cost, risk, and common pitfalls, and closes with a phased rollout roadmap. The core takeaway: judge a model not by how well it chats, but by whether it produces the right side effects on your evaluation set.
1. The Failure Scene: A Text-to-Speech Skill That Won’t Install
Adding a text-to-speech (TTS) capability to a general-purpose agent is a beginner-level task. A typical exchange goes like this:
- User: Install a text-to-speech skill for me.
- Model: Done, it’s installed.
- User: Then read something out loud.
- Model: I’ve already generated it for you.
- User: Where’s the file?
- Model: Give me your email and I’ll send it to you.
The working directory contains no audio file, and the skill’s dependencies were never resolved correctly, yet the model drove the whole conversation forward with “already completed” phrasing. These failures share three common symptoms:
- Process hallucination: Narrating “I’m about to do this” as “I’ve already done it,” with no confirmation of side effects.
- Tool idling: Function calls are structurally valid and the arguments look complete, but the target side effect never happens (missing dependencies, out-of-bounds paths, and insufficient permissions are easy to overlook).
- State loss: Once a task spans multiple turns, multiple dispatches, or a long time gap, earlier constraints (output format, naming rules, no external sending) get forgotten.
It isn’t accurate to simply call this “a dumb model.” The real problem is this: this brain was optimized on conversational data, not trained inside a “tool–environment–feedback” loop.
2. Root Cause: The Optimization Target Decides “Chat Brain” vs. “Work Brain”
The training pipeline for general conversational models is roughly “pretraining + instruction fine-tuning + human preference alignment.” The object scored during preference alignment is the response text: human annotators compare two answers and pick the one that is more polite, more helpful, and more human-sounding. Nowhere in this pipeline does any step check “whether the file was actually generated.”
This produces three structural gaps:
- Unverifiable rewards: Sounding right earns a score, while actually getting it done earns nothing extra.
- Different context distributions: Chat is short-range, side-effect-free, and needs no recovery; agent tasks are long-range, produce side effects, and must be rollback-capable.
- Missing failure signals: If signals like tool errors, timeouts, and empty results are never incorporated into training, the model won’t spontaneously form a “read back the result → verify → retry or reroute” behavior chain.
As a result, only models that have been reinforced in a real or high-fidelity environment carry the behavioral priors of a “work brain”: proactively confirming side effects, recovering from errors, and reporting early before exceeding budget.
3. Reinforcement Learning for Agents: Designing the Environment and Reward
The effectiveness of reinforcement learning depends heavily on environment quality. Reusable design points:
- Environment components: An executable sandbox, controllable stand-ins for the file system and network, state that can be snapshotted and rolled back, and full instrumentation of tool calls.
- Rewards based on final state: Replace text similarity with objective verification, for example whether a file exists and is non-empty, command return codes, and the state of external resources.
- Curriculum-style difficulty ramp: Single-step calls → chained multi-step flows → multiple stacked constraints → long-horizon and scheduled tasks.
- Explicitly reward key behaviors: Proactive verification, error recovery, rerouting when blocked, and reporting before the budget is exhausted, rather than rewarding only a “correct final answer.”
- Prevent reward hacking: If a model can earn points by writing a single line that says “done,” it will certainly learn to do exactly that.
def verify(task, env):"""Judge success from the environment's real state, never from the model's own claims."""checks = [env.file_exists(task.expect_path), # Did the artifact land on disk?env.file_non_empty(task.expect_path, min_bytes=64),# Is it more than an empty shell?env.artifact_consumable(task.expect_path), # Can the target capability consume the artifact?not env.has_side_effect(task.forbidden), # Were any forbidden actions triggered?]return 1.0 if all(checks) else 0.0
4. Four Capability Dimensions and Their Acceptance Criteria
| Dimension | Typical Defect | Quantifiable Acceptance Metric |
|---|---|---|
| Tool use | Missing arguments, illegal enum values, ignored error codes, broken multi-tool chaining | Argument validity rate, first-attempt success rate, average retries |
| Complex instruction following | Dropped constraints when several stack up, no clarification of implicit constraints, forced handling of conflicting constraints | Per-constraint pass rate, clarification-question accuracy |
| Scheduled and continuous tasks | Forgetting across cycles, duplicate execution, wrong time-zone handling, no resume after failure | Cross-cycle success rate, duplicate execution rate, resume recovery rate |
| Safety and boundaries | Unauthorized write operations, credential leakage, external content injection | Dangerous-operation block rate, unauthorized-attempt block rate |
These four capabilities are independent of one another, and a single “composite score” must not be allowed to mask a weak area. In engineering practice, it’s recommended to set a separate gate for each dimension: if any dimension falls below its threshold, the model is not allowed to ship.
5. Task-Level Evaluation Sets: More Trustworthy Than General Leaderboards
Public leaderboards measure knowledge and reasoning, but what matters when an agent goes live is “whether the side effects occurred correctly.” It’s recommended to build your own evaluation set whose core is labeling expected side effects rather than expected text.
{"id": "tts-render-001","instruction": "Synthesize this description text into audio, save it to the working directory, and tell me the filename","constraints": ["a common lossless audio format", "filename contains today's date", "must not make any external network requests"],"expect_effects": [{"type": "artifact_exists", "path": "out/*.wav"},{"type": "artifact_consumable", "path": "out/*.wav"}],"forbidden": ["external_send", "network_upload"],"budget": {"max_steps": 8, "max_tokens": 12000, "timeout_sec": 120}}
How to build and use it:
- Sampling: Extract high-frequency tasks from real usage records, then desensitize them and freeze them into test cases.
- Dual-track execution: Verify logical correctness through offline replay, and verify side effects through a real run in an isolated sandbox; cross-check the two.
- Complete metrics: Task success rate, average steps, token consumption, timeout rate, human takeover rate, and dangerous-operation block rate.
- Routine regression: Re-run the full case set every time you swap or upgrade a model, to avoid “leaderboard up, tasks down.”
6. Engineering Implementation: Connecting Model Capability to a Reliable System
The model is just one layer. It’s recommended to decouple the system into five layers: the model layer (routing dialogue, planning, and vectorization by task), the orchestration layer (driving flows with a state machine rather than free-form conversation), the tool layer (contract-based + idempotent), the environment layer (container sandbox + least privilege), and the observability layer (logging, metrics, alerting, and full-link tracing).
def run_task(task, tools, max_retry=2):plan = llm.plan(task) # May only produce structured stepsfor step in plan.steps:for attempt in range(max_retry + 1):r = tools.call(step.name, step.args, idempotency_key=task.id)if r.ok and verify_step(step, r): # Advance only after verification passesbreaklog.warn("tool_failed", step=step.name, attempt=attempt)else:return handoff(task, reason="retry_exhausted") # Escalate to a humanreturn done(task) if verify(task, env) else handoff(task, "verify_failed")
On the cloud, these layers typically map to general product categories: container platforms host the isolated sandboxes, object storage keeps the artifacts, message queues drive scheduled and continuous tasks, logging services and monitoring/alerting provide observability, and key management services handle credentials. Baidu AI Cloud can support this combination through its model services, its AI agent development platform, and the general cloud products above; refer to the official documentation for specific specifications and quotas.
7. Costs, Risks, and Common Pitfalls
- Token cost: Long-horizon tasks must apply context compression, summary-style memory, and tool-result truncation, otherwise cost rises non-linearly with the number of steps.
- Duplicate execution: Every write operation must carry an idempotency key, and scheduled tasks need a deduplication table, to avoid retries causing duplicate external sends.
- Progress hallucination: Any “completed” status shown in the UI must come from a system state query, never from model output.
- Prompt injection: Treat all web page, email, and file content as untrusted data; it must never be executed as instructions.
- Least privilege: A write-operation allowlist + secondary confirmation for dangerous operations + credentials issued at per-task granularity.
8. A Phased Implementation Roadmap
- Weeks 1–2: Pick 3–5 high-frequency, low-risk tasks, keep a human in the loop throughout, and collect real failure samples.
- Weeks 3–4: Introduce sandboxes, idempotency, and full-link tracing, and bring scheduled tasks into the scheduler.
- Month 2: Make the evaluation set routine, roll out the model via gradual (canary) release, and gate promotion by dimension.
- From Month 3 onward: Multi-agent division of labor, cost governance, and capacity planning.
The selection criterion can be simple: don’t look at whether it can chat, only at whether it gets the side effects right on your evaluation set. Only when a model has been repeatedly verified in a real environment does an agent shift from “looking busy” to “actually getting the work done.”
