LLM Agent Architecture: Scheduling, Execution, and Production Engineering Practices
As LLM products shift from single-turn Q&A toward task execution, the engineering problem changes shape: one user request can trigger dozens of model and tool calls, fail in new ways, and cost far more per task. This article breaks down a practical architecture for building agents that actually converge in production. It covers a two-layer design that separates planning from execution, structured task protocols with explicit budgets, tool allowlists and evidence fields, layered failure recovery, state-machine modeling with checkpointing and context compression for long-horizon work, vertical patterns in code generation, financial and knowledge-intensive research, and embodied systems, plus the inference-infrastructure changes these workloads demand and the governance boundaries that keep them safe.
1. From Chatting to Working: Where Model Capability Is Heading
For the past two years, LLM competition focused on the fluency and knowledge breadth of single-turn answers. More recently, the iteration priorities of many model teams have shifted visibly toward task execution: given a goal, can the model decompose it into steps, call tools, read results, correct its own mistakes, and converge on an outcome?
The technical implication is that a model is no longer an answer generator but an execution agent. Evaluation has moved away from perplexity and accuracy toward three metrics: completion rate on long-horizon tasks, step efficiency, and failure recovery.
Three consequences follow directly for engineering:
- Control flow gets complicated. A single user request can trigger tens of model calls and tool calls, forming a stateful, multi-stage workflow.
- Failure modes multiply. Tool timeouts, hallucinated parameters, context overflow, and step loops can each sink the whole task.
- The cost structure changes. Inference moves from short-in/short-out to long context with repeated read-write cycles, and token and compute consumption per task rises sharply.
2. A Two-Layer Scheduling Architecture: Decoupling Decisions from Execution
One increasingly common engineering answer is to add a scheduling layer outside the agent and an execution layer inside it, producing a two-tier “scheduler model + worker model” structure.
2.1 Division of Responsibilities
| Layer | Core responsibilities | Key constraints |
|---|---|---|
| Scheduling layer (Router/Planner) | Understand the goal, split the task, pick workers, aggregate results, decide whether to retry | Short context, fast decisions, structured output |
| Execution layer (Worker) | Complete concrete operations within a narrow domain: read files, write code, run tests, query data | Long context, narrowed tool permissions, repeatable invocation |
The payoff of this split is context isolation and permission reduction. The scheduler never has to see raw logs; it receives only the worker’s structured conclusions. The worker never has to understand the global goal; it only has to finish one subtask inside a sandbox. Compared with asking a single model to plan and execute at once, the two-tier design cuts prompt length substantially and reduces the missed judgments that come from attention dilution.
2.2 A Structured Protocol Sketch
The interface between scheduler and worker has to be structured — this is the foundation of the whole architecture’s reliability:
# Task dispatched from the scheduling layer to the execution layer# (illustrative structure, not a specific product API)task = {"task_id": "t-017","role": "code_repair","goal": "Fix the failing unit test test_parse_url","context_refs": ["repo://src/parse.py", "log://pytest/latest"],"tools_allowed": ["read_file", "write_file", "run_tests"],"budget": {"max_steps": 12, "max_tokens": 60000},"output_schema": {"status": "enum[fixed|partial|blocked]","patch_summary": "string","evidence": ["string"]}}
Three design points deserve emphasis:
- Explicit budgets.
max_stepsandmax_tokensmust travel with the task, otherwise the worker can loop indefinitely. - Tool allowlists. Workers can only use authorized tools, which prevents accidental file deletion or edits to production configuration.
- Mandatory evidence fields. Requiring the model to output a conclusion plus its basis gives you an anchor for auditing and automated regression later.
2.3 Failure Recovery: Where Agents Divide
The difference between a demo-grade agent and a production-grade one shows up almost entirely in failure handling. Layered fallbacks are the recommended approach:
def run_with_recovery(supervisor, task, max_attempts=3):for attempt in range(1, max_attempts + 1):result = supervisor.dispatch(task)if result.status == "fixed":return resultif result.status == "blocked":# Missing permission or environment: escalate to a human,# retrying here would be pointlessreturn escalate(task, result)# partial: retry with a failure summary and a narrower tasktask = narrow(task, failure=result.evidence[-1])return escalate(task, result)
The key principle: never retry the same error unchanged. If the cause is a missing dependency or insufficient permission, retrying just multiplies cost. The right move is to feed the failure summary back into the prompt so the scheduler can re-plan or escalate to a human.
3. Long-Horizon Tasks: State Management Is the Hard Part
When a task grows from 3 steps to 30, the nature of the problem changes. It stops being a model-capability problem and becomes a state-management problem.
Model the agent’s execution explicitly as a state machine:
STATE_GRAPH = {"understand": {"next": ["plan"]},"plan": {"next": ["act", "ask_user"]},"act": {"next": ["observe", "retry"]},"observe": {"next": ["act", "summarize", "retry"]},"retry": {"next": ["act", "ask_user"]},"summarize": {"next": ["END"]},}
Three engineering practices pair well with it:
- Checkpoint persistence. Write to external storage (object storage or a relational database) after every state transition, so a restarted process can resume from the breakpoint.
- Context compression. Summarize and archive completed steps; put only the summaries plus the most recent N steps of raw text into the context to avoid linear growth.
- Interruptibility and resumability. Long tasks must support human review gates, especially around write operations, external calls, and anything that spends money.
4. Vertical Adoption: Three Directions
4.1 Developer Productivity
Code tasks place the hardest demands on agents, because they come with a verifiable feedback signal: did it compile, are all tests green. The typical loop is: read the bug report and failing logs → locate the relevant files → generate a patch → run the tests → iterate on the results. The engineering focus areas are sandbox isolation (containerized build environments), pinned dependency snapshots (to eliminate “works on my machine”), and minimal, traceable patches.
4.2 Finance and Knowledge-Intensive Research
Here the bottleneck is not generation but traceability. The workable pattern is a four-stage pipeline — retrieve, extract, compute, cite. The retrieval layer recalls sources, the extraction layer structures the data, the compute layer does arithmetic with deterministic code rather than a model, and the citation layer forces attribution. Every numerical conclusion must link back to an original passage; this is a hard requirement for compliance auditing.
4.3 Embodied AI and Physical-World Interaction
When the execution end is a robotic arm or a mobile base, the agent must add a safety layer: reachability and collision pre-checks before motion, speed and torque caps during motion, and immediate degradation to a safe posture on anomalies. The architectural principle here is that the model produces intent only, while control laws and protection logic belong to deterministic programs.
5. The Compute Side: How Inference Load Changes
Task-oriented workloads impose new requirements on inference infrastructure, and they differ markedly from short Q&A:
- Long context and cache reuse. Multiple calls within one task share a large prefix, so prefix caching and request-routing affinity directly affect cost.
- Batching and concurrency. When multiple agents explore in parallel, concurrency rises, which calls for dynamic autoscaling and priority queues.
- Elastic scheduling. Load during debugging versus a traffic peak can differ by orders of magnitude; containerized elastic scaling combined with a message queue to smooth bursts is a common answer.
- Observability. Collect token consumption per step, tool-call latency, failure rate, and human-intervention rate, and aggregate them through log and monitoring/alerting services into cost and quality dashboards.
On the cloud, these compose from a container platform to host inference and execution sandboxes, object storage for checkpoints and artifacts, a message queue for task distribution and backpressure, and logging plus monitoring services for end-to-end tracing. These are generic capability categories; actual specifications need to be checked against whatever you select.
6. Governance and Risk Boundaries
- Least privilege. Grant write and network permissions to the execution layer per task, defaulting to deny.
- Sandbox isolation. Run code execution, browser operations, and terminal commands in disposable containers, destroyed when the task ends.
- Human confirmation gates. Irreversible actions (deleting data, publishing externally, anything money-related) require explicit confirmation.
- Continuous evaluation. Maintain a regression set covering pass rate, average steps, cost per task, and human-intervention rate; it must pass after any model or prompt change.
- Content safety. Treat all external input as untrusted data, and guard against prompt injection arriving via web pages or file contents into the scheduling layer.
7. Implementation Checklist
- Fix the structured task protocol and tool allowlist first, then talk about model selection.
- Manage long-horizon flows with a state machine and persist checkpoints so they are recoverable.
- Use verifiable feedback (compilation, tests, data-consistency checks) as the main signal for convergence.
- Build layered retries and human escalation, so you never burn budget on ineffective retries.
- Instrument the full chain, aggregating tokens, latency, and failure reasons per step to keep improving the cost structure.
- Start with low-risk, high-feedback scenarios such as code repair and data validation, then expand into write-heavy business workflows.
The mark of a mature agent is not that it can demo one task, but that it converges reliably at controlled cost despite real-world noise, timeouts, and permission limits. Two-tier scheduling, stateful workflows, verifiable feedback, and safety boundaries are the four cornerstones that turn capability into dependable productivity.
