LLM Agents: From 'Thinking' to 'Executing' in Production
This article explains how the evaluation and engineering of large language model agents are shifting from benchmark scores to execution capability. It covers sparse Mixture-of-Experts architectures, long-context management, native multimodal input, multi-agent orchestration, cost control, observability, and the situations where long-running agents are or are not a good fit.
The Shift: The Evaluation Axis Has Changed
Over the past two years, judging a foundation model was mostly about benchmark scores. But once a model is placed inside an agent framework, the evaluation criteria become three concrete questions:
- Can it extract useful information from screenshots, logs, and error pages?
- Can it still remember the original constraints after dozens of tool calls?
- Can it correct itself after a failed step instead of passing the error down the chain?
This is the difference between “more capable” and “smarter.” The former tests stability in long-chain, highly interactive, side-effect-producing scenarios, while the latter only measures the quality of a single-turn response. A model that produces an excellent single-turn answer may still be unusable in an automated pipeline because it lacks long-horizon consistency, reliable structured output, and failure recovery.
The engineering consequence is straightforward: selection criteria shift from “check the leaderboard” to “run a real task suite.” Teams should build end-to-end tasks that resemble their actual workloads, such as:
- Fixing a failed build
- Reproducing a UI issue from a screenshot
- Extracting structured fields from a batch of documents and writing them into a datastore
The comparison should then rely on four metrics: task completion rate, average number of steps, human intervention rate, and cost per task. General benchmark scores are only useful for initial screening.
Architecture: Engineering Implications of Three Design Choices
Modern native multimodal agent models tend to converge around three architectural directions. Understanding their engineering implications is more valuable than memorizing parameter counts.
Sparsity: Decoupling Total Capacity from Per-Token Compute
The core idea of Mixture-of-Experts, or MoE, architectures is to increase total parameter capacity while keeping the number of parameters activated for each token relatively small. A routing network selects only a few experts for each token.
The benefit is that GPU memory capacity determines how large a model can be, while the activated parameter scale determines how fast a single inference runs. This matters especially for agents: agent token consumption can be several times to tens of times higher than ordinary conversation, so growth in activated parameters can quickly erode the economics.
The tradeoffs are significant:
- High memory footprint
- Communication overhead from expert parallelism
- Load imbalance that can cause long-tail latency
For this reason, deployment usually requires expert parallelism and capacity factor tuning.
Long Context: The Agent’s Working Memory
While executing a task, an agent’s context window holds the system prompt, tool definitions, prior calls and results, intermediate artifacts, and failure logs. Without context management, the window can be flooded by raw logs after only a few dozen steps, causing the model to lose earlier constraints.
A common industry approach is layered context handling:
- Recent interactions are kept in raw form.
- Historical interactions are summarized into structured form.
- Large tool outputs, such as file contents or API responses, are stored as references in object storage and retrieved only when needed.
This resembles a human “notes plus recall” mechanism. It is also a key way to extend effective execution from a dozen steps to hundreds of steps.
Native Multimodal: Visual Information Enters the Main Representation Space
“Native” here means that the output of the visual encoder is aligned with the language model’s representation space and participates in the same attention computation as text tokens. It does not mean first using a separate model to describe the image and then feeding that description as text into the main model.
The practical difference is large. Information such as UI layout, colors, relative positions of controls, and error pop-ups is easy to distort during verbal description. Native multimodal input can preserve these details.
From an engineering perspective, input budget control is important. Visual tokens from high-resolution screenshots and long videos can cost far more than text tokens. Systems must set limits such as:
- Maximum number of images
- Image downscaling
- Frame sampling for video
Otherwise, per-call cost becomes difficult to control.
Architectural Stability Brings Operational Benefits
When a new model version keeps the same architectural form as the previous generation, including expert count, layer count, attention mechanism, and vocabulary size, many deployment components can be reused:
- Inference engine configuration
- Parallelism strategy
- Quantization strategy
- Canary release process
This means teams do not need to re-run load tests and retune the entire stack.
This benefit is often underestimated. Model iteration cycles are often measured in months, while revalidating a deployment stack can take weeks.
# Deployment configuration example: when the architecture is unchanged,# reuse the configuration and replace only weights and metadata.model:dtype: bfloat16parallel: { tensor: 8, expert: 16, pipeline: 1 }kv_cache: { block_size: 16, prefix_sharing: on, max_context: 262144 }serving:max_num_seqs: 256multimodal: { max_images: 4, max_frames: 8, max_pixels: 1280 }router: { timeout_ms: 120000, retries: 2 }
Orchestration: Making a Lead Agent Safely Command Multiple Sub-Agents
The ability to schedule many sub-agents in parallel and extend task chains to thousands of steps is one of the most attractive and also most dangerous capabilities of these models.
Increasing parallelism can reduce wall-clock time, but it also magnifies three things:
- The number of failures
- Cost variability
- The difficulty of converging state
The key design question for the orchestration layer is not “how many agents can be launched?” but “how do we guarantee convergence?”
class AgentRuntime:MAX_STEPS = 400 # Step budget to prevent infinite loopsMAX_PARALLEL = 32 # Maximum number of concurrent subtasksMAX_RETRY = 2 # Retries per subtask, with exponential backoffTOKENS_PER_STEP = 8000 # Context budget per stepdef run(self, goal: str):plan = self.planner(goal) # Decompose into subtasks with dependenciesqueue = plan.subtasks()while queue and self.budget.remaining_steps() > 0:batch = self.scheduler.pick(queue, self.MAX_PARALLEL)for task in batch:res = self.executor.run(task,budget=self.TOKENS_PER_STEP,sandbox=task.resource_scope)if self.verifier.check(task, res): # Success requires verifiable outputself.artifacts.commit(res)queue.remove(task)elif res.retries < self.MAX_RETRY:queue.push_back(task.with_hint(res.error))else:self.escalate(task, res) # Escalate to humans; no silent degradationself.budget.consume_steps(len(batch))return self.assembler.merge(self.artifacts)
Several constraints must be enforced in practice:
- Budgets are safety valves. Step count, token usage, concurrency, and per-task timeouts should all have hard limits. Remaining budgets should be propagated upstream so sub-agents know when to converge.
- Verifiability matters more than explainability. Use test cases, compilation results, field validation rules, and human acceptance checklists as completion criteria instead of letting the model simply claim “I am done.”
- Idempotence and isolation. When sub-agents share a file system or database, each task needs an independent working directory or transaction boundary to avoid concurrent writes contaminating each other. Retries must not create duplicate side effects.
- Aggregate failures instead of spreading them. If a subtask fails, only its local state should be rolled back. The error summary can then become input for the next subtask rather than crashing the whole chain.
- Summary backfill. Before sub-agent output enters the main context, it must be compressed into a structured summary, such as conclusions, changed items, and open questions. Raw long-form output should be stored in object storage with a reference.
Cost: Agent Workloads Have a Different Cost Model
Ordinary conversation costs roughly grow linearly with the number of turns. Agent task costs grow roughly with “steps multiplied by context length per step,” and each step’s context includes all tool outputs. As a result, real cost often scales superlinearly.
There are four main cost-control approaches:
Prefix reuse. System prompts, tool definitions, and codebase context remain stable across steps. Reusing them through paged KV cache and prefix sharing can significantly reduce repeated prefill cost.
Context compression. Summarize and deduplicate historical tool outputs. Keep references for large objects. For code retrieval results, keep only relevant snippets.
Tiered model routing. Send simple subtasks, such as format conversion, keyword extraction, and path generation, to smaller models. Reserve large models for planning, difficult diagnosis, and cross-module refactoring. This can reduce overall cost by an order of magnitude.
Elastic resource scheduling. Run inference services in containerized platforms and scale automatically based on queue length and GPU memory utilization. Separate offline batch workloads from online interactive workloads so long-running tasks do not compete with interactive latency.
Billing rules, free quotas, and rate-limiting policies vary by cloud provider and change frequently. Always refer to the official pricing and documentation of the cloud platform you use. Avoid hardcoding price assumptions in architecture design.
A common practice is to instrument cost tracking and perform capacity planning and alerting based on “cost per task” rather than “price per token.”
Evaluation and Observability: Long-Running Agents Need Monitoring
The failure modes of agent systems are very different from traditional services. Failures are often not crashes, but “polite mistakes.” Observability should therefore cover three layers:
Call-chain tracing. Record each step by task, including input summaries, model version, tool name, latency, token consumption, and result status. This creates a replayable execution trace. When something goes wrong, teams can identify the exact step, tool, or model version that introduced the deviation.
Quality metrics. Track end-to-end task success rate, average completion steps, tool-call argument correctness rate, ineffective retry rate, and human intervention rate. It is advisable to maintain a regression task set of 50 to 200 cases and run it fully whenever the model or prompts change. This prevents fixing one scenario while breaking another.
Security boundaries. Tool permissions should follow least privilege. Write operations and external side effects, such as deleting data, sending messages, or changing production configuration, must go through sandboxing or human confirmation. Model outputs may contain injected instructions and should be filtered. Tool outputs should not be treated as trusted instructions and executed directly.
After integrating these signals into logging and monitoring systems, threshold-based alerts are far more effective than manually searching logs after an incident. Useful alerts include:
- Success rate falling below baseline
- Average step count doubling
- Cost per task exceeding budget
When Long-Running Agents Are Not a Good Fit
Long-running agents are worth considering when:
- Task boundaries are relatively clear
- Results can be automatically verified
- Execution chains are long but decomposable
- Manual step-by-step execution is expensive
Typical examples include:
- Batch bug fixing
- Document extraction and ingestion
- Regression test preparation
- Operations inspection and initial diagnosis
However, they are not ideal for immediate production use when:
- The task is subjective and results cannot be automatically verified
- A single execution has highly destructive or irreversible effects
- Strong compliance requirements demand a complete audit trail for decision-making
In these cases, a semi-automated pattern where “the model suggests and humans decide” is more appropriate. Another option is to start with low-concurrency canary runs and accumulate data first.
Summary
Native multimodality, long context, and sparse MoE architectures are converging and pushing large language models from “conversation tools” toward “execution engines.”
But the gap between model capability and production readiness is filled by five engineering layers: orchestration, budgets, verification, isolation, and monitoring.
What ultimately determines success is often not which model was chosen, but whether the system includes two things:
- Verifiable completion criteria
- Hard resource budgets
The first determines whether the agent can converge on its own. The second determines whether failures can be contained.
A relatively safe three-step approach is:
- Build a task suite before model selection.
- Reuse mature architectures during deployment.
- Launch with canary releases and alerting.
On cloud platforms, including Baidu AI Cloud, general capabilities such as containerized inference services, elastic GPU compute, object storage, message queues, and monitoring alerts can usually support the architectures described above. What teams still need to build themselves are the business-specific task suites and validation rules.
