Native Agent LLMs: Architecture, Training, and Engineering Deployment Explained
This article breaks down what "native agents" are and how they differ from the bolt-on agent pattern, then walks through the architecture, training, cost, and engineering decisions behind them. It covers why mixture-of-experts (MoE) models fit agent workloads, how thinking and non-thinking modes trade latency against cost, a three-stage training recipe with verifiable reinforcement learning rewards, how to reason about total task cost instead of per-token price, how to integrate with coding toolchains through standard protocols, how to build evaluation sets that beat public leaderboards, and the production guardrails that keep agents safe and observable. Engineering teams will come away with a concrete checklist for allocating inference budgets, standardizing tool interfaces, controlling cost, and choosing a provider.
1. From Bolt-On Agents to Native Agents
The mainstream early path for building agents (Agents) was a “general-purpose chat model plus an external orchestration framework”: the model only generated text, tool descriptions were stitched into the prompt by the framework, and loop control, result trimming, and failure retries were all handled by the framework. This approach worked, but in long-horizon tasks it exposed three structural problems:
- Capabilities were forced together through prompting: the model had never seen the target distribution of “multi-step tool calling” during training, so format drift, fabricated parameters, and premature endings happened frequently.
- Context bloat: every tool result was fed back in full, and after dozens of steps the context was consumed by logs and file contents, diluting key instructions.
- Uncontrollable latency and cost: the same reasoning length was executed regardless of task difficulty, so even a simple rewrite paid the price of complex reasoning.
“Native agents” means jointly optimizing three capabilities—reasoning, code generation, and tool calling—within the same set of weights and training objectives, rather than stitching them together with outer-layer logic. It has three observable characteristics:
- Unified mode control, where the model can switch between “deep reasoning” and “instant response” through the same interface;
- Stable structured output, with a significantly higher rate of tool-call parameters conforming to the given schema;
- Better completion rates and average step counts on long-horizon, multi-step tasks, rather than just smoother single-turn Q&A.
Once this is clear, the architecture, training, and cost discussions that follow share a common frame of reference.
2. Sparse Architecture and Inference Budget: Why MoE Fits Agent Scenarios
The core of a Mixture-of-Experts (MoE) model is to decouple total parameter count from per-token activated parameters: each layer contains multiple expert networks, and a router selects a few experts to compute each token. As a result, a model can have very large knowledge capacity while the compute per forward pass depends only on the activated parameters.
This matters especially for agent scenarios. An Agent’s input is naturally “heavy”—tool return values, code files, error stack traces, and history trajectories, with a single request often reaching tens of thousands of tokens. Here:
- Activated parameters determine decode throughput (tokens/second), which directly affects end-to-end wait time for multi-step tasks;
- Total parameters determine knowledge breadth, affecting cross-file understanding and the accuracy of obscure framework APIs;
- Engineering still requires supporting infrastructure—load-balancing auxiliary loss, expert parallelism, and capacity-factor control—otherwise expert skew leads to long-tail latency.
On this foundation, a flagship version and a lightweight version form a gradient: the large model handles complex refactoring and cross-module defect localization, while the lightweight model handles high-concurrency, low-cost scenarios such as completion, unit-test generation, and log categorization. Working alongside this is a dual-mode mechanism:
| Dimension | Thinking mode | Non-thinking mode |
|---|---|---|
| Suitable tasks | Complex reasoning, multi-step tool orchestration, difficult debugging | Completion, rewriting, formatting, intent classification |
| Latency profile | Continuous output of reasoning content after the first token | Answer from the first token, fast response |
| Cost profile | Output tokens include the reasoning process, higher unit cost | Short output, low per-request cost |
| Controllability | Requires limiting max reasoning length and steps | Close to a traditional API experience |
Mode selection should be moved upstream to the business side rather than patched after the fact:
class ModeRouter:"""Select an inference budget by task complexity to avoid always using deep reasoning."""def select(self, task) -> str:if task.requires_multi_file_edit or task.tool_steps_hint > 5:return "think" # deep reasoning + tool callsif task.intent in {"completion", "reformat", "classify"}:return "no_think" # instant responsereturn "think" if task.risk_level == "high" else "no_think"
3. The Training Recipe: Three-Stage Progression and RL Reward Design
Native agent capabilities cannot come from a single stage; the industry generally adopts a three-stage recipe:
1. Pretraining stage
Build foundational capabilities on corpora on the order of trillions of tokens. The key is the mix rather than sheer volume: reasonable proportions of natural language, code, math, and multilingual data, along with deduplication, quality scoring, and contamination detection (removing evaluation-set leakage). A common pitfall is code corpus granularity being too coarse; repository-level dependency relationships must be preserved.
2. Mid-training stage (the watershed for agent capabilities)
- Long-context extension: adjust positional encoding and the training window so the model can still recall instructions in the middle of a very long context;
- Repository-level code understanding: organize samples as “file + dependency graph + retrieval snippets” to train cross-file localization;
- Tool-trajectory corpora: construct complete trajectories of “instruction → tool selection → parameter generation → observed result → next step,” exposing the model to both success and failure samples.
3. Post-training stage
First, use rejection-sampling distillation of high-quality reasoning trajectories for supervised fine-tuning, then use reinforcement learning with verifiable rewards to improve reasoning and tool use. Reward design is central; for code agents it can be:
def reward(trajectory, sandbox) -> float:"""Verifiable reward: objective signals replace manual preference labels."""r = 0.0r += 1.0 * sandbox.unit_tests_passed_ratio() # unit test pass rater -= 0.2 * sandbox.build_errors() # build/lint errorsr += 0.3 * trajectory.task_completed() # whether the task was completedr -= 0.05 * trajectory.tool_call_steps() # step penalty to discourage verbosityr -= 0.5 * trajectory.unsafe_operation() # high-risk operation penaltyreturn r
A self-developed training framework is almost unavoidable here: contention between reasoning-trajectory rollout and parameter updates for compute, long-sequence parallelism, asynchronous off-policy training, and executors for thousands of concurrent sandboxes—none of these are default capabilities of a general-purpose training stack.
4. Inference Economics: Doing the Token Math
Budget overruns in agent projects stem nine times out of ten from comparing only “price per million tokens” while ignoring total task cost:
Per-task cost = Σ(input tokens × input price + output tokens × output price)End-to-end cost = per-task cost × retry factor × average daily task volume
Here the retry factor is determined by tool-call failure rate and context-overflow rate, and is often the largest hidden term. Actionable cost guardrails include:
- Prompt caching: put invariant prefixes such as the system prompt, tool schemas, and code conventions at the front to improve prefix-cache hit rate;
- Tool result trimming: feed back only the N lines around an error in logs, and truncate search results by relevance;
- Retrieval instead of full context: use vector or keyword retrieval to fetch relevant snippets instead of stuffing an entire directory into the context;
- Model-tier routing: route simple tasks down to the lightweight version or non-thinking mode;
- Step and budget caps: hard limits on max tool-call rounds and max reasoning length per task, degrading or escalating to a human when exceeded.
def estimate(inputs: dict) -> dict:"""Illustrative cost estimate; unit prices follow actual service pricing."""think, fast = inputs["think"], inputs["fast"]cost = lambda m, it, ot: (m["in_tok"] * it + m["out_tok"] * ot) / 1_000_000return {"think_task": round(cost(think, think["in_price"], think["out_price"]), 4),"fast_task": round(cost(fast, fast["in_price"], fast["out_price"]), 4),"monthly": round(inputs["daily_tasks"] * 30 * cost(think, 2.0, 8.0), 2),}
Throughput also belongs in the cost picture: only when output speed reaches the order of hundreds of tokens per second can a thirty-step agent task converge within minutes, which is when engineers are willing to use it; otherwise even a low unit price is offset by the labor cost of human waiting.
5. Integrating with Coding Toolchains: Standardized Protocols First
The engineering value of a new-generation agent depends largely on whether it can seamlessly integrate with existing CLI-style coding assistants. The key is to keep interfaces standardized:
- Support the industry-standard function-calling protocol (
tools,tool_choice, JSON Schema constraints), avoiding separate adaptation for each toolchain; - Support interleaved streaming output and tool calls, so users can see diffs as generation proceeds;
- Use patch-based file writes (unified diff / line-range replacement) to reduce the context and conflict costs of whole-file rewrites;
- Automatic context compaction: when the window is exceeded, summarize earlier trajectories, keeping decision conclusions and discarding lengthy observations.
The way tool declarations are written directly determines the call success rate; strong constraints and low ambiguity are recommended:
{"name": "run_tests","description": "Run the specified tests in an isolated sandbox and return failing cases with stack traces","parameters": {"type": "object","properties": {"target": {"type": "string", "description": "Test file or case path, relative to the repository root"},"timeout_sec": {"type": "integer", "minimum": 5, "maximum": 600, "default": 120}},"required": ["target"],"additionalProperties": false}}
Because the weights are released under a permissive open-source license, commercial secondary development is feasible. For private deployment, tuning should focus on: quantization (FP8/INT4) in mainstream inference engines, expert-parallelism degree, KV-cache reuse, and prefix-cache hit rate—these usually deliver more direct returns than switching models.
6. Evaluation: A Self-Built Metric Set Beats Leaderboards
Public leaderboards are good for initial screening, not for procurement decisions. A three-layer self-built evaluation is recommended:
- Code capability layer: test patch correctness on real tasks in internal repositories (compilable + unit tests passing), along with average rounds of change;
- Tool-use layer: parameter-extraction accuracy, multi-step task completion rate, error-recovery rate, and average step count;
- Reliability layer: hallucination rate (proportion citing nonexistent APIs/files), instruction-following rate, and high-risk-operation trigger rate.
Statistical definitions must be fixed: use pass@k to measure the generation ceiling, success rate to measure engineering usability, and average cost and P95 latency to measure operational feasibility. Running the same task set once before and once after a model upgrade is the only way to tell whether the gains are real increments or the result of prompt tuning.
7. Engineering Guardrails for Production
- Sandboxing and least privilege: isolated container execution, no outbound network access by default, read-only credential mounts, and write operations confined to the workspace;
- Second confirmation for high-risk actions: deletion, deployment, dependency upgrades, and production config changes must require explicit confirmation or an approval flow;
- Full-chain observability: log each run’s tool trajectory, token accounting, mode selection, and failure reasons, archive trajectories to object storage, and connect metrics to logging and monitoring/alerting;
- Elasticity and degradation: run inference services on a container platform and scale by queue length, degrading to a lightweight model or non-thinking mode when overloaded rather than rejecting outright;
- Data security: redact keys and personal information before they enter the context, and grant authorization to code snippets by repository sensitivity level.
When deploying a solution like this on Baidu AI Cloud, the common combination is: the model service platform provides inference and tool-calling interfaces, the container platform hosts sandboxes and orchestration services, object storage archives trajectories and artifacts, and logging plus monitoring/alerting handle accounting and cost governance. Each component falls under general cloud-service categories and can be swapped independently according to business scale.
8. Selection Decision Checklist
| Focus | Questions to confirm with the provider |
|---|---|
| Architecture | Total/activated parameters, context window, whether long-context middle recall is supported |
| Modes | Whether thinking mode can be turned off, whether reasoning length can be capped, whether mode switching affects caching |
| Tool calling | Compatibility with the general function-calling protocol, parallel calls, strict schema mode |
| Deployment | Open-source license and commercial terms, private-deployment resource needs, quantization support |
| Metering | Input/output/cached token billing breakdown, rate-limiting and quota policies |
| Evaluation | Whether it can provide comparison results on your own task set |
Conclusion: The competitiveness of native agents comes not from a single benchmark score, but from the fit among “architecture—training—cost—engineering.” What engineering teams truly need to master is: allocate inference budgets by task complexity, standardize tool interfaces, make decisions on total task cost rather than unit price, and safeguard controllability with self-built evaluation sets and sandbox guardrails. Achieve these four points, and switching the underlying model becomes a configuration change rather than an architectural rewrite.
