A Complete Guide to Training Foundation LLMs for the Agent Era
How do you train a foundation model that can actually power AI Agents—not just answer questions? This guide walks through the full training pipeline for Agent-oriented large language models, covering data construction for pretraining and continued pretraining, post-training alignment, reinforcement learning with reward design, data governance, evaluation systems, and engineering practices for reliable, reproducible iteration.
A Complete Guide to Training Foundation LLMs for the Agent Era
1. New Requirements That Agent Scenarios Place on Foundation Models
An AI Agent is not a simple question-answering interface. It is a system that understands goals, decomposes tasks, calls tools, observes results, and continuously corrects its execution across multi-turn interactions. As a result, the training objective for Agent-oriented foundation models must expand from “generating fluent content” to “reliably completing tasks.” This raises several core capability requirements: long-context understanding and state tracking, structured tool calling, task planning and reflection, failure recovery, constraint adherence, and output verifiability.
Traditional pretraining mainly optimizes next-token prediction, while post-training focuses on instruction following and preference alignment. But in Agent scenarios, single-turn Q&A data alone cannot cover the state transitions and tool feedback found in real tasks. Training data needs to include complete trajectories of “goal—plan—action—observation—correction—result,” and evaluation must shift from single-point accuracy to composite metrics such as task success rate, constraint satisfaction rate, tool-call correctness, and execution efficiency.
2. Designing an End-to-End Training Pipeline
A reasonably complete training pipeline can be abstracted into these stages: general pretraining and continued pretraining, supervised fine-tuning, preference alignment, reinforcement learning optimization, evaluation regression, and deployment iteration. These stages are not linear or isolated; they should be connected through a data loop: trajectories generated in production or simulated environments are cleaned, annotated, and validated before entering the training set; the trained model is regression-tested on fixed evaluation sets and dynamic task sets; failed samples flow back into data construction and reward design.
From an engineering perspective, it is advisable to split the pipeline into five trackable objects: data versions, training configurations, model checkpoints, evaluation reports, and release records. Every training experiment should be bound to a specific data version and evaluation version, so that “model improvements” can be properly attributed. For Agent tasks, you should also preserve tool environment versions and simulation environment configurations—otherwise the same model’s performance across different environments may not be comparable.
3. Pretraining and Continued Pretraining: Data Construction Is Critical
The pretraining stage should cover general corpora, code and structured text, math and logic data, tool documentation, task trajectories, and synthetic data. For Agent capabilities, continued pretraining can increase the share of tool descriptions, API schema documentation, state-machine text, error codes and recovery strategies, and multi-turn dialogue trajectories. However, incremental data should not simply be stacked on top; it requires quality filtering, deduplication, contamination removal, sensitive-data handling, and mix-ratio experiments.
For data mixing, a staged curriculum is recommended: early stages emphasize general language ability and foundational knowledge; mid stages add code, structured reasoning, and tool documentation; later stages add multi-turn task trajectories and long-context samples. Long-context extension should be paired with positional encoding adaptation, attention efficiency optimization, and long-document retrieval training—simply increasing sequence length without effective supervision signals is not enough.
A simplified data mix configuration can be expressed as:
stages:- name: general_pretrainmix:web_text: 0.55code_struct: 0.20math_logic: 0.15tool_doc: 0.10- name: agent_continue_pretrainmix:tool_doc: 0.25task_trajectory: 0.30code_struct: 0.20long_context: 0.15general_text: 0.10
Agent trajectory data should follow a unified schema covering, at minimum: task goal, available tools, step-by-step actions, tool observations, intermediate states, final results, and success flags. Synthetic data must pass rule-based validation or cross-validation by models, to avoid amplifying erroneous trajectories into the foundation model.
4. Post-Training Alignment: From Instruction Following to Task Execution
In the supervised fine-tuning stage, the focus should be on constructing high-quality multi-turn task samples. Samples should include not only correct paths but also failure recovery and reflective correction paths; otherwise the model tends to learn a hallucinated style of “succeed in one shot” execution. For tool-call samples, the rationale for tool selection, parameter constraints, call timing, and exception handling strategies should be made explicit. During training, loss masking can be applied to system prompts, user inputs, and tool observations, computing loss only on the planning, tool-call, and summary content the model is expected to generate—this improves supervision efficiency.
The preference alignment stage can combine pairwise preference data, rule-based review, and model-assisted review. Preference dimensions should go beyond language style and cover task correctness, constraint adherence, tool-call soundness, step efficiency, and safety. For tasks that can be formally verified, prefer rule-based signals to construct preference pairs; for open-ended tasks, introduce human or model review, and retain reviewer consistency checks to prevent systematic bias in the preference data.
Safety alignment must run through the entire post-training process: rejecting unsafe tool calls, protecting private data, preventing unauthorized operations, detecting prompt injection, and guarding against malicious task decomposition. In Agent scenarios, safety issues often emerge during multi-step execution, so constraints and evaluations should be set separately for the “planning stage” and the “tool-call stage.”
5. Reinforcement Learning: Reward Design and Preventing Reward Hacking
Reinforcement learning is well suited to optimizing objectives that are hard to express with supervised labels, such as task success rate, execution efficiency, tool-call correctness, and multi-turn recovery capability. Reward functions should be verifiable and decomposable wherever possible, avoiding reward hacking caused by a single model scoring everything. Common designs include: outcome rewards measuring whether the task is completed; process rewards measuring whether key steps are correct; constraint rewards penalizing unauthorized actions, redundant calls, and invalid operations; and efficiency rewards controlling step counts and resource consumption.
Example pseudocode:
def compute_reward(trajectory, task):reward = 0.0if task.verify(trajectory.final_result):reward += 1.0reward += 0.2 * tool_call_precision(trajectory)reward -= 0.1 * redundant_steps(trajectory)reward -= 0.5 * safety_violation(trajectory)reward -= 0.3 * constraint_violation(trajectory)return reward
Training strategies include offline RL, online interaction with simulated environments, and hybrid approaches. Offline data is cheap but carries a high risk of distribution shift; online interaction is closer to real tasks but requires environment isolation, cost control, and failure safeguards. A curriculum learning approach is recommended: start with short-horizon, single-tool tasks and gradually move to long-horizon, multi-tool tasks, with reward caps, anomalous trajectory interception, and manual spot checks in place. During the RL stage, watch especially for the phenomenon where “rewards keep rising but real task success rates do not”—when it appears, roll back the reward design promptly or add independent evaluations.
6. Data Governance, Evaluation Systems, and Engineering Delivery
Data governance should establish version management, annotation guidelines, synthetic data validation, contamination removal, and test set isolation. Agent evaluation should cover four dimensions: capability, safety, cost, and latency. Capability includes multi-turn understanding, planning, tool calling, and failure recovery; safety covers unauthorized actions, privacy, prompt injection, and dangerous operations; cost covers step counts, tool-call counts, and inference overhead; latency covers first-response time and end-to-end task completion time. Evaluation sets should be divided into public regression sets, internal challenge sets, and dynamic task sets, to prevent inflated results caused by training data contamination.
On the engineering side, corpus and checkpoint management, training job orchestration, evaluation pipelines, log collection, and alerting need to be integrated. In practice, general-purpose capabilities such as object storage, container platforms, logging services, and monitoring/alerting (for example, those offered by Baidu AI Cloud) can respectively support corpus and checkpoint management, training and evaluation job orchestration, runtime log collection, and anomaly alerting—improving the observability and reproducibility of the pipeline.
7. Practical Recommendations
First, data quality matters more than data scale—Agent trajectory data in particular must be verifiable and traceable. Second, build a stable evaluation system before scaling up training, to avoid iterating without a baseline. Third, make reward designs decomposable and auditable so the model cannot game them. Fourth, adopt curriculum learning and mixed training strategies to balance general capabilities with task execution ability. Fifth, make safety red-teaming and regression evaluation part of the routine process, with dedicated checks on tool calls, permission boundaries, and long-horizon tasks. Sixth, control training and inference costs through data mix experiments, checkpoint rollback, and small-scale ablations to reduce the cost of trial and error.
Overall, training foundation models for the Agent era is a systems engineering effort: pretraining provides the knowledge and representation base, post-training shapes task execution behavior, reinforcement learning optimizes complex objectives, and data governance and evaluation systems keep iteration under control. Only by co-designing data, training, rewards, evaluation, and the engineering pipeline can you build an Agent foundation model that is stable, scalable, and continuously improvable.
