Advanced Post-Training: Engineering Infrastructure for Large-Scale Agent RL

The rapid iteration of Large Language Models (LLMs) may appear as simple version number bumps, but it fundamentally reflects the evolution of training paradigms. Historically, capability leaps relied heavily on pre-training scaling—packing in larger parameters and more data. Today, even without changing the base model, massive investments in post-training can yield significant breakthroughs in coding and complex task execution. What is driving this shift? This article breaks down the technical perspective and focuses on the engineering infrastructure supporting large-scale Agent Reinforcement Learning (RL).

1. Two Main Tracks for Capability Enhancement

From a model training perspective, LLM capability improvements generally follow two tracks:

1) Pre-training: Scaling model size, optimizing architecture, and improving training data scale and quality.

• Parameter Scale: From tens of billions to hundreds of billions and even trillions. The scaling dividend was once the primary engine for capability enhancement.
• Architecture Optimization: Sparsification and Mixture of Experts (MoE) technologies trade for higher effective capacity under the same compute power.
• Data Engineering: Cleaning, deduplication, ratio balancing, and synthetic data push the “quality and quantity of corpora” to their limits.

However, pre-training exhibits clear marginal diminishing returns. High-quality corpora are becoming exhausted, training costs are rising exponentially, and injecting general knowledge does not directly translate into the ability to “do things right.”

2) Post-training: Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and large-scale RL.

• SFT handles instruction following and basic alignment.
• Preference optimization deals with subjective alignment.
• Large-scale RL based on verifiable rewards has become the key method in recent years for improving model reasoning and complex task processing capabilities.

A notable industry trend is that many minor version upgrades (e.g., x.1 releases) do not change the base model. Instead, they significantly increase post-training investments on top of the existing base. This confirms the growing leverage effect of post-training: the base model determines the stock of knowledge, while post-training determines the capability ceiling. The essence is to expand the model’s accumulated capabilities in a specific domain (like coding) into more executable, repeatedly verifiable professional tasks.

2. From Judging Answers to Judging Processes: Why Real Execution Environments are Essential

To understand why post-training relies heavily on infrastructure, we must look at the differences between two task modes.

Math Problem Mode: The model outputs an answer, and the system compares it against a standard answer. The evaluation cost is low, requires no environment, and completes in a single interaction.

Complex Coding Task Mode: You cannot judge correctness solely based on code text—syntactically correct code doesn’t mean logically correct, and logically correct code doesn’t mean it passes real test cases. The model must:

1. Modify code.
2. Run tests in a real environment.
3. Identify issues from error messages.
4. Continue modifying and re-running.
5. Loop until the task is complete.

The training system provides rewards and updates the model based on these actual execution results, often over many rounds. This transforms training from “generating text” to “acting in an environment”—namely, Agent Reinforcement Learning.

This mode has a core requirement for reward signals: verifiability. Verifiable rewards offer three major advantages:

• Objective and Reliable: Whether tests pass or compilations succeed are objective facts, independent of subjective scoring.
• Anti-cheating: Compared to preference scoring, which is easily fooled by rhetoric, rewards based on execution results are much harder to game.
• Scalable: The evaluation process is fully automated, requiring no human intervention, thus supporting concurrent training across massive tasks.

3. Four Core Infrastructures for Large-Scale Agent RL

To simultaneously train tens of thousands of such tasks, four layers of engineering problems must be solved.

1) Large-Scale Sandbox Clusters
Each task requires an independent, isolated, and rapidly deployable/destroyable execution environment:

• Isolation: Tasks must not interfere with each other. Errors or malicious code cannot escape to affect the host or other tasks.
• Resource Quotas: CPU, memory, disk, and network must be capped to prevent a single task from crashing the entire cluster.
• Environment Consistency: Standardized images and snapshot mechanisms ensure the same code yields the same execution results in any sandbox—reproducibility is the prerequisite for trustworthy rewards.
• Second-level Creation/Destruction: Training throughput is directly constrained by sandbox turnaround speed.

2) High-Throughput Scheduling System

• Concurrently orchestrates thousands of execution environments, handling queuing, prioritization, preemption, and failure retries.
• Decouples “rollout” (sampling execution) from “gradient update” (training): execution happens in the sandbox cluster, updates happen in the training cluster, and the two are asynchronous and pipelined.
• Handles reward aggregation and training batch alignment.

3) High-Quality Task Sets
Tasks themselves serve as the training “curriculum”:

• Diverse Distribution: Covers different domains (programming, data analysis, ops scripts, office automation, etc.) and difficulty levels.
• Anti-leakage Isolation: Training tasks and evaluation tasks must be strictly isolated to prevent the model from “memorizing answers” and inflating its perceived capability.
• Difficulty Calibration: Tasks that are too easy provide no gradient signal because everyone passes; tasks that are too hard provide no signal because no one passes. Ideal tasks sit on the boundary of the model’s capabilities—“sometimes right, sometimes wrong.”

4) Reliable Evaluation System

• Automated Testing as Reward Source: Unit tests, integration tests, end-to-end assertions.
• Multi-verification Anti-cheating: A single signal is easily exploited; multiple verification dimensions must be combined.
• Combining Outcome and Process Rewards: Looking only at the final result yields sparse signals and slow convergence. Introducing process scoring (e.g., partial points for each passed test case) can accelerate convergence, but one must guard against rewards being gradually “arbitraged.”

4. Pseudocode: What a Training Episode Looks Like

The following illustrative pseudocode explains the flow and does not correspond to any specific product API.

def run_agent_episode(task, policy, sandbox_pool):
    # 1. Acquire an isolated sandbox matching the task
    sandbox = sandbox_pool.acquire(
        image=task.env_image,                       # Environment image matching the task
        limits={"cpu": 2, "mem": "4G", "net": "isolated"},
    )
    trajectory = []
    try:
        state = task.initialize(sandbox)            # Initialize the task environment
        for step in range(task.max_steps):         # Multi-turn interaction loop
            obs = state.get_observation()          # Observe the environment
            action = policy.generate(obs)          # Model decision (e.g., generate/modify code)
            result = sandbox.execute(action)       # Actual execution
            state.apply(result)
            trajectory.append((obs, action, result))
            if state.is_complete():
                break
        reward = task.verify(state)                # Verifiable reward: run tests, check assertions
        return trajectory, reward
    finally:
        sandbox.snapshot_and_release()             # Save snapshot and release resources

Concurrent sampling and updates on the scheduling side:

async def training_round(task_batch, policy, trainer):
    episodes = await asyncio.gather(*[             # Massive concurrent task execution
        run_agent_episode(t, policy, sandbox_pool)
        for t in task_batch
    ])
    advantages = compute_advantages(episodes)      # Intra-group relative comparison and normalization
    await trainer.update(policy, episodes, advantages)

These two snippets illustrate the composition of training throughput:

Training Throughput ≈ Concurrent Tasks × Average Steps per Task ÷ Average Execution Time per Step

Sandbox turnaround, scheduling efficiency, and execution speed—any bottleneck in these areas will slow down the overall training pace. This explains why the infrastructure requirements for such work are far higher than those for standard online services.

Observability is equally essential:

• Unified Log Collection and Retrieval: Standard outputs and errors for every step within the sandbox must be logged, leveraging general logging service capabilities.
• Metrics Monitoring: Task pass rates, reward distributions, episode length distributions, sandbox utilization, and failure cause classification rely on standard monitoring and alerting capabilities.
• Trajectory Archiving: Saving training trajectories to object storage for failure analysis, anomaly auditing, and subsequent data flywheeling.

From an infrastructure selection perspective, such training platforms are typically built on general-purpose cloud computing capabilities. Container platforms handle sandbox orchestration and isolation; object storage manages images, datasets, and trajectory archiving; message queues bridge the instruction flow between the training and execution sides; logging and monitoring services ensure end-to-end observability. For instance, Baidu AI Cloud’s general-purpose capabilities—such as its container platform, object storage, message queue, and logging services—can serve as foundational options when building this type of training infrastructure, combined as needed without reinventing the wheel.

5. The Essence of Post-Training Competition: Environment × Feedback

This leads to a core conclusion: the competition in post-training will increasingly depend on how many high-quality environments a model can repeatedly trial-and-error in to obtain reliable feedback.

This can be approximated by a multiplication formula:

Training Return ≈ Task Quality × Environment Fidelity × Feedback Reliability × Execution Scale

If any of these four factors approaches zero, the overall return approaches zero:

• Poor task quality (watered-down tasks, evaluation set leakage): The more you train, the further you deviate.
• Environment distortion (large gap between sandbox and real production environment): Learned capabilities fail to transfer.
• Unreliable feedback (flawed tests, exploitable rewards): The model learns to “game the test” rather than “do the work.”
• Insufficient execution scale (low concurrency, low throughput): Trial-and-error attempts are limited, and policy updates are slow.

This is fundamentally different from the competitive logic of the pre-training era. Pre-training competes on compute and corpora; post-training competes on environments and feedback. Building high-quality, executable task environments is much harder than scraping web corpora—it requires domain knowledge, engineering investment, and long-term maintenance. But precisely because it is hard, it constitutes a truly hard-to-replicate differentiation.

Furthermore, the execution environment itself spawns a data flywheel: the model executes tasks and generates massive trajectories; high-quality trajectories are filtered and returned to training; as model capabilities improve, it tackles harder tasks, generating more valuable new trajectories. The richer the environment, the faster the flywheel spins.

6. Conclusion: Pre-training Sets the Floor, Post-training Sets the Ceiling

Looking back at this technological evolution, it can be summarized in three points:

1. Pre-training injects knowledge and general capabilities, setting the floor for the model. Post-training uses large-scale RL to translate knowledge into executable professional capabilities, setting the ceiling for the model.
2. The success of large-scale Agent RL does not depend on a single-point algorithm, but rather on the composite completeness of the infrastructure: Sandbox Clusters × Scheduling Systems × Task Quality × Evaluation Reliability.
3. Implications for developers: Even if you are not training foundation models, this methodology applies to upper-layer applications. Building an executable, verifiable evaluation environment for your agents is essentially doing “application-layer post-training.” Those who can turn their business environments into training grounds where models can repeatedly trial-and-error and receive reliable feedback will gain a head start in implementation efficiency.