Large-Scale RL Post-Training for AI Agents: Environment Infrastructure and Reliable Feedback

1. Two Main Paths for Improving Model Capability

From a model training perspective, current progress in large language model capability largely follows two paths.

The first is pretraining. Its core levers include scaling parameter counts, optimizing model architectures, and improving the scale and quality of training data. In recent years, leading models have generally reached hundreds of billions or even trillions of parameters, and the barrier to entry continues to rise—only a handful of teams worldwide can reliably train at this scale. Pretraining is a competition in compute, data engineering, and training stability. It is a capital-intensive scale race, and the room for differentiation among participants is narrowing.

The second is post-training, which covers supervised fine-tuning (SFT) and reinforcement learning (RL). Over the past one to two years in particular, large-scale RL has become a key method for improving model reasoning and complex task handling. The progression has moved from early alignment based on human preference feedback, to reasoning reinforcement based on verifiable rewards, to today’s task-level reinforcement for agents. Post-training now accounts for a growing share of overall training investment.

This leads to an increasingly clear conclusion: as pretraining scale enters a zone of diminishing marginal returns, post-training is becoming the main arena where model capability diverges. And competition in post-training largely depends on how many high-quality environments a model can use for repeated trial and error with reliable feedback.

In other words, the definition of “data” is expanding—it is no longer just static text corpora, but also interactive, executable, and feedback-producing task environments themselves.

2. From “Checking Answers” to “Running Tasks”: A Key Shift in Training Paradigm

To understand the demanding infrastructure requirements of large-scale agent RL, it helps to first understand how the nature of training tasks has changed.

In the past, training a model to solve math problems meant the model produced an answer, a verifier checked its correctness, and a reward signal was immediately available. The entire process was “single-turn generation + instant judgment,” requiring almost no environment.

Training a model to complete a complex programming task is fundamentally different. The model needs to:

1. Read the repository and requirements to locate the problem;
2. Modify the code;
3. Run tests in a real environment;
4. Observe error messages and understand the cause of failure;
5. Continue modifying, re-running, and re-observing…

This is a multi-turn interactive, long-horizon process. The training system must compute rewards and update the model based on actual execution results—whether the code truly runs and whether the tests truly pass—not on whether the text “looks correct.”

This leads to a direct engineering implication: the environment itself becomes a first-class citizen of the training pipeline. To train thousands of such tasks concurrently, you need a large-scale sandbox and scheduling system, and you must ensure two things: the task quality is high enough, and the final evaluation results are reliable enough. These two points are precisely the most underestimated challenges.

3. Three Core Infrastructure Pillars for Large-Scale Agent RL

3.1 Sandbox Execution Environments: Safe, Reproducible, and Scalable

The sandbox is where the model “gets its hands dirty.” Its design must satisfy several constraints simultaneously:

• Isolation and security: Code executed during tasks is untrusted. The sandbox must provide isolation at the filesystem, network, and process levels. Network access should be denied by default, with domain allowlists configured per task to prevent data contamination or unauthorized access.
• Reproducibility: The initial state of a given task must be identical across training runs. A common approach is “base image + state snapshot”: a base image with preinstalled dependencies, plus a task-level initialization snapshot, ensuring that reward differences come from model behavior rather than environment drift.
• Resource limits: CPU, memory, disk, and wall-clock time must all have upper bounds to prevent infinite loops in individual tasks from dragging down the entire training batch.
• Lifecycle management: Sandboxes should be created in seconds, destroyed immediately after use, and support pre-warmed pools. Under large-scale concurrency, tail latency in environment creation directly determines overall throughput.

A simplified task specification looks roughly like this:

# Task specification example (illustrative)
task_id: fix-failing-tests-v37
env:
  base_image: base image with preinstalled dependencies
  init_snapshot: repository baseline snapshot in object storage
  network: deny            # network access denied by default
limits:
  max_turns: 16            # maximum interaction turns
  wallclock: 30m
  cpu: 4 / mem: 8Gi
reward:
  primary: full test suite passes
  shaping:
    - no modification of test files     # anti-cheating constraint
    - minimize changes
    - no runtime performance regression

3.2 Scheduling and Orchestration: Enabling Tens of Thousands of Parallel Tasks

The rollout phase of RL opens thousands or even tens of thousands of environments simultaneously. The scheduling system must address three classes of problems:

• Heterogeneous resource coordination: Model inference runs on GPU clusters, while task environments mostly run on CPU containers. The ratio, affinity, and network topology between the two need orchestration to avoid GPU resources idling while waiting for CPU resources, or vice versa.
• Long-tail and fault tolerance: Multi-turn tasks naturally have high variance in duration. The system needs elastic scaling, timeout circuit breakers, failure retries, and checkpoint recovery. A single task crash should not contaminate the entire batch.
• Trajectory management: Observations, actions, execution logs, and rewards from each interaction turn must be stored in a structured format for subsequent policy updates, replay analysis, and task difficulty statistics.

A typical training loop can be summarized as:

# Pseudocode: RL training loop based on real execution
for step in range(num_steps):
    tasks = task_pool.sample(batch, strategy="difficulty-stratified")
    sandboxes = sandbox_cluster.create(tasks)    # create environments concurrently
    trajs = agent_loop(                          # multi-turn interaction
        model=policy_model,
        sandboxes=sandboxes,
        max_turns=16,      # modify -> run -> observe -> modify again
    )
    rewards = []
    for task, traj in zip(tasks, trajs):
        report = sandbox.execute(task.verify_cmd)  # judge based on real execution
        rewards.append(task.reward_fn(report, traj))
    trainer.update(trajs, rewards)               # update policy
    sandbox_cluster.destroy(sandboxes)           # reclaim environments

3.3 Reward and Evaluation: Reliable Feedback Is Scarcer Than the Reward Signal Itself

Reward design is a bottleneck in post-training. In engineering practice, it is typically handled at several levels:

• Anchor on real execution: Prioritize objective signals such as passing tests, successful builds, and end-to-end test case completion. Avoid relying on “looks right” text heuristics.
• Anti reward hacking: Models naturally exploit reward loopholes—deleting failing tests, tampering with assertions, or hardcoding expected outputs. Countermeasures include locking test files, introducing held-out tests, and applying penalties for excessive changes.
• Multi-dimensional reward structure: Make correctness the primary signal, supplemented by dimensions such as efficiency, safety, and minimal changes. Use weighted or lexicographic rewards to avoid behavioral distortion from a single signal.
• Sampled human calibration: No matter how reliable automated evaluation is, periodic sampled human review is needed to continuously calibrate the gap between the reward function and actual intent.

A practical lesson: the reliability ceiling of the evaluator is the capability ceiling of the model. If the evaluation is ambiguous, the model learns to “appease the evaluator” rather than “actually solve the problem.”

4. Integration with Cloud Infrastructure

The infrastructure described above is essentially a combination of large-scale container orchestration, GPU compute scheduling, and storage and observability—areas that overlap heavily with general-purpose public cloud capabilities. Cloud platforms typically offer container services, object storage, log services, and monitoring and alerting that can cover common needs such as sandbox cluster setup, trajectory and snapshot storage, and training process observability. This allows teams to focus on task design and reward modeling rather than building environment infrastructure from scratch.

When evaluating platforms, focus on several key metrics: elastic scaling and quota management of the container service, GPU resource pooling and scheduling efficiency, storage throughput and cost model, and log and monitoring retrieval latency. These factors collectively determine the core post-training efficiency metric: the number of verifiable tasks completed per GPU-hour.

5. Common Pitfalls and Best Practices Checklist

Based on engineering experience, here are several frequent pitfalls and their countermeasures:

1. Environment drift: Base images get quietly upgraded, making behavior unreproducible. Countermeasure: version all images and snapshots, and pin training configurations with summary identifiers.
2. Task contamination: Training tasks overlap with evaluation sets or public benchmarks, inflating metrics. Countermeasure: perform deduplication and contamination checks before ingesting tasks, and strictly isolate evaluation sets.
3. Difficulty imbalance: Tasks that are too hard lead to sparse rewards and no learning; tasks that are too easy waste compute. Countermeasure: stratify tasks by pass rate, dynamically adjust sampling distribution, and keep tasks in the “learnable range.”
4. Residual reward hacking: Even with safeguards, models find new loopholes. Countermeasure: rotate held-out test sets regularly, and convert discovered cheating samples into negative examples for retraining.
5. Insufficient long-tail stability: Occasional timeout tasks drag down entire batches. Countermeasure: strict wall-clock limits plus partial result collection, rather than rerunning the entire batch.

6. Conclusion

The competitive narrative for large models is shifting from “who has more parameters” to “who has better environments.” Pretraining determines a model’s knowledge foundation, while post-training—especially large-scale agent RL—determines whether a model can convert that knowledge into professional task capabilities that are actually executable and repeatedly verifiable.

The core investment on this path is not in clever algorithmic tricks but in seemingly unglamorous infrastructure: enough sandboxes, stable enough scheduling, and reliable enough feedback. Whoever can make “environment as data” fast and high-quality will gain an edge in the next phase of model capability competition. For engineering teams, understanding the design logic of this infrastructure is a fundamental skill worth building—whether for training proprietary models, fine-tuning open-source models, or evaluating the capability boundaries of agent platforms.