1. Two Main Trajectories for Enhancing LLM Capabilities

It is widely recognized in the industry that the continuous improvement of Large Language Model (LLM) capabilities primarily follows two paths:

The first is pre-training. Core methods include scaling parameter sizes, optimizing model architectures (such as attention mechanism improvements and Mixture of Experts structures), and increasing the volume and cleaning quality of training data. Over the past few years, parameter sizes have climbed from tens of billions to hundreds of billions, with top-tier models now entering the hundreds of billions to trillions range. Pre-training determines a model’s “knowledge base” and its general capability ceiling. However, its marginal returns are gradually diminishing—the cost of acquiring compute power and high-quality text data is becoming increasingly prohibitive, making the gains from simply piling on scale less cost-effective.

The second is post-training. Building upon the pre-trained foundation, this leverages Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and the rapidly emerging large-scale reinforcement learning of the past two years to “unlock” the foundation model’s potential. The role of post-training has become especially critical in scenarios requiring deep thinking, such as mathematical reasoning, code generation, and complex task planning.

A notable industry trend is that as marginal returns from pre-training decline, more teams are shifting their resource focus to post-training, particularly large-scale reinforcement learning tailored for Agent scenarios.

2. Technological Evolution of Post-Training: From Imitation to Exploration

Post-training is not a single technology, but a continuously evolving set of paradigms:

1. Supervised Fine-Tuning (SFT): Uses high-quality instruction-response pairs for behavioral cloning, teaching the model to “follow examples.” It is stable and controllable, but its capability ceiling is limited by the quality and coverage of the demonstration data, and it struggles to teach the model how to “trial and error.”
2. Preference Alignment (RLHF/DPO, etc.): Optimizes output style and safety based on human preference data. It primarily addresses whether “responses meet expectations,” but its contribution to complex reasoning capabilities is relatively limited.
3. Reinforcement Learning with Verifiable Rewards (RLVR): In domains with objective grading criteria like math and coding, the model improves itself through extensive sampling and trial and error. The system provides rewards and updates parameters based on standard answers or test results.
4. Agent Reinforcement Learning: Expands RL from “single-turn Q&A” to “multi-turn, long-horizon, tool-calling” task sequences. The model must execute operations in real or simulated environments, observe feedback, revise strategies, and ultimately complete the task objectives.

The first two categories address “alignment,” while the latter two solve for “capability leaps.” The fourth category is currently garnering the most attention.

3. From “Judging Right or Wrong” to “Execution Capability”: The Essential Shift in Reward Signals

To understand why Agent RL demands extremely high infrastructure requirements, one must first recognize the shift in reward signals.

When training a model to solve math problems, reward acquisition is relatively straightforward: the model outputs a final answer, which is compared against the standard answer. The entire interaction is single-turn, static, and low-cost.

However, teaching a model to complete a complex programming task is entirely different:

• The model needs to understand requirements and break down implementation steps;
• It needs to write or modify code, call build tools, and run tests;
• It needs to observe error messages, locate defects, and continue modifying;
• This process may loop for multiple rounds until all tests pass.

At this point, the reward no longer comes from “answer comparison,” but from the results of real execution—whether compilation passes, whether tests pass, and whether program behavior meets expectations. The training system must calculate rewards and update the model based on this actual execution feedback.

This leads to two direct consequences: First, the environment becomes a necessity. Without a real execution environment, there are no reliable rewards, and reinforcement learning degrades into fitting superficial patterns. Second, interaction costs rise sharply. A single trajectory might contain dozens of tool calls and tens of thousands of tokens in context. Thus, a clear conclusion emerges: the competition in post-training is increasingly determined by how many high-quality environments a model can repeatedly trial-and-error in, and how reliable the feedback it receives is. “Environment quality and quantity” is becoming the new moat, with a status comparable to “data quality” in the pre-training era.

4. Four Major Infrastructure Challenges of Large-Scale Agent RL

Scaling the above closed-loop to an industrial scale—training thousands of tasks simultaneously—causes engineering complexity to rise steeply.

1. Sandbox Environments: Isolation, Security, and High Concurrency

When an agent executes code, the system must actually run arbitrary code generated by the model, which inherently carries security risks. Mature solutions typically build sandboxes based on containers or lightweight virtual machines: strong isolation (restricting file system, network, and system call permissions), snapshot and second-level recovery to support massive task parallelism, resource quotas (CPU/memory/time limits), and environment reproducibility (consistent execution results for the same task at different times to avoid introducing noise into reward signals).

2. Task Scheduling: Keeping Expensive GPUs Busy

The training loop is an alternating process of “sample—execute—evaluate—update,” and the variance in environment execution time is extremely high. Common optimizations include: asynchronous pipelines (decoupling and parallelizing sampling, execution, reward calculation, and parameter updates), long-tail task management (timeout termination and retry strategies), and elastic scaling (scaling sandbox instances up and down according to the peaks and troughs of the training rhythm).

3. Task and Environment Quality: The New “Data Engineering”

While the pre-training era focused on data cleaning, the Agent RL era focuses on “environment engineering”: task difficulty should cover a continuous range to avoid wasting compute on tasks that are too easy or too hard; test cases must be rigorously designed to prevent models from “gaming the system” for rewards; evaluation tasks must be de-contaminated to prevent leakage into pre-training corpora causing distorted assessments; and task types must be continuously expanded from code repair to executable and verifiable professional scenarios like data analysis and IT operations.

4. Evaluation Reliability: Reward Functions Are the Primary Target of Attack

As long as rewards can be gamed, models will inevitably find shortcuts, known as reward hacking. Countermeasures include: multi-dimensional evaluation (checking the rationality of intermediate products, not just the final result), hidden test sets (using different test cases for training and evaluation to suppress overfitting), and combining manual spot-checks with automated audits to continuously patch reward vulnerabilities.

5. A Representative Training Closed-Loop

Stringing the above elements together, a typical Agent RL training loop can be described with the following pseudo-code (for illustrative purposes only):

# Large-scale Agent RL training closed-loop (illustrative)
for step in range(train_steps):
    tasks = task_pool.sample(batch_size)          # 1. Sample tasks from the pool, stratified by difficulty
    envs  = [Sandbox.spawn(t) for t in tasks]     # 2. Spawn an isolated sandbox for each task
    trajectories = []
    for task, env in zip(tasks, envs):
        traj = agent.rollout(env, max_rounds=N)   # 3. Multi-turn interaction: edit code -> run -> observe -> re-edit
        reward = verifier.score(env.result,       # 4. Calculate reward based on real execution results
                                hidden_tests=task.tests)
        trajectories.append((traj, reward))
    policy.update(trajectories)                   # 5. Update the policy model
    telemetry.report(pass_rate, reward_dist)      # 6. Monitor pass rate and reward distribution

What truly determines the training ceiling is often not the algorithm itself, but the engineering system behind steps 2 to 4: whether the sandbox is stable enough, whether the scheduling is fast enough, and whether the reward is accurate enough.

6. General Technology Foundation for Engineering Implementation

From an engineering perspective, building this type of training platform typically reuses general cloud-native capabilities: container orchestration services host sandbox clusters, GPU compute scheduling supports training and inference, object storage saves massive trajectories and task data, and logging services with monitoring and alerting ensure the observability of long-term training. These capabilities are standard foundational services in mainstream cloud platforms (such as Baidu AI Cloud). Teams can combine them on demand and focus their efforts on genuinely differentiating aspects like task pool construction and reward system design.

7. Challenges and Trends

Despite rapid progress, large-scale Agent RL still has several open issues: high environment construction costs, where connecting real tools and bridging the sim-to-real gap require massive investment; generalization difficulties, where a model might perform excellently in training environments but degrade significantly in unfamiliar ones; immature evaluation systems, as interactive and executable evaluation environments are themselves scarce resources; and the token consumption and interaction overhead brought by long trajectories, which impose higher requirements on inference efficiency.

It is foreseeable that the focus of industry competition will gradually shift from “who has the larger foundation model” to “who can build more high-quality environments and whose feedback authority is more reliable.”

Conclusion

Post-training is redefining the growth curve of LLM capabilities: pre-training packs “knowledge” into the model, while post-training teaches the model to “apply it” through large-scale reinforcement learning. The success of this trajectory ultimately depends on the less glamorous infrastructure—a stable sandbox, an efficient scheduler, and a rigorous reward function. For teams planning to invest in this direction, it is advisable to start with a small-scale closed-loop to verify the reliability of the environment and rewards before gradually scaling up concurrency. After all, in reinforcement learning, the quality of feedback always takes priority over the quantity.