Post-Training Scaling vs. New Foundation Models: The Battle for LLM Upper Limits

I. Background: The Competition Enters the “Upper Limit” Phase

Recently, the large model industry has witnessed two types of landmark moves. One group of teams has completed a round of post-training iterations centered on reinforcement learning (RL) on existing foundation models, directly marketing them as “top-ranking among open-source models” and “leading in programming experience.” Another group has chosen a heavier route—building trillion-parameter-scale new foundation models from scratch, similarly focusing on programming and agent capabilities, and likewise opting for open source.

The technical propositions of these two routes are fundamentally different. The former believes that the foundation model’s potential is not exhausted, and that post-training scaling is sufficient to push the upper limit of intelligence a step higher. The latter believes that capability priors must be injected during pre-training, meaning the ceiling itself needs to be rebuilt. This divergence forms the most intriguing cross-section of current LLM technological evolution. Rather than fixating on the success or failure of a single release, this article breaks down both routes, analyzing their premises, benefits, and risks across three levels: technical principles, evaluation paradigms, and engineering costs. Finally, it provides actionable model selection methods for developers and enterprises.

II. Route 1: Keeping the Foundation, Making Post-Training the “Primary Rocket”

2.1 Why Post-Training Scaling Works

To understand this route, we must first clarify what pre-training and post-training actually change.

Pre-training performs statistical compression on massive corpora, determining what the model “knows”—language patterns, world knowledge, and foundational reasoning priors are all formed during this phase. Post-training (instruction tuning, RL, etc.) does not significantly increase the total amount of knowledge; rather, it alters the policy distribution of “how the model calls upon existing knowledge to solve problems.”

An invisible hierarchy has long existed in the industry: modifying the foundation is considered a “true upgrade,” while modifying post-training is dismissed as “tuning.” This judgment holds true if the foundation’s potential has already been exhausted. Conversely, if the capability upper limit of the foundation is far from being reached, compute invested in the post-training phase becomes the most cost-effective primary rocket. A repeatedly validated conclusion is: Pre-training determines the lower bound of a model, while post-training determines how high it can reach.

2.2 Three Key Levers

Based on public technical disclosures, this type of post-training iteration typically revolves around three things:

1. Expanding long-horizon task environments by multiples to tens of times. Traditional supervised fine-tuning data consists mostly of single-turn Q&A and isolated code snippets, which are far from real-world engineering. Long-horizon task environments require models to maintain goals across multi-step interactions, handle errors, and roll back and retry. Every decision step affects the final outcome. Letting models “soak” longer in real engineering task chains is the most direct path to improving agent capabilities.
2. Richer environment types. Covering terminal operations, repository maintenance, cross-tool collaboration, and more. Environment diversity correlates directly with policy generalization. Training in a single environment causes the model to overfit its reward patterns, failing when the scenario changes.
3. Extending training time and piling on compute. RL gains are positively correlated with sampling volume and training steps. This lever sounds brute-force, but it tests comprehensive engineering capabilities in compute scheduling, training stability, and sample efficiency.

2.3 Technical Details: What an RL Environment Looks Like

The core asset of post-training scaling is not compute itself, but the design of environments and rewards. Taking a typical “repo-level bug fix” task as an example, the schematic structure of a terminal agent RL environment is as follows:

# Schematic configuration of a terminal agent RL environment (pseudo-code)
task:
  type: repo-level-fix            # Repository-level bug fix
  init: "git checkout -b task-001"
  goal: "Fix the bug and make all unit tests pass"
  max_steps: 40                   # Maximum interaction steps
  tools: [shell, editor, test_runner]
reward:
  unit_test_pass: 1.0             # Sparse main reward: all tests green
  compile_ok: 0.1                 # Process shaping reward
  regression: -0.5                # Penalty for introducing new failures
sandbox:
  image: "linux-sandbox:stable"   # Fixed image for reproducibility
  network: disabled               # Network isolation to prevent reward hacking

Several design points are worth elaborating on:

• Rewards must be sparse and real. The main reward should be anchored to objective outcomes (test pass rate, command exit codes). Process rewards should only provide light shaping; otherwise, the model will learn to “perform intermediate steps” rather than solve the problem.
• The environment must be reproducible. Use fixed images, fixed random seeds, and network isolation. Otherwise, the RL signal will be polluted by environmental noise.
• Difficulty must be graded. Gradually increase difficulty from single-file fixes to cross-module refactoring, allowing the policy to learn near its capability boundary. This avoids wasting early samples on tasks that are too easy or too hard.

III. Route 2: Rebuilding Trillion-Parameter Foundation Models from Scratch

3.1 The Fundamental Motivation for Changing the Foundation

Post-training can only stimulate potential that “already exists” within the foundation. If a certain capability did not form a sufficient prior during pre-training—such as extremely long-chain cross-file reasoning or specialized knowledge in low-frequency domains—post-training cannot create it out of nothing. This is the fundamental motivation for rebuilding the foundation: not polishing the ceiling to make it brighter, but raising the ceiling itself.

3.2 Four Major Technical Challenges of New Foundation Models

1. Balancing parameter scale and sparsity. Trillion-parameter models typically rely on the Mixture of Experts (MoE) architecture, using routing mechanisms to limit each forward pass to a subset of experts, balancing capacity against inference costs. Routing stability and expert load balancing are major difficulties during training.
2. Data engineering. The ratio, cleaning, and deduplication of high-quality data, along with the introduction strategy for synthetic data, impact final capabilities no less than parameter scale itself—data ratios determine the shape of the priors.
3. Training stability. Fault recovery, gradient anomaly detection, and loss spike mitigation in ultra-large-scale distributed training. Mishandling any of these can reduce weeks of training to zero.
4. Inference costs. A larger total parameter count means higher deployment barriers, requiring techniques like sparse activation, quantization, and speculative decoding to bring single-inference costs down to a usable range.

3.3 Underestimated Hidden Costs

Beyond the explicit compute and time costs, changing the foundation carries a hidden ledger: the post-training pipelines polished around the old foundation—data ratios, reward models, environment difficulty distributions, and various hyperparameters—must all be re-adapted. When a new foundation goes live, it often experiences a running-in period where “capability priors are stronger, but policies are not yet aligned.” This highlights the temporal difference between the two routes: the post-training route shows results in months, while the new foundation route takes quarters to leap forward.

IV. Lower Bounds and Ceilings: Philosophies and Decision Frameworks

4.1 An Analogy: Talent vs. Training Volume

The foundation is like talent; post-training is like training volume. With the same foundation, mediocre post-training only turns it into a “chatty encyclopedia”; extreme post-training can turn it into an “agent capable of independently completing engineering tasks in a terminal.” Conversely, when talent is insufficient, no amount of training volume can do more than approach the limit of existing priors.

4.2 The Alternating Flywheel

The two routes are not mutually exclusive; they form a relay:

Pre-training (Knowledge priors / Determines lower bound)
      │
      ▼
Post-training RL (Policy stimulation / Raises upper bound)
      │
      ▼
Real tasks expose shortcomings (Policy issue or Capability gap?)
      │
      ├── Policy issue → Continue boosting post-training: expand environments, add compute
      │
      └── Capability gap → Feed back into data construction → Launch next foundation iteration

The more likely industry landscape is an alternating ascent: digging deep into post-training after foundation dividends are exhausted, then replacing the foundation after post-training has been pushed to its limits. Historically, the capability leaps of each generation of models are almost always the superposition of these two curves.

4.3 Two Signals for Decision-Making

To decide which route to take currently, observe two signals:

1. The marginal return curve of post-training. If expanding environments and adding compute still yields stable evaluation improvements, the foundation’s potential is unexhausted, and continued investment is rational. If investment doubles but returns plateau, resources should shift to the foundation.
2. The type of failure modes. Analyze the model’s failure cases on target tasks. If errors concentrate on the “policy level” (chaotic steps, inability to use tools, failure to roll back), post-training can fix them. If errors stem from “capability gaps” (fundamental misunderstanding of structures, inability to perform certain reasoning), only new pre-training can solve them.

V. Shifting Competition: From “Answering Questions” to “Working in the Terminal”

5.1 The Three Jumps of Evaluation Paradigms

Over the past two years, the evaluation of programming capabilities has undergone three clear phases:

1. Single-point generation phase: Function-level completion, isolated algorithm problems. Answers are unique and scoring is simple, but this is far from real-world engineering.
2. Repository-level task phase: Requires understanding project structure, cross-file localization, and modification, approaching real-world development.
3. Long-horizon agent phase: Multi-step operations within terminal/CLI environments—actually executing commands, modifying configurations, running tests, and iterating based on errors. Task chains can stretch to dozens of steps.

5.2 Why Terminal Environments Became the Touchstone

Recently, multiple teams releasing new models have unanimously highlighted “agent tasks in terminal environments” as a flagship metric. This is no coincidence:

• High verifiability. Task success can be determined by objective signals like test pass rates and exit codes. Rewards are hard to “game,” giving evaluations high credibility.
• Distribution alignment. A large portion of developers’ daily work happens in the terminal. The evaluation distribution highly matches real-world usage distribution, meaning leaderboard scores better predict actual user experience.
• Amplifying long-horizon shortcomings. Multi-step tasks systematically expose capability deficits in planning, memory, and error recovery—deficits that are easily masked in single-turn Q&A.

5.3 Beware of the “Leaderboard” vs. “User Experience” Disconnect

Maintain a clear head: leaderboard scores and user experience can diverge. Task distributions, prompt templates, and scoring criteria on leaderboards depend on the evaluator’s design. Conversely, claims like “internal experience improved by tens of percent” are neither cross-verifiable nor scenario-agnostic. For decision-makers, the only reliable path is to build an evaluation set using real business tasks and let the data speak for itself.

VI. Custom Evaluation: Turning the Route Debate into Selection Criteria

6.1 Minimum Evaluation Skeleton

Public leaderboards are only for initial screening; final decisions should rest on custom evaluations. Below is a minimal viable evaluation skeleton (generic pseudo-code):

import subprocess, pathlib, dataclasses
@dataclasses.dataclass
class CaseResult:
    case_id: str
    passed: bool
    steps_used: int
    wall_time_s: float
def run_case(case: dict, agent) -> CaseResult:
    """Execute a terminal agent task in an isolated sandbox and score objectively"""
    workspace = pathlib.Path(case["workspace"])
    # 1. Drive the tested model to execute the task in multiple steps within the sandbox
    transcript = agent.solve(
        task=case["instruction"],
        cwd=workspace,
        max_steps=case["max_steps"],
    )
    # 2. Objective scoring: use test pass rates as the main metric to avoid bias from model-as-a-judge
    proc = subprocess.run(
        case["test_cmd"], cwd=workspace,
        shell=True, capture_output=True, text=True,
    )
    return CaseResult(
        case_id=case["id"],
        passed=(proc.returncode == 0),
        steps_used=transcript.steps,
        wall_time_s=transcript.wall_time,
    )

6.2 Five Engineering Takeaways

1. Scoring must be objective. Use tests, assertions, and exit codes to determine success. Avoid the circular bias of “using another model to grade.”
2. Compare efficiency alongside capability. Record steps and time consumed. If one model passes a task in 8 steps and another takes 35, the actual cost difference could be multiples.
3. Samples must come from real business. Sampling 50 to 200 tasks from real tickets and codebases provides far greater discrimination than any public leaderboard.
4. Ensure reproducibility. Fix sandbox images and random seeds, and rerun the full suite every time models are switched to prevent environment drift from polluting conclusions.
5. Decouple evaluation scripts from models. Connect different model services through a unified abstraction layer, ensuring you can horizontally replace or roll back at any time.

VII. Long Context: Capacity Does Not Equal Capability

New-generation models generally push context windows to the million-token level, turning this from a scarce feature into a mainstream configuration. However, engineering practices should guard against two misconceptions:

1. A large window does not mean high utilization. Passing a “needle in a haystack” retrieval test does not mean the model can perform multi-hop reasoning and maintain global consistency over ultra-long contexts. The effective context length is often significantly smaller than the nominal window.
2. Filling the context is not optimal. Indiscriminately dumping an entire repository into the context dilutes key information and drives up costs and latency. In practice, retrieval augmentation, context compression, and task-relevant memory management are usually more effective than mindlessly filling the window. What the context window truly dictates is the upper bound of your engineering solution—how to achieve the most cost-effective combination within that space remains the developer’s homework.

VIII. Open Source: From Technical Choice to Product Strategy

Against a backdrop where top teams are generally tightening access to model weights, insisting on open source constitutes a clear differentiation strategy:

• For enterprises: Private deployment means data does not leave the domain, behavior is auditable, and customization can continue based on proprietary data.
• For the ecosystem: Open weights catalyze downstream practices like fine-tuning, distillation, and quantization. Community feedback, in turn, feeds back into model iteration.
• For the competitive landscape: When capability gaps narrow to the same order of magnitude, the degree of openness and the maturity of the toolchain become key variables influencing selection.

When evaluating open-source models, beyond leaderboard scores, pay attention to license terms, the completeness of weights and tokenizers, and the maturity of accompanying training/inference toolchains. These factors determine whether the model can actually run stably on your own infrastructure.

IX. Practical Selection: A Decision Matrix and Four Recommendations

Models from the two routes differ significantly in cost structure. Use the table below as a reference during selection:

Dimension	Post-Training Scaling Route	New Foundation Route
Core Action	Existing foundation + extreme RL	Rebuild larger-scale pre-training
Time to Results	Fast, monthly iterations	Slow, quarterly leaps
Capability Upper Bound	Limited by foundation priors	Priors themselves are raised
Cost Structure	Compute concentrated in RL phase	Compute concentrated in pre-training
Deployment Barrier	Relatively friendly	Usually higher, relies on sparsity/compression
Main Risks	Diminishing marginal returns	Training instability, ecosystem rebuild needed

Four practical recommendations:

1. Determine weights based on task distribution. If your business relies heavily on terminal automation and repo-level refactoring, assign the highest weight to programming and agent capabilities. If long-document analysis dominates, focus on examining the effective utilization rate of long contexts.
2. Focus on cost curves, not single-point pricing. Differences in steps, token consumption, and retry rates will scale into multiple-fold actual cost differences during batch tasks.
3. Use cloud platforms to quickly build evaluation and inference environments. Cloud platforms like Baidu AI Cloud provide general capabilities including model services, container platforms, logging, and monitoring. They allow you to batch-schedule and execute the aforementioned evaluation skeleton in isolated sandboxes, and continuously track online inference quality and latency via monitoring. This shifts selection from “looking at leaderboards” to “looking at your own data.”
4. Maintain switching capability. Models from both routes are iterating rapidly. Maintaining a model-agnostic abstraction layer in your architecture ensures you can re-evaluate and switch at the lowest cost with every new model release.

X. Conclusion

“Extreme squeezing without changing the foundation” and “rebuilding a larger foundation from scratch” appear to be a route debate, but they are essentially two stages of the same flywheel: pre-training raises knowledge priors, and post-training cashes those priors in as problem-solving policies. Real tasks expose shortcomings, and those shortcomings feed back into the next iteration. This competition has no endgame, and the standard for judging superiority remains singular: in real engineering tasks, who can get the job done with lower cost and higher stability. For developers, rather than chasing the slogans of every release, it is better to build your own evaluation system early. Let every new model release serve as an opportunity for data validation, rather than an occasion to take sides in a war of public opinion.