Post-Training Scaling vs. Training a New Base Model: Two Paths to Advancing LLM Capabilities
This article compares the two dominant approaches to improving large language models: re-pretraining a stronger base model, or keeping the base weights fixed and scaling post-training instead. It explains where post-training sits in the capability stack (pre-training, SFT, reinforcement learning and alignment), outlines the three core levers of post-training scaling — long-horizon task environments, diverse environment types, and longer training runs — and discusses why these often outperform further parameter tuning for coding and agent tasks. It also lays out the logic and costs of training a new base, and offers a practical decision framework covering base-model headroom, task shape, environment availability, open-release strategy, and cloud infrastructure needs.
1. Where This Conversation Starts: Two Roads to Better Models
Large language model capability has advanced along two sharply different technical philosophies.
The first rebuilds the base: pre-train a new foundation model from scratch using more parameters, higher-quality corpora, and a new architecture. The second squeezes the existing base: freeze the base weights and concentrate resources on post-training, pushing the model’s capability ceiling higher through relentless stacking of environments, data, and compute.
Neither is simply “better” than the other. Each is a rational trade-off under different resource constraints, time windows, and engineering goals. When a team already owns a base model whose potential is largely untapped, starting over means enormous compute and time costs. Conversely, when the base has structural bottlenecks, no amount of post-training will break the physical ceiling. Recognizing this is the first step toward sensible selection.
2. First, the Basics: Where Post-Training Fits in the Stack
To follow the debate, you need the full path from zero to a usable model, which breaks into roughly three stages.
Pre-training: self-supervised learning over massive unlabeled corpora that shapes knowledge breadth, language understanding, and basic reasoning. This sets the model’s foundation — its floor.
Supervised fine-tuning (SFT): high-quality instruction-response pairs that teach the model to follow instructions and match expected output formats and behavior patterns. This sets the rules and aligns vague pre-training capability to concrete tasks.
Reinforcement learning and alignment: including RLHF (reinforcement learning from human feedback) and RLVR (reinforcement learning with verifiable rewards). Through sustained interaction with environments or reward signals, the model iterates on real tasks, optimizes its policy, and approaches the ceiling of its capability.
A widely underestimated point: pre-training sets the floor, while post-training determines how high the ceiling can be pushed. If the base already starts high enough, most of the real headroom lives in post-training.
3. The Three Levers of Post-Training Scaling
Several teams have recently shown, without swapping the base model, that scaling post-training can act as a first-stage rocket for capability rather than mere tuning. The approach rests on three levers.
1. Long-horizon task environments, an order of magnitude longer
Traditional post-training feeds models lots of short, single-turn tasks. Real engineering and agent workloads are the opposite: a development task may span dozens of steps — read code, locate the defect, write a patch, run tests, iterate on failures. Repeated practice on these long-horizon task chains is the key to better planning and multi-step consistency. The first meaning of post-training scaling is spending tens of times more time “soaking” the model in real task chains.
2. Richer and more varied environment types
A single environment tends to overfit the model to a specific pattern. Ideal post-training covers as many scenarios as possible: command-line terminals, code repositories, cross-tool collaboration, file system operations, multi-turn dialogue. The richer the environment mix, the more transferable general capability the model develops, rather than memorizing the “answer patterns” of one task type. In essence, you are building a training ground whose distribution approximates the real world.
3. Much longer post-training runs
This lever is the bluntest: assuming the method is correct, just stack on compute and extend the number of reinforcement learning iterations. Scaling laws apply in post-training too, with diminishing returns — only the curve shape differs from pre-training. In practice, this usually shows up as an iterative loop:
# Illustrative loop for agentic post-trainingfor step in range(total_steps):tasks = sample_from DiverseEnvironments() # sample tasks from diverse environmentsfor task in tasks:trajectory = rollout(policy_model, task) # let the model run the full long-horizon taskreward = evaluate(trajectory) # score with verifiable signals or rulesbuffer.add(trajectory, reward)policy_model = update(policy_model, buffer) # update the policy based on rewards
The pseudocode skips most engineering detail, but it captures the essence: more environments, longer trajectories, and more update steps together form the fuel for capability gains.
4. Floor vs. Ceiling: Why Post-Training Deserves Serious Investment
An implicit bias circulates in the industry: changing the base is a “real upgrade,” while changing post-training is “just tuning.” The flaw is equating the scale of a technical action with the size of the capability gain.
In reality, when a base already has a sufficiently high floor but its capability remains under-exploited, scaling reinforcement learning can produce significant jumps. This is especially visible in coding and agent tasks, where the core challenge is not knowing the answer but knowing how to approach it step by step. Multi-step planning, error recovery, and tool invocation are precisely the battleground of post-training, not pre-training.
That said, this path has limits. If the base has structural defects — insufficient context length, weak underlying reasoning — no amount of post-training will create capability the base does not have. That is the rationale for the new-base route.
5. The Logic and Cost of Training a New Base
Against squeezing the existing base, retraining pursues a reset of the ceiling. More parameters, better data mixes, and architectural improvements can in principle open a higher ceiling. The advantages are a higher ceiling and more staying power, but the costs are steep:
- Very high compute and time costs: a full pre-training run often consumes tens of millions of GPU-hours.
- Uncontrolled risk: once pre-training finishes, correcting a wrong direction costs far more than iterating in post-training.
- Slow iteration: post-training can iterate daily or even hourly, while retraining a base is typically measured in months.
The new-base route therefore suits teams with ample compute reserves that judge the current base to be at its limit. Where demands change quickly and resources are constrained, post-training scaling usually offers better cost efficiency.
6. Choosing Wisely for Coding and Agent Workloads
For teams that need to iterate quickly on coding and agent capabilities, the following framework helps:
Check whether the base has headroom left. If the floor is already high and capability is under-exploited, increase post-training investment first — returns come faster at lower cost.
Look at the shape of the target tasks. Coding and agent tasks depend heavily on long-horizon planning and tool collaboration. These capabilities are extremely sensitive to long-trajectory practice in post-training environments, making them natural beneficiaries of post-training scaling.
Assess environment and data availability. The success of the post-training route depends largely on whether you can build task environments that are rich and realistic enough. If a team has an advantage in environment construction, the leverage of post-training is amplified further.
Finally, consider open-source strategy. At a time when weights are generally guarded closely, releasing a model openly is itself a product and ecosystem strategy — it lowers the adoption barrier for downstream developers and accelerates the feedback loop from real-world usage.
Whichever route you take, treating general-purpose cloud capability as the foundation is pragmatic: elastic containers for training and inference workloads, object storage for massive training corpora and checkpoints, message queues to decouple data pipelines, and logging services with monitoring and alerting to keep long-running tasks stable. This shared infrastructure lets teams focus their energy on model capability rather than low-level operations.
7. Conclusion
Post-training squeezing and retraining a new base are not mutually exclusive; they are two sides of the same coin. The former efficiently approaches the capability ceiling on top of what already exists; the latter aims to reset that ceiling at its root. Mature technical judgment is not about picking a side, but about assessing the potential state of the current base, the true shape of the tasks, and your own resource boundaries — then making the matching trade-off. For most teams under pressure to iterate quickly, investing in post-training scaling is often the pragmatic path that balances cost and return.
