LLM Scaling Law Revisited: Pretraining vs. Post-Training Scaling and MoE Capacity
Two similarly sized open-source LLMs recently landed at the same score on a widely cited comprehensive benchmark by taking completely different paths: one pushed pretraining to the trillion-parameter, many-expert extreme, while the other left its base model untouched and gained its ground in post-training. This article revisits how scaling laws have evolved—from the roughly 2.7:1 parameter-to-data ratio to the compute-optimal ratio of about 20 tokens per parameter—and explains how inference cost shifts the optimum toward smaller models trained for longer. It then breaks down the three levers of post-training scaling (environment count, environment diversity, and RL scale), clarifies how MoE models separate capacity (total parameters) from reasoning bandwidth (activated parameters and effective depth), and offers practical guidance on where to invest, why leaderboard ties can hide very different paths, and when smaller-models-longer-training wins on total cost of ownership.
Introduction: Same Score, Different Origins
A recently released open-source LLM matched another open-source model of comparable size on a widely cited comprehensive benchmark, and even pushed into the score range occupied by closed-source flagship models. What is worth studying here is not the “who won” leaderboard game, but the fact that the two models reached comparable capability by fundamentally different routes: one poured its resources into the pretraining stage to push parameter count higher, while the other barely touched its base model and instead squeezed out the latent capability through post-training alone. This contrast puts the most important question in today’s LLM competition squarely on the table: when we invest to grow capability, which knob should we turn?
1. Same Performance Level, Two Technical Routes
Taken apart, the divergence between the two models is clear.
Route A is the “brute force” approach: trillions of total parameters, composed of hundreds or even thousands of experts, with only a small subset activated for each token at inference time. This is the classic pretraining-heavy strategy—a bigger base, more experts, more compute, trading scale for capability.
Route B is entirely different: total parameters are exactly the same as the previous generation, and nothing changed in the architecture or parameter scale. The only change happened in the post-training stage. The team spent roughly a month substantially expanding the number and types of long-horizon task environments, and scaled reinforcement learning up by an order of magnitude, which finally produced a jump in benchmark scores.
In one sentence: one route is doing “addition” (making the base bigger), the other is doing “deep mining” (using the existing base to its fullest). The fact that the two routes can arrive at nearly the same result is itself evidence that post-training has become a capability lever that can no longer be ignored.
2. The Evolution of Scaling Laws: More Than One Knob
To understand why Route B works, we need to go back to the history of scaling laws.
A classic study from around 2020 produced an empirical rule: the growth rate of parameters should outpace the growth rate of data, at roughly a 2.7:1 ratio. The whole industry followed it—generation after generation of super-large models were built on essentially this recipe.
A 2022 follow-up study corrected that bias, proposing that the “compute-optimal” training mix is roughly 20 tokens per parameter. Models should not simply grow mindlessly; parameters and data need to scale together at a better ratio.
But here is the turning point in the story: even if you achieve “compute-optimal”, what it optimizes is only the static cost of “train once, evaluate once”. In the real world, a model is called billions of times per day, and inference cost is the dominant share over the long run. Once inference overhead is counted into the total bill, the optimum is no longer “big and heavy”—it shifts toward “smaller models, longer training”: fewer parameters, more data fed in, cheaper per-token inference, and a lower total cost of ownership.
The core conclusion follows: scaling is never a single knob, but a dynamic balance determined jointly by training cost, inference cost, and task type.
3. Post-Training Scaling: Extracting More from an Existing Base
The essence of Route B is extending “scaling” from pretraining into post-training.
Pretraining determines what the model “knows”—the foundation of knowledge, facts, and language ability; post-training determines what the model “can do”—whether it can reliably organize its capability when facing complex, multi-step, long-horizon real tasks. When the base model is already strong enough, piling more parameters into pretraining yields diminishing marginal returns, whereas post-training—a resource long underestimated—can deliver steep returns.
The three key variables of post-training scaling:
- Environment scale: The more usable training tasks/environments there are, the more generalizable the policies the model learns, instead of memorizing specific problem types.
- Environment diversity: Returns from scaling up a single environment will saturate; what really creates separation is coverage of environment types—combinations of tasks with different lengths, different tools, and different constraints.
- RL scale: More thorough RL training lets the model learn long-horizon planning and self-correction in a “try–feedback–adjust” loop.
Schematically, post-training resources can be organized as follows (pseudocode):
# The three levers of post-training scaling are not independent of each otherdef post_training_budget(base_model, months=1):env_pool = build_task_environments(count="tens of times the previous generation",types=["long-horizon planning", "multi-tool collaboration", "constraint solving", ...],)curriculum = sample_curriculum(env_pool) # schedule by difficulty and coveragepolicy = rl_finetune(base_model, # base frozen, parameter scale unchangedenvironments=curriculum,steps="larger scale",)return evaluate(policy)
Note that the base model’s parameters do not change “at all” at this step; what changes is how rich the world fed to it is, and how thoroughly it is trained.
4. A Key Insight Under MoE: A Division of Labor Between Capacity and Inference
In a Mixture of Experts (MoE) architecture, parameters need to be viewed as two distinct categories with entirely different responsibilities:
- Total parameters: determine the model’s “capacity”—how much knowledge, how many facts, and how much long-tail information it can hold; closer to the model’s “memory”.
- Activated parameters & effective depth: determine how far the model “can think”—how many steps of a causal chain it can carry consecutively; closer to the model’s “thinking bandwidth”.
A counterintuitive but important conclusion: at a fixed tokens-per-parameter (TPP) ratio, pushing total parameters even higher actually hurts reasoning ability, while reliably activating more experts consistently helps reasoning.
With an extremely simple compute estimate, you can get a feel for how “capacity” and “inference overhead” come apart:
def moe_active_params(total_params, num_experts, top_k):# The elegance of MoE: capacity grows with the total number of experts,# but per-token compute depends only on the activated top_k expertsper_expert = total_params // num_expertsactivated = per_expert * top_kreturn activated # inference cost ∝ activated, not total# Example: fixed total params, finer-grained experts with different top-kp_total = 743_000_000_000activated_few = moe_active_params(p_total, num_experts=64, top_k=8)activated_more = moe_active_params(p_total, num_experts=256, top_k=16)print(activated_few, activated_more)
What this shows: if you want to push reasoning ability further, instead of blindly expanding total parameters (which only adds capacity and memory pressure), it is better to deepen the “effective compute depth” through finer-grained experts with more activations. This also explains why two models with vastly different total parameter counts can still tie on a comprehensive reasoning benchmark—they allocate their budgets differently between these two accounts, “capacity” and “reasoning”.
5. Implications for Engineering Practice and Investment
Turning the analysis above into concrete decisions yields several actionable takeaways:
- Don’t fixate on the overall leaderboard score. The same score can come from completely different technical paths; understanding the path is more valuable than memorizing the ranking.
- Post-training is an underestimated lever. When base capability is already sufficient, post-training scaling (more and richer long-horizon task environments plus larger-scale RL) can often buy a significant jump at a lower cost.
- Separate “capacity needs” from “reasoning needs”. If the business bottleneck is long-tail knowledge, lean toward higher total parameters and better data quality; if the bottleneck is multi-step reasoning and long-horizon planning, focus on activated parameters, effective depth, and how thoroughly post-training is carried out.
- Put inference cost into the total bill. When evaluating the merits of a route, training cost alone is not enough; you must estimate total cost of ownership against call volume—“smaller models, longer training” is often more cost-effective in high-concurrency scenarios.
- Environment engineering is a core competency. The ceiling of a post-training route depends, to a large extent, on how large and how diverse a set of task environments you can build.
Conclusion
The same score can be the result of being “bigger”, or the result of using things “better”. The history of scaling laws reminds us again and again that there is no single optimal knob—only the optimal tradeoff for a given cost structure and task distribution. Pretraining determines the model’s knowledge foundation, post-training determines how it converts that knowledge into capability, and the MoE architecture tells us that capacity and reasoning are two accounts that need to be calculated separately. Understanding these three threads is what lets you see the genuinely valuable technical signals amid the flood of “matched” and “surpassed” headlines.
