Post-Training at Scale: How a Controlled Experiment Reveals Where Model Capability Really Comes From
A recent model release sparked industry discussion: a new version of a large language model caught up with a previously leading competitor in third-party benchmarks without changing its base model—relying entirely on one month of post-training and reinforcement learning. This article abstracts the event into general technical questions, examining what Scaling Laws actually say, how much post-training alone can improve a model, why chain-of-thought length matters in evaluation, and what this means for assessing code models.
A recent model release sparked notable industry discussion: a new version of a large language model matched a previously leading competitor in third-party comprehensive benchmarks. Crucially, the improvement did not come from swapping the base model—it was achieved entirely through one month of post-training and reinforcement learning. Around the same time, the head of the model’s research team shared his systematic thinking on Scaling Laws, providing theoretical context for understanding this “stronger without a new base” phenomenon. This article abstracts the event into general technical questions: where does model capability really come from, what are the limits of post-training, and how should code intelligence be evaluated?
Rethinking Scaling Laws: Parameter Count Is Not the Only Variable
Scaling Laws were once the most important empirical rules of the LLM era, but the industry’s understanding of them is undergoing a correction.
The early consensus was “bigger models are stronger.” As a result, many teams scaled up model size while under-feeding training data, and optimized only for training cost while ignoring the long-term inference cost once the model went live. This is a classic single-variable mindset: focusing all attention on parameter count.
In reality, model capability is a multivariable function with at least the following key dimensions:
- Total parameters vs. activated parameters: Dense models and Mixture-of-Experts (MoE) architectures have completely different compute characteristics at the same total parameter count. An MoE model activates only a subset of experts per forward pass, so its activated parameter count is far smaller than its total—leading to very different training and inference cost structures.
- Training data quality and scale: An under-trained large model suffers from “parameter starvation”—its parameters are never fully trained, and its capability falls well short of what its size should deliver.
- Training duration and compute: For the same model, different training compute budgets can produce large capability gaps.
- Long-term inference cost: A deployed model may be called billions of times, and inference cost often far exceeds training cost over its lifecycle. A model designed purely for training efficiency may not be cost-effective in total lifecycle terms.
Therefore, “more parameters means stronger” is only an approximation that holds under the assumption of sufficient data and compute. The real engineering problem is finding a Pareto-optimal solution across these constraints.
A Controlled Post-Training Experiment
The most noteworthy aspect of this release is its experimental design: same base model, same architecture, same total parameters, same activated parameters—the only variable was an additional month of post-training and reinforcement learning.
This is essentially a controlled experiment. In typical model iterations, improvements mix multiple factors—a new base, refreshed data, architectural tweaks—making it hard for outsiders to attribute capability gains to any single cause. This experiment deliberately fixed every condition except post-training, effectively asking a pure scientific question: How much can a model improve through post-training and reinforcement learning alone?
The answer was affirmative. The new version matched the previously leading competitor in third-party comprehensive benchmarks and occupied a position on the Pareto frontier in the most competitive mid-to-high capability tier—meaning that at equivalent cost or capability levels, there was no clearly superior alternative.
The technical implication: the base model determines the “upper-bound range” of capability, while post-training determines how far the model travels within that range. Reinforcement learning is especially critical because it gives the model continuous feedback signals on verifiable tasks (e.g., whether code passes tests, whether a math answer is correct), converting capabilities that “exist latently in the base but were not fully elicited” into stable outputs.
Chain-of-Thought Length: An Overlooked Evaluation Dimension
Beyond benchmark scores, another dimension deserves analysis: the length of the reasoning chain.
A repeatedly observed phenomenon is that some smaller models match larger ones on comprehensive benchmarks by producing significantly longer reasoning chains. The principle can be understood as follows:
A Transformer has a fixed number of layers, so a single forward pass can only perform a bounded amount of computation. But each reasoning step the model generates writes intermediate results back into the context, and subsequent computation builds on that new context. This effectively lets the model “draft multiple rounds,” trading generation time for equivalent computational depth—essentially trading time for space.
Code tasks are especially suited to this compensation mechanism because correctness can be verified repeatedly through execution: the model can write a version, check it, find problems, and fix them. Multiple rounds of self-verification compensate for weaker single-pass reasoning.
However, the model in this experiment behaved differently: in real-world use it did not produce excessively long reasoning chains, and thinking time stayed within normal ranges, yet it still achieved top-tier benchmark results. This means its capability gain does not rely on “thinking longer”—its single-pass reasoning quality is simply higher. This usually points to two things: first, the base model’s underlying capability is more solid; second, post-training compressed the reasoning path to be more efficient—the model learned to take the “correct path” directly instead of relying on exhaustive trial and error.
A Judgment on Code Model Scale
Based on these observations, one can draw a debatable but discussion-worthy conclusion: for code tasks, a model scale in the hundreds of billions of parameters (around 700B) may already be “enough.”
The logic supporting this judgment: once a model reaches a certain scale, the marginal returns of adding parameters diminish, and the remaining capability gap depends more on training-side conditions—particularly the ability to build enough high-quality executable environments for reinforcement learning. Code is one of the few knowledge domains where correctness can be automatically verified. Whoever can mass-produce “problem + executable verification” training environments can keep pushing code capability upward with RL, largely independent of parameter scale.
This judgment has direct implications for engineering practice:
- Don’t evaluate models on benchmark scores alone—also consider the inference cost paid to achieve that score. Two models with identical scores, one with a chain of thought three times longer than the other, will differ enormously in latency and cost in production.
- Focus on the Pareto frontier, not single-point rankings. Capability, speed, and cost form a multidimensional space; only models on the frontier have real deployment value.
- Personal hands-on testing has limitations. On medium-difficulty everyday tasks, adjacent model versions often feel indistinguishable because the task difficulty never approaches the capability ceiling. To differentiate models, you need sufficiently hard, long-horizon real-world tasks.
Conclusion
This experiment—“matching the frontier without changing the base, purely through post-training”—sends an important signal to the industry: competition around Scaling Laws is moving from the first half, defined by “parameter count and pretraining compute,” into a second half defined by “data quality, verifiable environments, and post-training craftsmanship.” For developers who use these models, evaluation frameworks need to evolve as well: beyond scores, inference cost, reasoning-chain efficiency, and task fit must all be considered. The truth about model capability has never lived in a single number—it lives in the multidimensional trade-off among capability, efficiency, and cost.
