Scaling Laws Revisited: Should You Grow Models in Pretraining or Unlock Potential in Post-training?

    When two open-source large language models post nearly identical benchmark scores, they may have followed entirely different technical paths. This article examines the two mainstream approaches—scaling parameters during pretraining versus unlocking potential during post-training—through the evolution of scaling laws. It covers how scaling law corrections have shifted the industry's view of model size, how parameter allocation works in Mixture-of-Experts (MoE) architectures, and how engineering teams should evaluate both routes and allocate resources, with practical Python examples for estimating training token budgets and sampling plans.

    1. Background

    Recent shifts in the leaderboard rankings of open-source large language models have sparked considerable discussion. Within the same period, two models with similar scores may have followed completely different technical routes—and that difference is more worth examining than the rankings themselves. Understanding these route differences helps teams decide where to invest resources in the next phase.

    Rather than discussing specific models’ scores, this article focuses on how scaling laws have evolved, and explains two mainstream approaches: one that scales up model size during pretraining, and another that mines the potential of an existing base model during post-training.

    2. Two Technical Routes

    2.1 Scaling Parameters During Pretraining

    The core assumption of pretraining-scale expansion is that increasing parameter count, training data, and compute leads to steady improvements in model capability. This route typically requires larger training clusters, longer training cycles, and more robust data cleaning pipelines. Its advantage is a high capability ceiling and broad knowledge coverage; the trade-offs are high training costs, long iteration cycles, and inference costs that rise in step with parameter count.

    2.2 Unlocking Potential Through Post-training

    Post-training refers to further optimization after pretraining completes, using supervised fine-tuning, reinforcement learning, and similar techniques. Without changing the model’s architecture, teams can improve specific capabilities by increasing long-horizon task samples, enriching task environments, and scaling up reinforcement learning. Its advantages are fast iteration and controllable costs; the limitation is that capability gains are bounded by the base model’s existing knowledge and capacity—post-training cannot fill in capabilities the base model lacks.

    2.3 How the Two Routes Relate

    The two routes are not mutually exclusive. The base model determines the ceiling of capability, while post-training determines how much of that capability gets released on specific tasks. Engineering teams need to weigh the investment ratio between the two based on their target scenarios. Treating either route as the single correct answer is not advisable.

    3. Three Revisions to Scaling Laws

    3.1 Early Findings: Parameter Growth First

    Around 2020, researchers found experimentally that under a fixed compute budget, increasing parameter count reduced loss more effectively than increasing data volume. The industry formed a consensus around “build bigger models first,” and many early large models followed this ratio.

    3.2 Rebalancing Data and Parameters

    In 2022, another line of research showed that the earlier parameter-to-data ratio was not optimal. Under the same compute budget, training data volume should grow in step with parameter count. Empirically, roughly 20 training tokens per parameter yields good results. This finding changed how the industry allocated training budgets and made “smaller models trained more thoroughly” a competitive option.

    3.3 Factoring in Inference Costs

    The revisions above considered only training costs. In real deployments, a model may be called billions of times, and inference costs often exceed training costs. Once inference is factored in, the optimal strategy shifts further toward “smaller models trained longer.” The smaller the model, the cheaper each inference call—and the same budget buys more training data.

    Note that these ratios are empirical conclusions. The specific values vary with data quality, architecture, and task distribution, and should not be applied as fixed formulas.

    4. Parameter Allocation in Mixture-of-Experts Architectures

    A Mixture-of-Experts (MoE) architecture splits a model into multiple expert sub-networks, activating only a subset during each forward pass. This structure allows total model capacity and per-inference compute to be designed separately.

    4.1 Total Parameters Determine Knowledge Capacity

    Total parameter count determines how much information the model can store, including factual and long-tail knowledge. More total parameters mean broader memory coverage, but each inference does not use all of them.

    4.2 Active Parameters Affect Reasoning Depth

    Active parameters and effective network depth determine how long a reasoning chain the model can handle in a single inference. With total parameters fixed, increasing the number of experts activated per pass typically improves multi-step reasoning. Conversely, raising total parameters without increasing activation may not improve reasoning at all, and the added routing complexity can even hurt performance.

    4.3 A Simple Calculation Example

    The following code demonstrates how to compute the activation ratio and the empirical training token count for roughly comparing different configurations. The example values are for illustration only:

    1. def active_ratio(total_params: float, active_params: float) -> float:
    2. """Return the ratio of active parameters to total parameters."""
    3. if total_params <= 0:
    4. raise ValueError("Total parameter count must be positive")
    5. if not 0 < active_params <= total_params:
    6. raise ValueError("Active parameters must be greater than 0 and no more than total parameters")
    7. return active_params / total_params
    8. def compute_optimal_tokens(params: float, tokens_per_param: float = 20.0) -> float:
    9. """Estimate the number of training tokens based on the empirical ratio."""
    10. return params * tokens_per_param
    11. if __name__ == "__main__":
    12. total = 100e9 # Example value: total parameter count
    13. active = 10e9 # Example value: parameters activated per inference
    14. print(f"Activation ratio: {active_ratio(total, active):.2%}")
    15. print(f"Estimated training tokens at 20:1: {compute_optimal_tokens(active):.2e}")

    This code is a conceptual estimate only and does not represent the actual configuration of any specific model. Real-world selection also needs to account for routing efficiency, communication overhead, and memory footprint.

    5. How to Evaluate the Two Routes

    A single leaderboard score rarely captures a model’s full capability. Evaluations should focus on the following aspects:

    • Task distribution: Whether the benchmark’s tasks align with your business scenarios.
    • Cost metrics: Training cost, per-inference cost, and response latency.
    • Stability: How much the same capability fluctuates across different input distributions.
    • Long-horizon performance: Completion rates on multi-step tasks, not just single-turn Q&A accuracy.

    Teams can build internal evaluation sets, break them down by business task dimensions, and compare results side by side with cost data. Relying on rankings alone easily overlooks cost and fit.

    6. Engineering Recommendations

    6.1 Resource Planning

    If your team mainly uses off-the-shelf base models, focus resources on post-training data construction and evaluation systems. If your team needs to build its own base model, you must trade off pretraining scale, data quality, and inference costs—and estimate total serving costs at your expected deployment scale in advance.

    6.2 Building Post-training Environments

    Post-training effectiveness depends heavily on the quality of the training environment. Recommended practices include:

    • Covering multiple task types to avoid capability imbalance caused by homogeneous samples.
    • Adding long-horizon tasks to test the model’s ability to maintain state across multi-step processes.
    • Designing automatically verifiable reward signals to reduce reliance on manual annotation.
    • Gradually scaling up reinforcement learning while monitoring the risk of overfitting to specific environments.

    6.3 Example: Managing Environment Sampling Ratios

    1. from dataclasses import dataclass
    2. @dataclass
    3. class EnvironmentSpec:
    4. name: str
    5. weight: float # Sampling weight
    6. max_steps: int # Maximum steps per task
    7. def sample_plan(envs: list[EnvironmentSpec], total_tasks: int) -> dict[str, int]:
    8. """Allocate the number of tasks per environment based on weights."""
    9. total_weight = sum(e.weight for e in envs)
    10. if total_weight <= 0:
    11. raise ValueError("The sum of weights must be positive")
    12. return {e.name: round(total_tasks * e.weight / total_weight) for e in envs}

    This example shows how to allocate a task budget across environments by weight. Production systems also need to adjust weights dynamically based on each environment’s difficulty and feedback quality.

    7. Conclusion

    The difference between the two routes is fundamentally a question of where to invest resources. Pretraining expansion raises the ceiling of what a model can know; post-training determines how much of that potential is realized on real tasks. Teams that understand how scaling laws have evolved—and how parameters, data, and inference costs interact—will be better positioned to decide where their next batch of compute and engineering effort should go.