Scaling vs. Post-Training: How to Evaluate LLM Capability Gains
Benchmark scores alone do not reveal how a model achieved its performance. This article breaks down the two primary paths to improving large language models—pre-training scale-up (parameters, data, compute) and post-training refinement (fine-tuning, reinforcement learning, and long-horizon task practice). It covers the evolution of scaling laws from parameter-first to compute-optimal training, the impact of inference cost on model design, and how Mixture-of-Experts (MoE) architectures decouple total parameters from activated parameters. Practical engineering challenges such as routing imbalance, communication overhead, and training stability are also discussed, providing a framework for evaluating model upgrades beyond surface-level scores.
Why Benchmark Scores Are Not Enough
A single benchmark score only reflects a model’s aggregate performance on a fixed set of tasks—it says nothing about how that result was achieved. Two models can post identical scores through very different means: one relies on a larger base architecture and more pre-training compute, while the other squeezes more out of the same base through post-training investments. The two paths differ substantially in cost, iteration speed, and risk. Before judging whether a performance jump is meaningful, you should first understand how it was produced, then look at the score itself.
There are two main routes to improving LLM capability. The first is pre-training at scale: increasing parameter count, training data, and compute. The second is post-training depth: investing in additional fine-tuning and reinforcement learning on an existing base, often with interactive, long-horizon task environments that let the model practice complete workflows. These routes are not mutually exclusive, but they represent a real trade-off in resource allocation. This article first covers pre-training scale-up, then Mixture-of-Experts (MoE) architecture, then post-training, and closes with a framework for evaluating model choices and benchmarks.
Pre-Training at Scale: From Parameter-First to Compute-Optimal
Early Scaling Laws
Around 2020, researchers observed a power-law relationship between model loss and three factors: parameter count, dataset size, and training compute. The prevailing interpretation at the time was: given a fixed compute budget, prioritize scaling up parameters, and let training data grow at a slower rate. This conclusion guided model design for a period, with substantial resources poured into larger dense models. It was, however, an interpretation based on specific experimental setups—subsequent research revised it.
The Compute-Optimal Correction
In 2022, a different research group re-fit the same class of problems and found that the earlier conclusion undervalued training data. Under a fixed compute budget, parameter count and training tokens should grow roughly in tandem. The study reported an empirical ratio of approximately 20 training tokens per parameter. With equal compute, a smaller model trained on more data can outperform a larger model that was undertrained.
Two clarifications are necessary. First, the 20-token ratio is not a universal optimum—data quality, model architecture, training strategy, and data repetition all shift the ideal balance. Treat it as a starting point for budget estimation rather than a formula to be applied directly. Second, the correction reframes the problem: model size and data volume belong to the same optimization problem; they are not independent levers.
Inference Cost Changes the Objective
Compute-optimal analyses only count the cost of a single training run. For large-scale online services, inference is often the real bottleneck. A model is trained once but may be invoked billions of times, and each invocation’s compute cost scales with its activated parameters.
Let C_train be total training cost, c_inf the per-inference cost, and N_req the total number of inference requests. Total cost is approximately:
C_total = C_train + c_inf × N_req
Here, c_inf can be approximated as proportional to activated parameters per token. This formula deliberately ignores memory bandwidth, storage, scheduling, and networking costs—use it for order-of-magnitude judgment, not precise budgeting.
When N_req is large enough, reducing c_inf yields greater savings than the added one-time training cost. This drives the “smaller model, longer training” approach: trade more training data for lower inference overhead. For teams deploying at scale, this consideration often outweighs raw training efficiency.
Mixture of Experts: Decoupling Total Parameters from Activated Parameters
Mixture-of-Experts (MoE) is central to understanding current model-size competition. The architecture splits the feedforward network into multiple expert sub-networks, and a routing network selects only a few experts per input token.
This design separates two concepts:
- Total parameters: the sum of all expert parameters, determining the model’s capacity ceiling and storage footprint.
- Activated parameters: the parameters actually computed per token, determining per-inference compute cost.
The larger the ratio of total to activated parameters, the more capacity a model can bring per unit of compute. This opens the door to “high capacity, low inference cost.” However, raising this ratio also increases memory pressure, communication overhead, and load-balancing difficulty—trade-offs that must be kept within the tolerance of the routing design and engineering infrastructure. When the ratio approaches 1, nearly all parameters participate in every computation, and the model behaves much like a dense model, offering limited MoE benefit.
MoE also introduces a set of engineering challenges.
- Routing imbalance: Some experts become overloaded while others sit idle. Common remedies include adding an auxiliary load-balancing loss or setting a capacity factor that limits how many tokens each expert processes. A capacity factor set too low drops tokens that exceed capacity, hurting quality; set too high, it wastes compute. During training, monitor each expert’s token share and utilization continuously.
- Memory and communication overhead: When experts are distributed across multiple devices, tokens must be dispatched and aggregated across them, and cross-device data exchange can easily become the bottleneck. The communication pattern of expert parallelism determines cluster interconnect requirements and directly affects scaling efficiency.
- Training stability: Routing can collapse, with most tokens concentrating on a few experts while the rest receive little training. A heavy auxiliary loss weight interferes with the main task; too light a weight fails to suppress imbalance. Common mitigations include tuning the auxiliary loss weight and adding constraints to routing logits. Additionally, when expert partitioning changes, checkpoints require strict conversion and validation—otherwise parameter misalignment may occur during training resumption, and it can be difficult to detect.
