Trillion-Parameter LLMs: The Compute and Data Bottlenecks of Scaling Up
As large language models push toward trillion-parameter scales, understanding the relationship between parameters, data, and compute becomes critical. This article explores why parameter scale remains a key variable for model capability, distinguishes between total and activated parameters in Mixture of Experts (MoE) architectures, and examines the three primary bottlenecks of scaling: compute, data, and engineering. It also introduces preliminary coping strategies like knowledge distillation for scenarios where compute resources are limited.
1. Why Parameter Scale Remains a Critical Variable for Capability
In large language model (LLM) research, parameter scale has long been considered a key factor influencing model capability. The industry consensus stems from extensive experiments on scaling laws: under the premise of similar model architectures, when training data volume, parameter count, and compute power scale simultaneously, the loss on language modeling tasks typically shows a smooth downward trend.
Two prerequisites of this law require attention. First, returns are not linear; every doubling in scale typically yields diminishing improvements. Second, the parameter count must match the training data volume. If parameters are numerous but data is insufficient, the model is prone to undertraining, and the value of the parameters cannot be fully realized.
Therefore, looking solely at the total parameter count is incomplete when assessing a model’s capability ceiling. A more reasonable approach is to concurrently monitor three metrics: total parameters, actual activated parameters per inference, and the effective volume of training data.
1.1 Total Parameters vs. Activated Parameters
The Mixture of Experts (MoE) architecture is a common design today. The model contains numerous expert sub-networks, but each input token is routed to only a few of them. The total parameter count can be massive, while a single forward pass activates only a fraction of them.
This implies that “total parameters” and “inference cost” are not simply directly proportional. When reading model scale information, it is necessary to verify which type of parameter the reported figure refers to.
2. The Threefold Constraints of Scaling Up
2.1 Compute Constraints
The total compute required to train a model can be roughly estimated as:
Total FLOPs ≈ 6 × N × D
Where N is the parameter count and D is the number of training tokens. The coefficient 6 comes from common estimations of forward and backward propagation. Assuming N is 1 trillion (1e12) and D is 10 trillion (1e13), the total compute is approximately 6e25 FLOPs. As parameter scale and data volume must be scaled up together, the compute demand grows with the product of both.
Compute is not just about the number of chips; it also includes interconnect bandwidth, cluster stability, and training cycles. In a cluster composed of thousands of accelerator cards, a single card failure can interrupt training. Therefore, checkpoint saving and fault recovery mechanisms are essential engineering components.
2.2 Data Constraints
The volume of high-quality text is limited. The growth rate of high-quality corpora available in the public web for training can hardly keep pace with the expansion of parameter scale. This leads to several practical issues:
- Data deduplication: Duplicate samples amplify the model’s memorization of specific content, weakening generalization capabilities.
- Data cleaning: Low-quality, noisy text reduces training efficiency.
- Data mixture: The ratio of code, math, multilingual, and conversational data affects the model’s performance across different tasks.
- Synthetic data: Data generated by existing models can supplement scarce domains, but erroneous content must be filtered out, otherwise, the model may degrade.
Investment in data engineering typically increases in tandem with model scale; it is insufficient to only look at the parameter count.
2.3 Engineering Constraints
When calculating only the weights, each parameter stored in 16-bit floating-point format occupies 2 bytes. The weights for 1 trillion parameters are roughly 2TB. This does not include optimizer states, gradients, and activation values; the memory required during training is far greater than the weights themselves.
The following code snippet estimates the weight memory usage:
def weight_memory_gb(num_params: float, bytes_per_param: int = 2) -> float:"""Estimate the GPU memory occupied by model weights in GiB. Only includes weights, not optimizer states or activations."""return num_params * bytes_per_param / (1024 ** 3)# 1 trillion parameters, 16-bit storageprint(round(weight_memory_gb(1e12), 1)) # Approx. 1862.6 GiB
To accommodate these parameters, training systems typically combine multiple parallelization strategies, including data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism in MoE architectures. Each approach introduces communication overhead, and the choice of partitioning method directly impacts training efficiency.
3. Compression and Post-Training: Coping Strategies Under Compute Limits
When compute cannot scale indefinitely, enhancing model capability can proceed from other directions. The following types of techniques are common in engineering.
3.1 Knowledge Distillation
Knowledge distillation allows a smaller student model to learn the probability distribution output by a larger model, rather than just learning the ground truth. The soft labels output by the teacher model contain relative relationships between classes, conveying more information.
The schematic code is as follows. Functions like softmax and kl_divergence are placeholders, used only for
