Why Ultra-Large Language Models Are Hard to Scale: Compute Complexity and Engineering Efficiency
Parameter count alone does not determine a model’s training cost, inference speed, or practical value. This article explains the differences between total and active parameters, the storage, communication, and energy bottlenecks behind large-scale models, why matrix multiplication dominates deep learning workloads, and why smaller models can sometimes deliver better efficiency. It also outlines the metrics and engineering techniques that matter when evaluating and deploying modern AI systems.
1. What Does “Tens of Trillions of Parameters” Really Mean?
One of the most common mistakes when discussing large language models is treating total parameters, parameters used during a single inference pass, and effective parameters used during training as the same thing.
A model’s total parameter count usually refers to the number of all learnable weights it contains. In a dense model, most parameters participate in computation for every input token. In a mixture-of-experts architecture, however, a model may contain a very large number of total parameters while activating only a subset of experts for each token. As a result, two models with the same total parameter count can have completely different training costs, inference latency, and memory requirements.
The following concepts should also be distinguished:
- Total parameters: The total number of weights stored in the model.
- Active parameters: The number of parameters actually used in a single forward pass.
- Training compute: The total floating-point computation required to train on the full dataset.
- Inference compute: The computation required to generate one or more tokens.
- Memory usage: The memory required not only for weights, but also for activations, caches, gradients, and optimizer states.
Therefore, a model with tens of trillions of parameters does not necessarily cost dozens of times more to train or run than another model. Conversely, a model with fewer parameters is not automatically inefficient. Its effectiveness also depends on the data, training objectives, architecture, optimization methods, and inference system.
2. Why Parameter Counts Cannot Grow Without Limit
Increasing model size can improve representational capacity in theory. In engineering practice, however, larger models encounter several bottlenecks at the same time.
2.1 Storage Costs Rise Quickly
Assume a model has (P) parameters and each parameter requires (b) bytes of storage. The weights alone require:
[
M=P\times b
]
If a model contains (10^{12}) parameters and uses a low-precision format requiring two bytes per parameter, the weights alone occupy approximately 2 TB. During training, gradients, master weights, replicas, and optimizer states may increase actual memory requirements by several times the size of the weights.
As a result, ultra-large models generally cannot be trained or served on a single device. They require distributed storage, parameter sharding, and coordination across multiple devices. The central question then changes from “How fast can a device compute?” to “How efficiently can devices exchange data?”
2.2 Communication Overhead Can Eliminate Compute Gains
In distributed training, parameters, gradients, and intermediate activations must move between computing nodes. As the number of devices increases, both communication frequency and data volume can grow significantly.
If a matrix operation requires relatively little arithmetic but involves transferring a large amount of data between devices, the compute units may spend much of their time waiting. Adding more devices therefore does not necessarily reduce training time linearly. Synchronization, queuing, network congestion, and failure recovery can even reduce overall efficiency.
This explains a common phenomenon in large-model engineering: theoretical compute capacity continues to grow, while actual utilization does not increase at the same rate.
2.3 Energy and Thermal Management Become System Constraints
Large-model training often runs continuously for extended periods. Even if the power consumption of an individual accelerator is manageable, a large cluster creates significant requirements for power delivery, cooling, data-center space, and operations.
Model scale is therefore not just an algorithmic issue. It is determined jointly by algorithms, hardware, networking, storage, and data-center infrastructure. Comparisons based only on parameter counts often overlook the true cost of the entire system.
3. Why Matrix Multiplication Is Central to Large-Model Compute
Linear layers, attention projections, and many feature transformations in neural networks can ultimately be represented as matrix multiplications.
For two (n\times n) matrices (A) and (B), their product (C=AB) is defined as:
[
C{ij}=\sum{k=1}^{n}A{ik}B{kj}
]
With the straightforward implementation, each output element requires (n) multiplications and additions. Since there are (n^2) output elements, the total computational complexity is approximately:
[
O(n^3)
]
From an input-output perspective, reading the two matrices and writing the result requires at least (O(n^2)) data movement. Matrix multiplication therefore has a clear gap between computation and memory access: the theoretical data-movement lower bound is quadratic, while the naive algorithm requires cubic arithmetic operations.
This does not mean real systems can easily achieve (O(n^2)) complexity. Matrix multiplication must still perform a large number of independent multiply-accumulate operations. Even when more advanced mathematical algorithms are used, constant factors, numerical stability, parallel implementation difficulty, and memory-access patterns affect real-world performance.
In deep learning, matrices are also not limited to simple square matrices. They appear as batched tensors with many different shapes. System performance depends on factors such as:
- Whether matrix dimensions are suitable for hardware parallelism;
- Whether data is contiguous and has good locality;
- Whether already-loaded data can be reused;
- Whether the required precision is suitable for low-precision compute units;
- Whether devices need frequent synchronization;
- Whether operators can be fused to reduce intermediate writes.
Peak compute is therefore only a theoretical indicator. Effective throughput, memory bandwidth, communication efficiency, and overall utilization are often more important.
4. Why “Larger” Does Not Always Mean “More Capable”
Increasing parameter count can improve a model’s fitting capacity, but the benefits do not grow indefinitely.
First, model quality is constrained by training-data quality. If the data contains duplication, noise, errors, or distribution bias, adding parameters may simply allow the model to memorize problematic data more thoroughly rather than improve generalization.
Second, training objectives matter. Models of the same size can achieve significantly different results because of differences in data mixtures, task design, optimization procedures, and training stability.
Third, architecture determines how efficiently parameters are used. A dense model may contain many parameters but call on all of them during every inference step. A model with sparse activation can preserve a large total capacity while reducing the computation required for each input.
Finally, model capability is multidimensional. Language generation, code reasoning, mathematical problem-solving, long-context processing, multimodal understanding, and tool use may not improve at the same rate as parameter count. A single parameter metric cannot fully describe model quality.
5. Why Some Models May Choose Smaller Parameter Counts
A smaller model does not necessarily indicate weaker technology. It may represent a rational choice under practical constraints.
5.1 Focusing on Results per Unit Cost
Real-world applications usually care about how many useful tasks can be completed for a given cost, not simply how many parameters a model contains. If a smaller model delivers sufficient quality for most target scenarios while significantly reducing inference costs, it may be more attractive for enterprise deployment and high-volume usage.
5.2 Supporting Local and Industry Deployment
Many organizations need to run models in private environments or on restricted networks. Larger models require more memory, networking capacity, and operational expertise. More compact models are easier to deploy in resource-constrained environments and are also easier to adapt to industry-specific data.
5.3 Enabling Faster Iteration and Optimization
The larger the model, the longer each training run takes and the more expensive experimentation becomes. Smaller models support more rounds of architectural testing, data cleaning, and training-strategy comparison. This can make it easier to identify problems and improve the system continuously.
5.4 Prioritizing Inference Efficiency
Training is generally a one-time cost, while inference costs continue to accumulate with user volume and request frequency. Quantization, pruning, distillation, caching, and batching can reduce long-term operating costs while preserving useful capabilities.
6. Which Metrics Actually Matter?
Objective model comparisons should consider at least the following dimensions:
- Task quality: Accuracy, completeness, consistency, and factual alignment on target tasks.
- Cost per unit of work: The computing resources required to process a given amount of input or output.
- Response latency: Time to first output and subsequent generation speed.
- Throughput: The number of tasks the system can process per second under concurrent workloads.
- Resource usage: Requirements for memory, storage, and network bandwidth.
- Context capability: Stability when understanding and retrieving information from long inputs.
- Maintainability: The difficulty of version upgrades, failure recovery, monitoring, and security management.
- Generalization: Performance on previously unseen data and tasks.
- Reliability: Stability during long-running and high-concurrency workloads.
- Energy efficiency: The useful output or task results produced per unit of electricity.
Only by considering these metrics together can organizations determine whether a model architecture has long-term value.
7. Engineering Techniques for Improving Model Efficiency
7.1 Low-Precision Computing
Using lower-bit data formats, when accuracy remains acceptable, can reduce storage and bandwidth pressure while increasing matrix-computation throughput. Lower precision is not always better, however. Different layers, operators, and tasks require separate error evaluations.
7.2 Quantization
Quantization maps high-precision weights to lower-precision representations, reducing the model’s memory footprint. In production deployment, engineers must consider outliers, activation distributions, and accumulated error rather than focusing only on the compression ratio.
7.3 Knowledge Distillation
A teacher model can guide a smaller model to learn some of its capabilities. Distillation is not simply a process of copying parameters. The student may learn output distributions, reasoning patterns, or task-specific features.
7.4 Sparsity and Expert Architectures
The core idea of sparsity is to access only the parameters needed for a particular computation. This allows a model to increase its total capacity while controlling per-request compute. However, sparse architectures introduce additional complexity in routing, load balancing, and communication management.
7.5 Operator Fusion and Memory Optimization
Combining consecutive operations can reduce the creation and reading or writing of intermediate tensors. In inference systems, reducing memory access can sometimes be more effective than adding more arithmetic units.
7.6 Caching and Batching
For generative workloads, previously computed results can be cached to avoid redundant computation. For concurrent requests, dynamic batching can combine multiple requests and improve device utilization.
8. From a Parameter Race to a Race for Effective Intelligence
Model development should not become a competition based solely on parameter counts. Parameters are an important metric, but they are only one variable in the relationship between capability, cost, and engineering complexity.
A more practical goal is effective intelligence: Can a model complete real tasks with acceptable resources? Can it run reliably? Can it adapt to different scenarios? Can organizations and individuals use it at scale?
From this perspective, smaller models are not inherently inferior, and ultra-large models are not automatically superior. Smaller models may offer advantages in deployment cost and response speed, while larger models may perform better on complex reasoning and broad knowledge coverage.
The future is more likely to involve layered cooperation among models: large models handling complex tasks, smaller models serving high-frequency workloads, and specialized models addressing specific domains. A routing system can then direct each request to the most appropriate model.
Ultimately, model competition is not about who has the most parameters. It is about who can achieve the best balance among capability, cost, speed, reliability, and maintainability. That balance is also a key measure of whether AI infrastructure has reached operational maturity.
