Why Compute Demand Remains High After Ultra-Large Open-Source Models Are Released
A recent open-source model with nearly 3 trillion parameters has renewed discussion about model capability and compute demand. This article explains why stronger models often increase training and inference compute, how MoE architecture affects activated parameters and weight memory, how scaling laws estimate training FLOPs, and how KV Cache shapes long-context inference memory costs.
A recent open-source large model with nearly 3 trillion parameters has been officially released. The developers say its overall performance is slightly below the strongest current closed-source models, but it has reached the frontier in multiple technical evaluations and ranks high on a well-known programming benchmark leaderboard. The original announcement did not specify whether this figure refers to total parameters or activated parameters, and evaluation conditions should be checked against the official report.
After the announcement, some semiconductor and AI infrastructure-related stocks declined. Some investors argued that stronger models require less compute, which would hurt demand for compute infrastructure. This reasoning ignores the actual compute consumption of large-model training and inference. This article starts from the underlying mechanisms to explain why stronger models usually lead to more compute investment, and it identifies the conditions under which this conclusion could change.
1. Open Source Lowers the Access Barrier, Not the Runtime Barrier
Ultra-large models often use a mixture-of-experts (MoE) architecture. In this architecture, only a subset of experts is activated for each token computation. Therefore, the compute for a single inference is determined by the activated parameters, while the total parameters determine the GPU memory occupied by the weights. MoE can reduce per-computation cost, but it cannot reduce the storage requirement for the weights themselves.
Using 3 trillion parameters as the basis, the weights alone require about 6 TB of GPU memory in FP16, about 3 TB in INT8, and about 1.5 TB in INT4. Even with quantization, a single server can hardly hold the complete weights, so deployment usually requires multi-GPU or multi-node distributed configurations. Open source gives developers the model files, but running the model still requires substantial high-bandwidth GPU memory and cluster interconnect resources.
2. Training: Compute Grows with Both Parameters and Data
The industry commonly uses Scaling Law to describe the relationship between model performance, parameter count, and training data volume. To achieve better results with a large model, the parameter count and the number of training tokens usually need to grow together.
Total training compute can be roughly estimated as C ≈ 6 × N × D. Here, C is the total number of floating-point operations, N is the number of parameters involved in computation, and D is the number of training tokens. This formula has several preconditions: the model is a dense Transformer, the coefficient 6 comes from the compute of forward and backward passes, and the estimate does not include overhead such as activation recomputation, hyperparameter search, failed reruns, and data cleaning.
Two points require attention when using this formula. First, for an MoE model, N should be the activated parameters; otherwise, the per-step compute will be overestimated. Second, total compute is proportional to both N and D, but actual cluster size also depends on training duration, hardware utilization, and parallel efficiency. An increase in parameters does not necessarily lead to a proportional expansion of the cluster. Under the same training duration and hardware conditions, larger activated parameters and more training data do mean more computation.
For example, if the activated parameters are 100 billion and the training tokens are 15 trillion, then C ≈ 6 × 100 billion × 15 trillion = 9 × 10^24 FLOPs. If the activated parameters double while the token count remains unchanged, total compute also doubles.
3. Inference: KV Cache Determines the Memory Cost of Long Contexts
Autoregressive generation is divided into two stages: Prefill and Decode. The Decode stage needs to store the Key and Value tensors of historical tokens, namely the KV Cache. The size of the KV Cache is proportional to the number of layers, the number of KV heads, the dimension per head, the sequence length, and the number of concurrent requests.
The type of attention mechanism significantly changes this cost. In standard multi-head attention (MHA), each query head has a corresponding KV head. Grouped-query attention (GQA) lets multiple query heads share a group of KV heads, while multi-query attention (MQA) keeps only one group of KV heads. Fewer KV heads mean a smaller cache.
The corrected estimation function is shown below:
def estimate_kv_cache_bytes(num_layers, num_kv_heads, head_dim,batch_size, seq_len, dtype_bytes=2):"""Return the number of bytes occupied by the KV Cache.num_layers: Number of Transformer layersnum_kv_heads: Number of KV heads (equals query heads for MHA; 1 for MQA)head_dim: Dimension of a single head; under MHA, hidden_size = num_heads * head_dimbatch_size: Number of requests processed concurrentlyseq_len: Context length per request (number of tokens)dtype_bytes: Number of bytes per element; 2 for FP16, 1 for INT8 KV Cache"""per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytesreturn per_token * batch_size * seq_len
