The LLM Parameter Count vs. Compute Cost Debate: From Matrix Multiplication to the Roofline Model and Scaling Laws

The Real Cost of LLM Parameters: Why Size Is Only Half the Story

Debates about the scale of large language models (LLMs) almost always turn to the same question: whose model has more parameters? Some tallies of leading closed models put the largest of them in the tens of trillions of parameters. Publicly disclosed parameter counts for many other models still range from hundreds of billions to a trillion. So the next question follows: is a larger parameter count always better?

Frankly, comparing parameter counts in isolation is of limited value. Whether parameters translate into capability depends on three things:

• whether the available compute was fully used;
• whether compute and data are well matched;
• whether the output per unit of compute is worthwhile.

Rather than argue over which number is larger, this article starts from the basic computational unit of LLMs, matrix multiplication. It works through the full path from theoretical complexity to real compute cost in both training and inference, and closes with a framework for assessing parameter scale rationally.

1. Three “Scales” That Often Get Confused

1.1 Total Parameters: The Ceiling on Model Capacity

Total parameters are all the elements across a model’s weight matrices.

• Dense models: use every parameter at every computation.
• Mixture-of-Experts (MoE) architectures: split the feed-forward layers into several “experts.” A routing mechanism selects only a few of them for each token.

Total parameters therefore reflect the upper bound on how much knowledge a model can hold.

1.2 Activated Parameters: What a Single Token Actually Computes With

• Dense models: activated parameters equal total parameters.
• MoE models: activated parameters make up only part of the total, for example 2 experts out of several dozen.

As a result, a model with trillions of total parameters may use roughly the same compute per token as a dense model with several hundred billion parameters.

1.3 FLOPs per Token: The Real Yardstick for Training and Inference Cost

In a forward pass, each parameter contributes roughly two floating-point operations per input token (one multiplication and one addition), or about 2N FLOPs, where N is the number of activated parameters. This is the most basic formula for estimating compute cost.

Beyond FLOPS, engineers often measure hardware compute capacity in TOPS (tera operations per second). By convention, 1 TOPS corresponds to the throughput of 2 FLOPS.

1.4 The Differences at a Glance

Dimension	Total parameters	Activated parameters	FLOPs per token
Meaning	Total number of weight elements	Parameters that actually compute for a single token	Floating-point operations each token performs in a forward pass
Mainly determined by	Model capacity	Routing strategy and activation ratio	Model architecture and attention implementation
Effect on training memory	Decisive (weights, optimizer states, gradients)	No direct effect	No direct effect
Effect on per-token compute cost	No direct effect	Decisive	Decisive
Typical case	MoE models	Dense models	Linear attention vs. standard attention

MoE trades a large but sparse design for capacity, while dense models stay small and dense so that every token has predictable compute. Which approach wins depends on the compute budget, data quality, and inference latency requirements. Parameter count alone cannot settle the question.

2. Back to the Starting Point: The Complexity of Matrix Multiplication

No matter how architectures evolve, most of an LLM’s compute is concentrated in General Matrix Multiplication (GEMM), including:

• the Q, K, and V projections and the output projection in attention;
• the attention score and attention weight matrices;
• the layer-by-layer transformations in feed-forward and expert networks.

These operations are essentially products of high-dimensional tensors. For a two-dimensional example, let matrix A be m×k and matrix B be k×n. Each element of the product C is defined as:

C[i][j] = Σ(t = 1 → k) A[i][t] × B[t][j]

Each of the m×n output elements needs k multiply-adds, for a total of about 2m·n·k FLOPs. When A and B are both n×n square matrices, the work falls to the order of 2n³. For a linear layer with input and output dimension d (so m = n = k = d), it is exactly 2d³. Put simply, every step deeper or wider increases compute cubically.

2.1 Why the Naive Implementation Is Slow

The textbook triple loop follows the definition literally: the outer loops enumerate output positions (i, j), and the inner loop accumulates over t from 1 to k. Pseudocode:

# Naive implementation: accumulate element by element per the definition. Θ(n³) compute, Θ(n³) data movement.
for i in 0 .. m-1:
    for j in 0 .. n-1:
        acc = 0
        for t in 0 .. k-1:
            acc += A[i][t] * B[t][j]
        C[i][j] = acc

This has two problems:

• The loop order does not match the storage layout. C and B are usually stored in contiguous rows. If the outer loop increments j, each write to C[i][j] jumps across a large span of memory, and so do the reads of B[t][j]. The compute stays the same, but memory accesses multiply.
• Data reuse is minimal. Each A[i][t] and B[t][j] is used immediately after it is read. A is read in full n times and B m times, so most of the bandwidth goes to repeated transfers.

The fix is clear: reorder the loops (i-k-j), make the innermost loop a contiguous pass over j, and hold A[i][t] in a register so each element is read from memory only once.

2.2 The Trivial Lower Bound and Fast Matrix Multiplication

Matrix multiplication cannot use fewer than Θ(n²) operations. The inputs hold 2n² elements, and each must be read at least once. This is the trivial lower bound.

The classic divide-and-conquer algorithm lowers the exponent. It splits a 2×2 block multiplication into 8 half-size multiplications, then uses a clever combination of additions and subtractions to replace those 8 with 7:

T(n) = 7·T(n/2) + Θ(n²)  ⇒  T(n) = Θ(n^log2(7)) ≈ Θ(n^2.807)

This kind of algorithm was generalized further over time. After decades of research, the best known upper bound on the time complexity of matrix multiplication has been pushed down to about 2.372 (usually written as ω). However, these “faster” algorithms are almost never used for training or inference in practice, for three reasons:

• Constant factors and numerical stability. Fast algorithms trade many additions and subtractions for fewer multiplications. The deeper the recursion, the more floating-point rounding error accumulates.
• Memory traffic outweighs the compute savings. Additions and subtractions still need to read and write data. When the data far exceeds cache capacity, the extra memory traffic often cancels out the gains from the algorithm.
• Ecosystem inertia. The core GEMM kernels in high-performance computing libraries have been refined for decades. They are heavily tuned for cache use, instruction-level parallelism, and memory access efficiency. Improving their asymptotic complexity without losing constant-factor performance is extremely hard.

In short, algorithms better than O(n³) exist in theory, but practice is dominated by highly optimized O(n³) implementations. That is the real picture of matrix multiplication.

2.3 What Ignoring Constant Factors Means

Suppose two implementations differ by 100× in data movement. Memory access alone then opens a gap of two orders of magnitude. Theory studies growth as n approaches infinity, while engineering optimization targets constant factors at finite sizes. The two do not conflict, but looking at only one side leads to one-sided conclusions.

3. From Asymptotic Complexity to Real Throughput: The Roofline Model

In theory, GEMM compute grows in proportion to data size, so larger workloads should be more efficient. In practice, this is not always true. Performance is constrained by both peak compute and memory bandwidth, and where the two intersect is decided by arithmetic intensity.

3.1 Two Key Metrics

• Arithmetic intensity: the number of floating-point operations per byte moved from memory (FLOPs/Byte).
• Ridge point: the ratio of a hardware’s peak compute to its peak memory bandwidth (ridge point = peak FLOPS ÷ peak bandwidth).

This gives a simple test:

• Below the ridge point: the workload is in the memory-bound region, where performance is limited by bandwidth.
• Above the ridge point: the workload is in the compute-bound region, where performance is limited by compute.

3.2 Using the Roofline Model to Estimate GEMM Performance

Set m = n = k again. In the ideal case, A and B are each read once and C is written once. The data volume is about 24n² bytes (assuming single precision, 4 bytes per element), which gives an arithmetic intensity of about n/12 FLOPs/Byte.

The larger n is, the higher the arithmetic intensity, and the more likely the workload is to become compute-bound. Real performance still depends on whether the blocking strategy actually achieves data reuse.

3.3 Tiling: From “Can’t Finish Computing” to “Computes Fast”

Tiling cuts matrices into blocks sized to fit the cache. Each data block is then reused as many times as possible before it is evicted from fast memory. Simplified pseudocode:

# Tiled implementation: reuse each data block after loading to cut repeated memory traffic.
for i0 in 0 .. m-1 step Bm:
    for j0 in 0 .. n-1 step Bn:
        for t0 in 0 .. k-1 step Bk:
            # Load the A and B sub-blocks into the on-chip cache (or shared memory).
            for i in i0 .. min(i0+Bm, m)-1:
                for j in j0 .. min(j0+Bn, n)-1:
                    for t in t0 .. min(t0+Bk, k)-1:
                        C[i][j] += A[i][t] * B[t][j]

Three levels of design are worth noting:

• Register blocking. The innermost micro-kernel has each thread compute several elements of C while reusing the same A and B values. This keeps memory traffic to a minimum.
• On-chip cache blocking. The middle level keeps sub-blocks in shared memory for the threads in the same thread block to share.
• Main memory blocking. The outermost level sets how many times each sub-block is read from main memory. Larger blocks mean more reuse, as long as cache capacity allows.

3.4 A Simple Memory Hierarchy Analysis

Suppose fast memory can hold M elements. A classic analysis gives an instructive result: the lower bound on memory accesses for matrix multiplication is about n³/√M.

More precisely, let fast memory hold M elements, and let each transfer move b contiguous elements. Once you balance “larger blocks mean more reuse” against “blocks must fit in fast memory,” the block size settles at about √M elements. Data movement then falls to the order of n³/√M.

For the same compute, the data movement before and after optimization can differ by a factor proportional to √M. That is the root cause of “same complexity, tens of times the performance.”

3.5 Verifying the Trend with a Small-Scale Benchmark

A short benchmark script shows how real performance changes with size. Software-only implementations reach an arithmetic intensity far below the hardware’s theoretical level, so this is mainly for observing trends.

# Software-only benchmark: measure actual throughput at each size and compare it with the theoretical compute.
sizes = [256, 512, 1024, 2048, 4096]
for n in sizes:
    A = random_matrix(n, n, dtype=float32)
    B = random_matrix(n, n, dtype=float32)
    warmup(iterations=3)          # Compile the kernels and warm the cache to exclude first-run overhead.
    t = measure_time(() => matmul(A, B), iterations=10)
    tflops = 2 * n**3 / t / 1e12  # TFLOPS actually achieved.
    print(n, tflops)

When reading results like these, note the following:

• Small n: throughput usually rises quickly as size grows. Fixed overheads are spread over more work, and cache reuse improves.
• Beyond cache size: software-only implementations may hit the memory bandwidth ceiling and flatten out, or even drop.
• Highly optimized GEMM libraries: throughput keeps growing at medium and large sizes. Optimized implementations organize memory access around computation, which carries them past the memory-bound region.

The conclusion is clear: with the same complexity, different implementations can differ in performance by an order of magnitude.

4. From FLOPs to Scaling Laws: Balancing Compute, Parameters, and Data

Analyzing a single compute path is straightforward, but training a model chains trillions of matrix multiplications together and repeats them over many iterations. Scaling laws describe the relationships among loss, parameters, data, and compute. Three rules of thumb are widely used:

• Compute formula: total training compute C ≈ 6·N·D, where N is the number of parameters and D is the number of training tokens. The coefficient 6 is the 2N of the forward pass plus the 4N of the backward pass. Since forward and backward together take about 3× the compute of the forward pass alone, training costs far more than inference.
• Compute-optimal training: with a fixed compute budget, model size and training tokens should grow at roughly the same rate, empirically about 20 tokens per parameter.
• Inference cost constraint: when the inference budget is limited, favor a larger model and train it on correspondingly fewer tokens.

These rules point to a simple but important principle. When compute is limited, scaling parameters alone leaves the model having seen too little. Scaling data alone leaves the model unable to fit everything in. “More parameters is always better” holds only when compute and data grow together.

Frontier models have to go a step further. High-quality content in the data is finite, so the returns from scaling parameters and data alone diminish step by step. The industry has therefore broadly turned to efficiency gains from high-quality data, post-training, inference-time compute, and MoE architectures.

The scaling law formulas are also sensitive to constant factors. The coefficient 6 accounts only for the arithmetic work of the forward and backward passes. Real training adds:

• the products of the attention matrix and its transpose;
• normalization operations such as LayerNorm;
• optimizer updates;
• parallel communication;
• pipeline bubbles.

Each iteration also runs the full chain of data loading → forward → backward → communication → optimizer update. A single matrix multiplication running close to peak throughput does not mean end-to-end training does too.

5. Beyond the Total Parameter Count: Attention, Context, and the Hidden Growth of Inference Cost

5.1 The Quadratic Cost of Attention

Standard attention computes a relevance score between each token and every token in the sequence, so its compute grows with the square of the sequence length. One compute path shows this:

Q = X · Wq        # Shape n×d, ~2n·d² FLOPs.
K = X · Wk        # Shape n×d, ~2n·d² FLOPs.
V = X · Wv        # Shape n×d, ~2n·d² FLOPs.
S = Q · Kᵀ        # Shape n×n, ~2n²·d FLOPs. Grows quadratically with sequence length.
P = softmax(S)
O = P · V         # Shape n×d, ~2n²·d FLOPs. Grows quadratically with sequence length.

When the sequence length n is on the same order as the hidden dimension d, the two n×n matrix multiplications (about 4n³ FLOPs) cost more than the four projection matrices combined (about 6n·d² FLOPs). This is the mathematical root of the sharp rise in long-context inference cost.

5.2 Mainstream Optimization Directions

To ease the quadratic cost of attention, several mature techniques have emerged:

• Sparse and local attention: compute relevance only between some pairs of tokens.
• Multi-query and grouped-query attention: multiple query heads share one set of keys and values, reducing cache reads during decoding.
• Sliding window attention: each token attends only to a fixed-length window of nearby context.
• Linear attention: recast attention so its compute scales roughly linearly with sequence length.

All of these cut the compute and storage at the n×n scale. In effect, they reallocate the budget between expressiveness and compute cost.

5.3 The Two Stages of Inference

Decoding has two stages:

• Prefill: process all input tokens in parallel. This stage is compute-intensive.
• Decode: generate one token at a time. This stage is memory-intensive, and its bottleneck is reading model weights and cache.

Measuring inference efficiency therefore takes more than parameter count. You also need to look at batching strategy, cache management, quantization precision, and request scheduling.

5.4 The Price of Sparse Architectures

MoE lowers per-token compute but creates new engineering challenges:

• the total storage cost of expert weights remains;
• the routing All-to-All communication can become a bottleneck;
• an uneven expert load slows down overall progress;
• the routing decisions themselves require extra compute.

Judging whether a model is “smart” means comparing training cost, inference cost, memory footprint, and communication overhead side by side.

6. Five Dimensions for Assessing Parameter Scale

Back to the original question: parameter scale cannot be judged by a single number. Five dimensions help:

1. Transparent reporting. Is the published figure the total, activated, or effective parameter count? Does it separate the language model from any multimodal encoder?
2. Compute utilization. Actual hardware utilization during training (Model FLOPs Utilization, or MFU) reflects the quality of the engineering. A higher ratio means core operators such as matrix multiplication and the parallel strategies are used more fully.
3. Scaling efficiency. Under a fixed compute budget, compare the final loss and downstream task performance of models at different sizes. This measures output per unit of compute.
4. Inference cost. Look at the cost per million tokens and time to first token, not just parameter count.
5. Architecture fit. Different business needs, such as long context, low latency, or high concurrency, call for different architectures.

In one sentence: parameter scale is “capacity,” and compute utilization efficiency is the “conversion rate.” You need both. Seeing only capacity while ignoring the conversion rate easily overestimates a model’s real value. Conversely, talking only about efficiency while ignoring capacity makes it hard to support complex tasks.

7. Engineering Practice

7.1 Build a Layered Benchmarking System

• Operator level: benchmark matrix multiplication across different sizes and data types, and plot throughput against size.
• Layer level: test modules such as attention and feed-forward networks individually.
• Model level: measure throughput for the forward pass, the backward pass, and end-to-end training.
• Cluster level: assess multi-node, multi-GPU scaling efficiency and communication overhead.

At every level, fix the test conditions, record peak utilization, and compare results against the Roofline ceiling. That keeps you from being misled by single-point benchmark scores.

7.2 Use General Cloud Capabilities

LLM training and inference usually run on cloud platforms, and a few general-purpose building blocks are worth using:

• Container platforms: orchestrate distributed training and inference jobs and scale them on demand.
• Object storage: store datasets and training checkpoints and support parallel reads across nodes.
• Message queues: decouple data preprocessing from training.
• Logging, monitoring, and alerting: continuously track compute utilization, memory usage, and job failure rates.

On platforms such as Baidu AI Cloud, developers can build their training and inference infrastructure directly on these general capabilities and focus on the model and the business itself.

7.3 Make Good Use of Mixed and Low Precision

Mixed-precision training runs most of the compute for weights, activations, and gradients in half precision. It keeps single-precision master weights to preserve numerical stability.

For inference, 8-bit integer quantization cuts memory usage and decode bandwidth demand, with accuracy loss kept under control. In practice, keep these points in mind:

• different precisions have different requirements for numeric range and rounding error;
• key operators such as softmax, normalization, and loss functions usually need higher precision;
• mechanisms such as loss scaling are needed to avoid gradient underflow.

7.4 Prioritize Parallel Strategies and Communication Optimization

Data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism each fit different cases, and how they are combined directly determines scaling efficiency. Communication can be optimized by overlapping compute with communication, compressing gradients, and using hierarchical communication. Metrics such as the pipeline bubble ratio and communication wait time should be part of routine monitoring.

Conclusion

The story of matrix multiplication runs from the trivial Θ(n²) lower bound, through the Θ(n³) engineering reality, to the memory access savings of cache blocking. It shows that complexity analysis sets the direction, while real performance comes from careful tuning of constant factors, memory hierarchy, and parallel communication.

The parameter debate works the same way. The number itself is only the starting point. What really determines a model’s value is how efficiently compute, data, architecture, and engineering work together. Rather than argue over who has more parameters, it is better to return to measurable engineering metrics: compute utilization, output per unit of compute, inference cost, and scaling efficiency. These are the true yardsticks of a model’s strength.