Sparse LLMs: Total vs. Activated Parameters and What Really Drives Inference Cost

    Sparse mixture-of-experts (MoE) models decouple total parameters from activated parameters, so a single "parameter count" no longer tells the full story. This article explains how routing, expert parallelism, and load balancing work, breaks down how total parameters drive memory while activated parameters drive compute, and shows why the real bottleneck often lies in weight residency and decode bandwidth rather than in activation. It also covers memory budgeting, latency metrics (TTFT and TPOT), cold starts, fine-tuning costs, a scenario-first selection methodology, and production observability for running sparse LLMs cost-effectively.

    1. Parameter Count Is No Longer a Single Scale: From “Size” to “Structure”

    For a long time, we evaluated large models on one scale only: how many billions of parameters. But once sparsification became mainstream, “parameter count” stopped being a single number and turned into a set of numbers:

    • Total parameters: the size of all the model’s weights. This determines how much storage is needed to hold the model, both in GPU memory (VRAM) and host memory;
    • Activated parameters: the subset of parameters that actually take part in computation when processing a token. This sets the rough order of magnitude of compute consumption.

    In a classic dense architecture, the two are equal. Every time a token is generated, all weights participate in one forward pass, so “parameter count” is simultaneously a memory metric and a compute metric. But in a Mixture of Experts (MoE) architecture, the two are decoupled, and the gap can reach several times or even an order of magnitude. As a result, you get situations like this: a model is advertised as “hundreds of billions of parameters,” yet a single inference pass actually activates only a small fraction of them.

    For developers, the question that really matters is not “which number is bigger,” but:

    Under my own traffic profile, which parameter actually determines the bill?

    The answer depends on three variables: concurrency level, sequence length, and whether expert distribution is uniform. More notably, the industry has gradually converged on a clear trajectory — from “sparse, smaller-scale versions rebuilt around the base-model design” to “flagship sparse models”:

    • Instead of simply scaling the flagship version down proportionally, they are redesigned across attention structure, expert count and activation ratio, context-window length, and more;
    • The goal is to push inference cost into a lower range.

    The value of this trajectory lies not in being a “smaller version,” but in rematching the activation ratio and the resource ledger for the sparse architecture. Let’s break it down layer by layer.

    2. Three Key MoE Mechanisms

    2.1 Routing and Experts: Where Sparsity Happens

    The basic idea of a sparse mixture-of-experts model is to split the wide feed-forward network of a traditional Transformer into several smaller “expert” subnetworks, then use a lightweight router to decide which expert processes each token. A typical compute flow:

    1. For each token’s hidden representation, compute its match score against every expert;
    2. Take the top-k experts by score (k is usually 1 or 2);
    3. The token passes through only these k experts, and the output is aggregated as a weighted sum according to the routing weights.

    In mainstream designs, the attention layer is usually a dense computation shared by all tokens; sparsification happens only in the feed-forward layers. So here “sparse” means token-level: each token “awakens” only a small subset of parameters — it is not the case that an entire sequence goes through a single compute path.

    1. # Simplified illustration: a sparse expert layer with Top-k routing (pseudocode, to show the compute path)
    2. def moe_layer(x, router, experts, top_k=1):
    3. # x: a batch of token representations with shape [num_tokens, hidden_dim]
    4. logits = router(x) # score each expert
    5. topk_idx, topk_w = softmax(topk(logits, top_k))
    6. y = zeros_like(x)
    7. for eid in unique(topk_idx): # only iterate experts actually hit in this batch
    8. mask = (topk_idx == eid).any(dim=-1) # tokens routed to this expert
    9. if not mask.any():
    10. continue # empty expert: skip, consumes no compute
    11. # expert weights stay resident in memory, but the unselected part takes no part in the computation
    12. y[mask] += (x[mask] @ experts[eid].W) * topk_w[mask][:, eid, None]
    13. return y

    This pseudocode captures one key fact: expert weights do not disappear from memory just because they weren’t called. They are kept resident so the next token might need them.

    2.2 Expert Parallelism: Where the Cost of Sparsity Begins to Shift

    Because total parameters are often far larger than activated parameters, a single device usually can’t hold the whole model. In practice, different experts are sharded across different devices and deployed in parallel — this is expert parallelism. The direct consequences:

    • Trading communication for memory: devices must exchange tokens and their routing results. A common approach is to first send tokens to the device hosting their target expert (dispatch), then send them back in the original order once expert computation finishes (combine). This is typical all-to-all communication. Under high-speed interconnects, this latency is usually manageable; but in deployments with weaker interconnect bandwidth — such as using standard Ethernet across nodes — this overhead rises sharply;
    • Weight reads get spread out: at low concurrency, each request activates only a few experts, so very few weights actually need to be read. But as requests pile up and different requests’ routing becomes scattered, most experts get hit frequently. At that point the weights that must be read gradually approach the full set, and the “read only a small subset of parameters per token” advantage fades accordingly.

    2.3 Capacity Factor and Load Balancing: The Hidden Cost of Hot Experts

    Routing is decided by the model itself through learning, but the learned result isn’t necessarily uniform. A small number of popular experts may be selected by many tokens while others sit idle for long stretches, and inference latency gets dragged up by a handful of devices. Three common countermeasures:

    • Capacity limit: during training, cap the number of tokens each expert can process; tokens beyond the cap are dropped or rerouted to a shared pathway;
    • Routing balance: add an auxiliary loss during training to encourage tokens to spread evenly across experts;
    • Inference-side scheduling: dynamically assign requests by expert load, maintain expert replicas, or cache high-frequency experts’ weights in a faster memory tier.

    These are all engineering-level responses, which shows that a sparse model’s real efficiency depends not only on the activation ratio but also on whether routing is evenly distributed. So when evaluating such models, you can’t look at architecture parameters alone — you also have to observe the expert hit distribution under production traffic.

    3. Cost Breakdown: Total Parameters Determine Memory, Activated Parameters Determine Compute

    Translate the mechanisms above into a resource bill, and sparse models show a clear “dual-track” structure:

    Resource Main determining factor Actual behavior in sparse models
    Weight memory Total parameters × bytes per precision Grows linearly with total parameters; activation sparsity can’t help
    Runtime memory KV cache, in-batch intermediate tensors Basically independent of sparsity; grows with concurrency and context length
    Per-token compute Roughly proportional to activated parameters Significantly reduced, but expert parallelism adds communication overhead
    Decode bandwidth Expert weights actually read per token Greatly reduced at low concurrency; approaches the full set at high concurrency
    Fine-tuning cost Total parameters and available memory Full-parameter fine-tuning barrier rises; usually relies on parameter-efficient methods

    The industry commonly uses a few rough but reliable estimation formulas (for order-of-magnitude judgment only):

    1. Forward compute ≈ 2 × activated parameters × number of output tokens (unit: FLOPs)
    2. Weight memory ≈ total parameters × bytes per precision (FP16 ≈ 2, INT8 ≈ 1, INT4 ≈ 0.5)
    3. KV Cache ≈ 2 × num_layers × seq_length × concurrency × num_kv_heads × head_dim × bytes per precision

    3.1 The Real Bottleneck in Decoding Is Bandwidth, Not Compute

    LLM inference splits into two phases with very different bottlenecks:

    • Prefill phase: processes the entire input in one pass; it is compute-intensive, and GPU compute utilization is high;
    • Decode phase: generates one token at a time; it is memory-bandwidth-intensive. Each token requires moving the corresponding weights from memory into the compute units, so latency is dominated by bandwidth.

    Sparse architectures mainly optimize the compute and weight-read volume per token, which helps the prefill phase markedly. In the decode phase, the effect is also significant at low concurrency, because a single request hits only a few experts; at high concurrency, the effect is partly offset by scattered routing and expert-parallelism communication.

    3.2 An Illustrative Example

    The assumed values below only demonstrate the algorithm and don’t represent any specific model:

    • 100 billion total parameters, 10 billion activated, all in FP16;
    • 60 layers total, with attention and MoE layers each making up half.

    We can work through it in sequence:

    1. Weight footprint: 100B × 2 bytes ≈ 200 GB. Even on accelerator cards with 80 GB each, this needs to be sharded across multiple cards;
    2. Per-token forward compute: 2 × 10B ≈ 200 billion FLOPs, only about one-tenth of a dense model at the same parameter scale;
    3. Long-context cost: assuming KV heads × head dim = 1024, when the context extends to 128K, a single request’s KV cache is about 2 × 60 × 131072 × 1024 × 2 bytes ≈ 30 GB;
    4. Memory budget: assuming weights are sharded across 8 cards, that’s about 25 GB per card; with roughly 50 GB left on an 80 GB card, in theory it can hold at most one long-context request at a time.

    This example reveals an easily overlooked fact: for a sparse architecture, what often determines concurrency capacity is the memory taken up by weights, not the activated parameters. Small activated parameters only lower the compute cost; they do not lower the “must be resident” cost.

    4. Once the Deployment Barrier Drops, Four Things to Watch

    Sparse architectures and redesigned lightweight specs significantly cut inference compute and per-token cost. But “fits in memory” and “runs stably” are two different things. What really determines the online experience and total cost is the following four areas.

    4.1 Memory Budget: The Triangle of Weights, KV Cache, and Concurrency

    Take available memory, subtract weights, framework overhead, and a fragmentation reserve — what remains is the KV cache budget, and that directly determines max concurrency. Common techniques include:

    • KV cache quantization: reduces the cache footprint per token;
    • Paged cache management: reduces memory fragmentation;
    • Prefix cache reuse: common prefixes like system prompts and knowledge-base documents only need to be computed once;
    • Context tiering: proactively truncate or reroute requests that don’t need long context.

    Also, multi-card sharding isn’t limited to expert parallelism. Tensor parallelism further splits each expert’s weights, and pipeline parallelism shards by layer. Real deployments often combine several of these to balance memory, communication, and latency.

    4.2 Latency Structure: Measure First-Token Latency and Token Speed Separately

    Online latency usually has two parts:

    • TTFT (time to first token): from request sent to first token received, mainly determined by the prefill phase’s compute. The longer the input, the higher the TTFT. Sparse activation delivers its most direct gains here — scenarios like long-document summarization and code completion benefit especially;
    • TPOT (time per output token): mainly determined by the decode phase’s memory bandwidth, and affected by expert-parallelism communication and routing balance.

    A sparse model can have a clear edge in TTFT yet be dragged down in TPOT by communication. So you must measure both metrics separately, and weigh them together against the business’s differing sensitivity to first-token speed versus reading speed.

    4.3 Elasticity and Cold Start: Don’t Let Small Specs Become Small Headaches

    Smaller total parameters and sparser activation mean each unit of compute can carry more requests. This cuts both ways:

    • Positive: for the same traffic volume, fewer devices are needed, and the peak-to-trough gap is easier to absorb with autoscaling;
    • Negative: model weights are still huge; scaling out requires distributing weights to new nodes or reloading them from storage, and cold start can take several minutes. Sudden spikes at peak time, if they rely on temporary scaling, often can’t respond in time, while permanently reserving for peak drags down overall utilization.

    In practice, a common approach is to keep a baseline pool of always-on replicas, then combine it with traffic forecasting and advance warming. Meanwhile, moving latency-insensitive tasks (like batch document processing and offline data generation) onto async queues to run off-peak from online inference noticeably improves overall resource utilization.

    4.4 Fine-Tuning and Private Deployment: Total Parameters Become the Barrier Again

    Inference-side costs dropped, but customization-side costs haven’t necessarily. Full-parameter fine-tuning needs to hold weights, gradients, and optimizer states at the same time. Taking an AdamW-type optimizer as an example, the extra overhead in FP16 mixed precision is about 12× the parameter count in bytes (parameter copy 2 + momentum 2 + variance 4 + FP32 parameter copy 4). Even training in BF16 (about 10×), a model with 100 billion total parameters needs at least 1 TB of memory just for these states.

    So parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) are almost mandatory for such models. Also, LoRA only trains the newly added small matrices, and the routing logic itself is usually kept frozen. If business data clearly skews toward certain domains, the routing distribution may drift away from the balance assumption established at pretraining, and you need monitoring to catch it in time.

    5. Selection Methodology: Ask About the Scenario First, Then Look at Parameters

    Facing choices like “flagship or lightweight” or “sparse or dense,” here’s a suggested decision order:

    1. Traffic profile: high-concurrency online services with short inputs and short outputs benefit most from sparse architectures’ compute advantage; single-user scenarios with low concurrency and extreme low-latency goals may find expert-parallelism communication overhead cancels the gains, making it not necessarily worthwhile;
    2. Context length: in ultra-long-context scenarios, the KV cache is often the biggest memory consumer, so prioritize comparing cache-optimization capabilities rather than parameter counts;
    3. Domain-adaptation depth: for deep customization, assess the fine-tuning resource budget in advance;
    4. Deployment form: public cloud on-demand, on-premise private deployment, and edge devices have very different resource ceilings;
    5. Migration cost: different models don’t fully match in support for prompts, output formats, and structured outputs (such as function calling), so leave a round of regression testing when swapping models.

    A practical rule of thumb:

    At an equivalent quality level, prefer the sparse model with smaller activated parameters and more balanced routing; when total parameters become the memory bottleneck, or expert-parallelism communication becomes the latency bottleneck, then consider a dense model or a smaller dedicated model.

    6. Verification and Rollout: From Benchmarks to Production Observability

    6.1 Stress-Test With Business Data, Not Just Public Leaderboards

    Public benchmarks reflect average capability, but business performance depends on the long tail. It’s recommended to sample from real logs to build an eval set, then separately tally quality metrics (accuracy, pass rate, human ratings) and performance metrics (P50/P95/P99 latency, throughput, peak memory). Points to watch during stress testing:

    • Input/output length distribution: fixed-length tests can badly misjudge; sample according to the real distribution;
    • Concurrency gradient: ramp pressure from low to high in steps to find the throughput inflection and the latency inflection;
    • Cold start and steady state: measure separately the first request after scaling out and the behavior after stable running.
    1. # Illustration: a lightweight stress-test script that outputs comparable percentile metrics
    2. import random, time
    3. def sample_cases(n=100):
    4. # Sample cases from real logs; here we approximate with randomized lengths
    5. return [
    6. {"prompt": "example user question " * random.randint(10, 200),
    7. "max_tokens": random.randint(32, 512)}
    8. for _ in range(n)
    9. ]
    10. def benchmark(endpoint, concurrency=8):
    11. cases = sample_cases()
    12. ttfts, tot = [], time.perf_counter()
    13. # In a real implementation, send requests concurrently with a coroutine pool of size `concurrency`
    14. for c in cases:
    15. t0 = time.perf_counter()
    16. first = request_first_token(endpoint, c["prompt"])
    17. ttfts.append((time.perf_counter() - t0) * 1000)
    18. for _ in range(c["max_tokens"] - 1):
    19. request_next_token(endpoint)
    20. ttfts.sort()
    21. elapsed = time.perf_counter() - tot
    22. print(f"TTFT P50={ttfts[len(ttfts)//2]:.0f}ms "
    23. f"P95={ttfts[int(len(ttfts)*0.95)]:.0f}ms "
    24. f"throughput≈{elapsed:.1f}s / {concurrency} concurrency")

    6.2 Make Routing Behavior Observable

    In production, it’s recommended to add the following observability items to the inference service:

    • Record per-request first-token latency, total time, input/output length, and response status code;
    • Periodically sample each expert’s hit ratio and per-card load to detect routing drift promptly;
    • Monitor the memory water level, KV cache utilization, and request queue time;
    • Set up sampled tracing for slow requests to pin down whether the cause is expert-parallelism communication, long-sequence prefill, or memory shortage causing queuing.

    Collecting and visualizing this data can rely on general capabilities such as cloud platforms’ log services and monitoring/alerting, forming a closed loop from metrics to alerting to troubleshooting.

    6.3 A General Path for Cloud Deployment

    On cloud platforms such as Baidu AI Cloud, you can containerize the inference service and deploy it on a container service, store model weights in object storage for on-demand loading, pair it with autoscaling to absorb traffic fluctuations, and observe the whole thing uniformly through logs and monitoring. The pipeline doesn’t need to bind to specific hardware or proprietary interfaces, which makes migration between environments easy.

    6.4 Open Weights Don’t Mean Zero Cost

    The value of sparse small-spec models is a “low barrier,” but a low barrier isn’t no barrier. A complete cost account includes at least:

    • Purchasing or renting inference accelerator cards;
    • The tuning and operations effort from the engineering team;
    • Idle costs during off-peak periods.

    Only under a stable business volume does self-hosting’s cost advantage gradually emerge. For low-volume, high-volatility workloads, choosing a pay-as-you-go model service in the cloud is often the more economical option.

    7. Common Misconceptions

    1. Small activated parameters mean low memory footprint: weights stay resident according to total parameters, and sparse activation saves none of that memory;
    2. Large total parameters necessarily mean slow: at high concurrency, many weights are already resident in cache anyway, and sparse models’ per-unit compute cost still wins;
    3. Sparse architecture is always cheaper than dense: at low concurrency or with limited interconnect bandwidth, expert-parallelism communication overhead may cancel the gains;
    4. Context length can be stretched without limit: the KV cache grows linearly with context length and concurrency, and long-context scenarios hit the ceiling first;
    5. Switching models is just switching an interface: differences in routing distribution, output style, and structured-output capabilities all affect downstream business.

    8. Conclusion

    The real value of sparse large models is decoupling “model scale” from “inference cost”: total parameters determine the capability ceiling and the memory barrier, while activated parameters determine compute overhead. The sparse small-spec versions redesigned from the base model are essentially rebalancing this relationship for a lower unit cost.

    For developers, what matters more than “which version to watch” is building your own evaluation methodology: understand the routing mechanism, calculate the resource ledger accurately, run business stress tests, and complete production observability. When you can precisely state “which parameter determines cost in my scenario,” model selection is no longer a matter of chasing parameter numbers, but a rational engineering decision.