Dense vs. Sparse-Activated LLMs for Local Inference: VRAM and Throughput Trade-offs

    This article examines how dense and sparse-activated large language models trade off VRAM, prefilling throughput, and decoding throughput under a roughly 20 GB VRAM budget and a ~70K context length. It explains why activation parameters affect compute while total parameters set the weight-VRAM floor, breaks down an observed ~18 GB peak, and discusses KV cache, runtime buffers, and long-context effects. It also provides deployment selection guidance and an optimization checklist for benchmarking and stabilizing local inference.

    1. Background: Local Inference Is Not Only About Model Quality

    When deploying large language models locally or on a single machine, developers usually care about three metrics at the same time: quality, VRAM usage, and generation speed. Quality determines task completion, VRAM determines whether the model can run on existing hardware, and speed directly affects the interaction experience. Many model-selection discussions fall into the single dimension of “larger parameter count means better quality,” but in real engineering, model architecture, quantization method, context length, batching strategy, and runtime implementation all significantly change final performance.

    The core phenomenon in the original discussion can be abstracted into a general technical question: under the same VRAM budget of about 20 GB and a context length of about 70K, a smaller dense model and a sparse-activated model with a larger total parameter count may use similar VRAM, but their prefilling and decoding throughput can differ noticeably. In an anonymized comparative observation, the dense model had a peak VRAM usage of about 18 GB, prefilling throughput of about 800 tokens/s, and decoding throughput of about 20 tokens/s. Compared with the sparse-activated model that had a larger total parameter count, its prefilling throughput was about half, and its decoding throughput was about one-third. This observation represents only specific experimental conditions and should not be directly extrapolated to all models and hardware, but the trade-off it reveals has general reference value.

    2. Fundamental Differences Between Dense and Sparse-Activated Models

    2.1 Dense Models: Every Token Passes Through All Parameters

    During inference, a dense model typically involves all weights in the model for each token’s computation. Its advantages are a stable computation path, relatively direct implementation, and simple runtime scheduling. Under the same training quality, it often provides more stable expressive capability. The cost is higher computation and weight reads per token. Especially in the autoregressive decoding stage, the model generates token by token, computation is difficult to fully parallelize, and memory bandwidth and compute resources can become bottlenecks.

    2.2 Sparse-Activated Models: Large Total Parameters, but Only Some Parameters Are Activated per Token

    Sparse-activated models often use structures such as mixture-of-experts. The total parameter scale is larger, but each token is routed only to some experts, reducing per-token computation. This can provide higher throughput in both the prefilling and decoding stages. However, sparse activation does not mean “VRAM is calculated by activated parameters.” Model weights still need to reside in memory or be swapped in quickly, and total parameters determine the lower bound of weight VRAM. The routing mechanism also introduces extra overhead, such as uneven expert load, routing decisions, communication, or scheduling costs.

    2.3 Key Conclusion: Activated Parameters Affect Compute, Total Parameters Affect VRAM

    Therefore, VRAM and speed must be analyzed separately. VRAM is mainly composed of quantized weights, KV cache, runtime temporary buffers, and framework overhead. Speed is affected by the number of activated parameters, memory bandwidth, batch size, sequence length, and runtime optimization. A sparse-activated model with larger total parameters can use VRAM similar to that of a smaller dense model while achieving faster decoding throughput.

    3. Breakdown of an ~18 GB VRAM Peak

    Observing a peak usage of about 18 GB on a 20 GB accelerator is not surprising. VRAM usually consists of the following parts:

    1. Quantized weights: 4-bit quantization can significantly reduce weight size, but in actual runtime there are often dequantization, mixed-precision computation, and extra buffers, so it cannot simply be calculated from theoretical values.
    2. KV cache: KV cache grows noticeably with long contexts. A 70K context forces the runtime to retain a large amount of historical key-value state, making it an important source of peak VRAM.
    3. Runtime buffers: Attention computation, temporary tensors, batching queues, logs, and the framework itself all consume VRAM.
    4. Safety margin: If the peak reaches 18 GB, about 2 GB remains. Longer contexts, larger batches, or fragmentation can easily cause OOM.

    The following pseudocode can be used to estimate KV cache size. Actual values must account for the number of layers, KV heads, head dimension, and data type:

    1. def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem, overhead=1.2):
    2. # 2 accounts for key and value
    3. base = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
    4. return int(base * overhead)
    5. # Example: formula structure only; fill parameters according to the actual model configuration.
    6. # size = kv_cache_bytes(layers=..., kv_heads=..., head_dim=..., seq_len=70000, batch=1, bytes_per_elem=2)

    Formula notes: sequence length and batch scale KV cache linearly. KV cache quantization, paged management, and prefix caching can reduce peak usage, but they increase implementation complexity. During deployment, reserve 20%–30% VRAM headroom instead of running close to the limit.

    4. Why Decoding Speed Differences Are More Obvious

    4.1 Prefilling Stage: High Parallelism, Affected by Both Compute and Bandwidth

    The prefilling stage processes a longer input at once, and matrix operations have high parallelism, so throughput is usually higher. The anonymized observation of about 800 tokens/s prefilling throughput shows that input processing at a 70K context is still acceptable, but it is about half that of the sparse-activated model, indicating that the dense model’s full-parameter computation still has a clear cost for long inputs.

    4.2 Decoding Stage: Token-by-Token Generation Makes Memory Bandwidth Critical

    In the decoding stage, each generated token requires reading weights and maintaining the KV cache. Computation parallelism is low, and memory bandwidth and memory access efficiency are critical. A dense model activates all parameters for each token, so decoding computation is larger. A sparse-activated model activates only some experts, so per-token computation is lower, and decoding throughput can reach several times that of a dense model. In the observation, the dense model decoded at about 20 tokens/s, only about one-third that of the sparse-activated model, which directly affects interaction experiences such as chat and code completion.

    4.3 Long Contexts Amplify the Differences

    At a 70K context, both KV cache usage and attention computation increase significantly. Lower prefilling throughput, higher decoding latency, and a VRAM peak approaching the limit constrain one another. If batch size or context is increased further, a 20 GB card can easily move from “it runs” to “it is unstable.”

    5. Deployment Selection: Engineering Trade-offs Between Quality and Experience

    Scenario Priority metrics Suggested direction Risks
    Real-time interaction, low latency Decoding throughput, time to first token Evaluate sparse-activated models with faster decoding or smaller dense models first Routing overhead, uneven expert load
    High-quality offline tasks Quality, batch throughput Slower decoding may be acceptable; choose dense models with more stable expressive capability Long-context VRAM peak
    Long-document analysis Context capability, KV cache Trim context as needed; enable KV cache optimizations OOM, prefilling latency
    Multi-user serving Concurrency, VRAM headroom Continuous batching, prefix caching, monitoring and alerting Batching amplifies VRAM usage

    If there is no especially strong quality requirement, speed is very important to the experience. For a single-card 20 GB-class environment, it is advisable to first run a small-scale stress test with target business prompts rather than looking only at parameter count or theoretical compute. The stress test should record at least: time to first token, per-token output latency, output tokens/s, prefilling tokens/s, peak VRAM, OOM boundary, and long-context stability.

    6. Optimization Checklist

    1. Set context length as needed: Do not enable the maximum context by default. The KV cache difference between 70K and 8K is huge.
    2. KV cache quantization and paged management: Reduce cache usage with acceptable precision loss and reduce fragmentation.
    3. Prefix caching: Reuse cache for fixed system prompts and repeated documents to reduce prefilling cost.
    4. Continuous batching: Improve throughput, but monitor peak VRAM to avoid OOM caused by batching amplification.
    5. Quantization strategy evaluation: 4-bit weights reduce weight VRAM, but KV cache and runtime buffers still need separate accounting.
    6. Expert load monitoring: Sparse-activated models require attention to routing distribution to avoid a few experts becoming bottlenecks.
    7. Build reproducible benchmarks: Fix input length, output length, batch size, quantization, and context; record TTFT, TPOT, throughput, and peak VRAM.

    7. Conclusion

    Model selection between dense and sparse-activated models is not simply a question of “which is stronger,” but a three-way trade-off among quality, VRAM, and speed. Dense models have a stable computation path, but all parameters participate for each token, so decoding throughput may be noticeably lower. Sparse-activated models trade fewer activated parameters for higher throughput, but total parameters still determine the VRAM lower bound, and routing and load balancing also require engineering work. Under the constraints of 20 GB-class VRAM and a 70K context, a peak of about 18 GB means limited headroom, so deployment must be stress-tested with real business workloads beforehand. For scenarios sensitive to interaction experience, prioritize decoding throughput and VRAM headroom. For scenarios with higher quality requirements, then evaluate whether slower generation speed is acceptable.