MoE vs. Dense LLM Selection: Architecture, VRAM, and Deployment Considerations

    Choosing between Mixture-of-Experts (MoE) and dense large language models depends on your workload, hardware, and deployment goals. This article explains how MoE and dense architectures differ in parameter usage and compute cost, how to estimate VRAM requirements including KV Cache for long-context inference, and provides a practical checklist and infrastructure guidance for selecting and deploying the right model for your use case.

    The Short Answer

    MoE and dense models solve different problems. MoE models suit scenarios that prioritize capability ceilings and high concurrency. Dense models suit scenarios with limited hardware or a need for private, on-premise deployment. Whichever you choose, calculate VRAM requirements first, then consider context length.

    This article makes no claims about specific models’ parameter sizes or benchmark scores. When making a selection, rely on official technical reports and your own test results.

    1. MoE Models: Large Total Parameters, Small Active Parameters

    A MoE (Mixture-of-Experts) model consists of multiple expert subnetworks and a gating network. The gating network selects only a few experts to process each token, while the remaining experts stay idle for that step. As a result, the compute per token depends only on the activated parameters, not the total parameter count.

    This design offers two benefits: the model can hold more knowledge, while the compute cost of a single inference remains close to that of a dense model with a similar number of active parameters.

    A common misconception arises during deployment. Because the gating happens at runtime, all expert weights must be quickly accessible at all times. MoE models occupy VRAM based on their total parameters, not their active parameters. A MoE model with a very large total parameter count needs multi-GPU or high-VRAM nodes to load, even if only a small portion is used at any moment.

    2. Dense Models: Every Token Uses All Parameters

    A dense model invokes all its parameters to generate every token, so compute scales roughly linearly with parameter count. Its architecture is simple and deployment is well established—mid-sized models can run on a single GPU in a single machine.

    Take a 65B-parameter dense model as an example. Storing weights in FP16 requires about 130GB of VRAM (65 × 2 bytes). With 4-bit quantization, weights drop to roughly 35GB. Actual usage will be slightly higher due to the quantization scheme and runtime overhead.

    3. The Key to Single-GPU Deployment: KV Cache, Not Just Weights

    Calculating only model weight VRAM is often insufficient for long-context scenarios. During inference, the model stores a Key and Value for every processed token—the KV Cache. Its size can be estimated with this formula:

    KV Cache bytes per token = 2 × number of layers × number of KV heads × head dimension × bytes per element

    Assume a configuration of 80 layers, 8 KV heads (using GQA), head dimension of 128, stored in FP16. Each token takes about 320KB, so a 1M-token context would require roughly 320GB of KV Cache—far beyond a single 128GB GPU. With 8-bit KV quantization, it’s about 160GB, still too much. With 4-bit quantization, it drops to about 80GB, finally within feasible range—but this still excludes activations and runtime overhead.

    Therefore, the claim that “a single 128GB GPU can run a 65B 4-bit model” only holds for short or medium contexts. A 1M-token context typically requires one or more of the following:

    • More aggressive KV Cache compression;
    • Multiple GPUs sharing weights and KV Cache, e.g., tensor parallelism or sequence parallelism;
    • Accepting lower concurrency and throughput.

    PagedAttention reduces memory fragmentation and pre-allocation waste, but it does not reduce the total KV Cache required for a given number of tokens. It complements quantization and multi-GPU parallelism—it cannot replace them.

    4. MoE vs. Dense Models at a Glance

    Dimension Very Large MoE Model Cost-Effective Dense Model
    Parameter organization Large total parameters; each token activates only some experts Every token uses all parameters
    Compute per token Depends on active parameters Depends on total parameters
    Deployment VRAM Loaded by total parameters; usually multi-GPU or high-VRAM nodes Can be compressed to a single GPU with quantization
    Long context Limited by KV Cache; usually needs multiple GPUs Limited by KV Cache; single GPU requires shorter context
    Operational complexity Requires expert parallelism and similar configurations; more complex Simple architecture; straightforward deployment
    Best suited for Complex agents, high concurrency, capability-first Private deployment, edge scenarios, limited budgets

    5. Selection Checklist

    1. First define your real context requirements. Many workloads don’t need 1M contexts—estimate based on actual document lengths. A 1M-token context roughly corresponds to several hundred thousand to one million Chinese characters, depending on the tokenizer.
    2. Use the formula above to compute KV Cache per token for your target model, then multiply by concurrency and context length to get total VRAM needs.
    3. Confirm whether the license permits commercial use and private deployment.
    4. Run evaluations on your own data. Public benchmark scores only reflect performance on specific test sets—task effectiveness must be validated yourself.
    5. Compare accuracy before and after quantization, focusing on mathematical reasoning, code generation, and structured output.
    6. During load testing, record time-to-first-token, tokens per second, peak VRAM usage, and tail latency under high concurrency.

    6. Common Infrastructure Requirements for Deployment

    Whichever model type you choose, an inference service typically needs:

    • Elastic GPU resources that scale with load;
    • Object storage for model weights, paired with local caching to speed up loading;
    • Monitoring of GPU utilization, VRAM usage, inference latency, and error rates, with alerting configured.

    7. Summary

    Open-source models iterate quickly and options keep growing, but the right selection order hasn’t changed. Calculate VRAM first, then review context and concurrency requirements, and finally validate with your own data. Following this order helps you quickly find a model suitable for enterprise AI agent applications.