Deploying Large Language Models Locally: MoE Architecture, Resource Planning, and Deployment Guide

    This article explains why parameter count alone is not enough when deploying large language models locally. It introduces the core mechanics of Mixture of Experts (MoE), including routing, experts, and sparse activation, and clarifies how to interpret naming conventions such as 35B A3B. It emphasizes the key resource rule for MoE models: memory capacity is determined by total parameters, while inference speed is driven by active parameters. The article then covers practical deployment factors such as memory budgeting, quantization, inference framework support, and hardware selection, and offers actionable guidance for choosing and deploying MoE models under local constraints.

    1. The Real Bottlenecks of Local Deployment

    When deploying large language models in local environments, developers most often run into three constraints:

    • The capacity of available VRAM or system memory
    • The compute ceiling of the hardware
    • End-to-end inference latency

    Many teams focus only on “parameter count” during selection. As a result, they may buy hardware that cannot run the model, or find that actual performance is far below expectations. To understand this gap, it is important to recognize that parameter count alone does not determine resource requirements. The model’s compute architecture is equally important.

    This is why the Mixture of Experts (MoE) architecture is worth attention. It changes the traditional intuition that increasing parameter count leads to a linear increase in resource cost.

    2. Core MoE Principles: Sparse Activation and Expert Division of Labor

    A useful analogy is a company with many specialists. When a task arrives, not every specialist is involved. Instead, the system selects a small number of the most suitable experts to handle it.

    From an engineering perspective, a typical MoE structure usually contains three key components:

    • Gating / router network: Scores each input token and decides which experts should process it. Usually, only the Top-k experts are selected, for example k=1 or k=2.
    • Multiple parallel experts: These are independent feed-forward network submodules that learn different patterns and feature representations.
    • Sparse activation mechanism: Because each token passes through only a few selected experts, the number of parameters actually used in computation is much smaller than the model’s total parameter count.

    The key concept is sparsity. Unlike a dense model, where every input flows through all parameters, MoE separates storage capacity from the scale of computation for a single forward pass.

    3. Understanding Model Naming: Total Parameters vs Active Parameters

    MoE models are often described using a compact notation such as 35B A3B or 26B A4B. This notation is not tied to one specific product. It is a common convention for expressing model scale:

    • First part (for example, 35B): Represents the total number of parameters in the model. In this case, all weights together add up to about 35 billion.
    • Second part (for example, A3B): The letter A stands for Active. It indicates that about 3 billion parameters are actually used in a single forward pass.

    For example, 35B A3B means the model stores about 35 billion parameters in total, but when processing one token, only about 3 billion parameters are activated for matrix operations. This is the main reason MoE models can run relatively fast: compute cost is determined by active parameters, not total parameters.

    4. MoE vs Dense Models: Fundamental Resource Differences

    Understanding this distinction is essential for correct resource planning:

    Dimension Dense model MoE model
    Parameters used per computation All parameters participate Only selected Top-k experts participate
    VRAM / memory usage Determined by total parameters Also determined by total parameters
    Inference speed / compute requirement Proportional to total parameters Proportional to active parameters

    The most important conclusion is: memory capacity is determined by total parameters, while inference speed is determined by active parameters.

    This means MoE is a structure that trades memory capacity for speed. It can keep compute cost relatively low during a single inference, resulting in faster response times. However, the tradeoff is that all expert weights must be loaded into VRAM or memory, because routing may select any expert at any time. Therefore, the VRAM requirement of an MoE model is usually based on its total parameters rather than its active parameters.

    5. Key Considerations for Deploying MoE Models Locally

    1. VRAM / Memory Budget: Estimate Based on Total Parameters

    Before deployment, estimate VRAM requirements based on total parameters. A rough formula is:

    VRAM requirement ≈ total parameters × bytes per parameter + additional overhead such as KV Cache.

    For example, even if a model with about 35B total parameters has a small number of active parameters, its weight footprint still needs to be planned around 35B parameters.

    2. Quantization: An Effective Way to Reduce Memory Usage

    When the available VRAM is not enough to hold full-precision weights such as FP16 or BF16, quantization becomes a common approach. By converting weights into lower-bit representations, such as 8-bit or 4-bit, memory usage can be significantly reduced.

    However, quantization may introduce some accuracy loss while saving memory. The tradeoff should be evaluated against the accuracy requirements of the application, and the result should be validated on real datasets.

    3. Inference Framework and Operator Support

    MoE’s dynamic routing places additional requirements on the inference framework. The framework needs to support expert dispatch, sparse computation, and memory scheduling efficiently. During selection, it is important to confirm how well the inference framework natively supports MoE structures. This directly affects throughput and latency. Without dedicated optimization, the speed advantages of MoE may not be fully realized.

    4. Hardware Selection: Prioritize Memory Capacity, Consider Bandwidth

    • Memory capacity is a hard requirement. It must be enough to hold all expert weights.
    • Memory bandwidth has a significant impact on inference speed, especially during decoding, which is often bandwidth-intensive.
    • For consumer or workstation scenarios, it is necessary to evaluate single-GPU or multi-GPU configurations and consider framework support for multi-GPU tensor parallelism.

    6. Practical Recommendations for Model Selection and Deployment

    1. Define constraints before choosing a model: First estimate the VRAM limit of available hardware, target latency, and expected concurrency. Then determine the total parameter scale that can be supported.
    2. Estimate speed using active parameters and memory using total parameters: This is the most easily confused and most critical step in MoE planning.
    3. Prefer a combination of quantization and inference frameworks with native MoE support: This can help balance speed and capacity when memory is limited.
    4. Run small-scale benchmarks: Use real application samples to measure first-token latency, throughput, and peak VRAM usage. This helps avoid the gap between theoretical parameters and actual performance.
    5. Reserve monitoring and elasticity capabilities: In production deployments, container platforms can be used for environment isolation, while monitoring and alerts can continuously track memory, latency, and throughput metrics. This helps with capacity planning and troubleshooting.

    7. Conclusion

    The difficulty of deploying large language models locally is often not whether the model can be loaded, but whether developers understand the real impact of model architecture on resources. MoE decouples storage capacity from compute scale through sparse activation. This allows a model to maintain a large total parameter count while achieving faster single-pass inference. By following the main rule “memory is determined by total parameters, speed is determined by active parameters,” and combining it with appropriate quantization, inference frameworks, and hardware choices, developers can make more reliable and cost-effective model selection and deployment decisions under local resource constraints.