Optimizing Ultra-Large MoE Models for Inference on Consumer GPUs

    This article examines how expert-level caching, a three-tier storage architecture, asynchronous heterogeneous computing, and speculative decoding can make ultra-large Mixture-of-Experts models practical on consumer hardware. Using a 12 GB GPU and 64 GB of system memory, the approach achieves substantially higher decoding throughput than traditional layer-level offloading, while also exposing trade-offs in memory usage, quantization consistency, performance attribution, and reproducibility.

    As large language models continue to grow exponentially, Mixture-of-Experts (MoE) architectures use sparse activation to expand model capacity without increasing computation proportionally. However, the weights of ultra-large MoE models can easily occupy hundreds of gigabytes, creating major challenges for local deployment.

    Traditional layer-level offloading strategies often move substantial computation to the CPU when a complete layer cannot fit into available GPU memory. This can severely reduce decoding speed. The central challenge is therefore to exploit MoE sparsity as effectively as possible within limited GPU VRAM.

    1. Moving Beyond Coarse-Grained Offloading: Expert-Level Cache Scheduling

    Conventional inference engines typically manage MoE weights at the layer level. If an entire layer cannot fit in VRAM, its computation is executed on the CPU. This coarse-grained strategy overlooks the naturally uneven routing behavior of MoE models.

    A newer scheduling approach manages individual experts across layers rather than placing weights one layer at a time. In a large network with 48 layers and more than 20,000 routed experts, processing a single token can trigger hundreds of expert lookups.

    The system first uses an offline frequency table to preload the most frequently used experts into GPU memory. During inference, it periodically updates decaying usage counters after several rounds of generation. Experts that cool down are evicted, while hotter experts from the same layer are loaded into the cache.

    This creates a dynamically balanced expert cache. Instead of allocating VRAM uniformly, the system directs its limited capacity toward the computation nodes that are most active.

    2. A Three-Tier Storage Architecture for Heterogeneous Execution

    To support expert-level scheduling, the design uses a three-tier storage hierarchy:

    1. GPU VRAM: The 12 GB of VRAM is used aggressively. Approximately 5 GB stores copies of several thousand frequently used experts. The remaining capacity is allocated to attention components, dense weights, speculative decoding heads, and preallocated KV cache space for very long contexts. Frequently accessed computation remains in this tier whenever possible.

    2. Host memory: Within 64 GB of system memory, a complete expert pool is pinned as a second copy. When an expert is not found in VRAM, the system searches host memory and executes the required computation through the CPU.

    3. NVMe SSD: Large sharded tables are stored on a high-speed NVMe drive. Memory mapping enables lazy loading without occupying GPU VRAM or consuming the corresponding portion of host memory.

    The key assumption behind this architecture is the uneven distribution of routing demand. In testing, approximately 13% of experts handled three-quarters of all lookups, allowing the GPU-side expert cache to achieve a hit rate of up to 73.8%.

    This naturally occurring long-tail distribution is what makes it possible for a relatively small amount of VRAM to support a much larger model.

    3. Hiding Latency with Asynchronous Pipelining and Speculative Decoding

    A cache hit rate of 75% does not mean that 75% of tokens can be processed without CPU involvement. If each inference step triggers 480 lookups, a 75% hit rate still leaves roughly 120 lookups that must fall back to host memory and CPU execution.

    To reduce the impact of these cross-device accesses, the system introduces asynchronous PCIe stream overlap. GPU computation, CPU computation, and PCIe transfers run in parallel. The design also reduces the amount of weight data transferred per token to approximately one-quarter of the original volume.

    This pipeline-style execution helps hide memory-access latency and keeps the GPU highly utilized.

    In addition, speculative decoding heads, such as those used for multi-token prediction, remain resident in GPU memory. A single forward pass can verify up to three draft tokens. Combined with the draft acceptance rate, this further reduces the average computation time per generated token.

    4. Performance Evaluation and Stress Testing

    On a single system equipped with a 12 GB consumer GPU and 64 GB of DDR5 memory, the approach demonstrated substantial performance potential.

    With prompts of approximately 60,000 tokens, out-of-the-box decoding throughput reached 53.2 tokens per second, while prompt processing exceeded 2,000 tokens per second. After applying full-precision sharding optimizations and a specific sampling strategy, sustained programming workloads achieved up to 74.8 tokens per second, with a draft acceptance rate of 78.2%.

    By comparison, the best decoding speed of a traditional layer-level offloading approach on the same hardware was only 20.8 tokens per second. Even with speculative decoding and NVMe prefetching, throughput reached only approximately 27 tokens per second.

    For long-context scanning, decoding speed remained broadly stable between 52 and 60 tokens per second as prompt length increased from short inputs to ultra-long contexts. This indicates strong stability under varying context lengths.

    5. Costs and Limitations

    Despite the impressive results, this approach places considerable pressure on the available hardware and has several important limitations:

    1. Severe memory pressure: Actual host-memory usage exceeded 50 GB, leaving only a little more than 10 GB for the operating system and desktop environment. This can significantly affect the overall usability of a consumer system.

    2. Inconsistent quantization: Different components used different quantization precisions. The primary test used lower-bit quantization, while the comparison setup used higher-bit quantization. Because there was no matched perplexity comparison, the impact on model quality remains difficult to assess.

    3. Difficult attribution of individual components: Speculative decoding could not be completely disabled, making it difficult to determine how much of the performance improvement came specifically from the expert-cache strategy.

    4. Limited reproducibility: The reported results came from a self-test conducted in one specific environment. Broader third-party replication is still needed.

    6. Conclusion and Broader Implications

    This implementation demonstrates that expert-level cache scheduling, multi-tier storage coordination, and asynchronous computation overlap can enable efficient inference of ultra-large MoE models even with limited GPU memory.

    The same principles may be valuable in cloud-native and edge-computing environments, where heterogeneous memory management and workload scheduling can reduce deployment costs and improve hardware utilization.

    Future work could focus on consistent quantization across components, improving cache-hit rates, and refining communication protocols for heterogeneous computing. These directions may help bring large-scale models to a broader range of hardware configurations.