Memory-Based Inference for Trillion-Parameter MoE Models: A Practical Analysis
This article examines how a trillion-parameter Mixture-of-Experts (MoE) model can run on system memory when its parameters far exceed accelerator VRAM. Using a real-world case of a 2.8-trillion-parameter model with roughly 104 billion active parameters per token running at about 0.85 tokens per second, it explains why capacity and memory bandwidth—not compute—are the true bottlenecks. Readers will learn the engineering logic behind MoE sparse activation, a quantitative view of memory-bound inference, and practical levers such as quantization, caching, tiering, and batching, along with a general cloud path for moving from a single-machine experiment to a production-grade inference service.
1. The Core Problem Behind the Phenomenon
A practice case circulating in a developer community shows that a hobbyist successfully ran a Mixture-of-Experts (MoE) large model with roughly 2.8 trillion total parameters—activating only about 104 billion parameters per token—on a single high-memory host, reaching an inference speed of about 0.85 tokens per second. That speed is nearly unusable for production-level interaction, but it raises a highly valuable technical question: when model size far exceeds accelerator VRAM, can large-capacity system memory be used to complete “barely usable” inference?
To understand this case, three things must first be clarified: why the MoE architecture can achieve “huge total parameters with controllable per-pass compute,” why VRAM capacity is a hard constraint, and how the memory medium (especially high-bandwidth, high-capacity persistent memory) fits into the inference pipeline.
2. MoE Architecture: The Engineering Meaning of Sparse Activation
A traditional dense model involves all parameters in the forward pass every time it generates a token. The more parameters there are, the greater the compute and memory-access pressure, and both grow roughly linearly.
The MoE architecture works differently. It splits the feed-forward layers into multiple parallel “expert” subnetworks and introduces a gating/routing network (the Router). For each token it processes, the router selects only a few experts to compute, while the rest stay “silent.”
# Illustration: MoE routing and sparse activation (pseudocode)def moe_forward(x):logits = router(x) # compute gating scores for all expertstopk_idx = top_k(logits, k=2) # select only the top-k expertsout = 0for idx in topk_idx:out += gate_weight[idx] * expert[idx](x) # activate only the selected expertsreturn out
Using the numbers from this case: 2.8 trillion total parameters, but only about 104 billion are activated per token. In other words, the compute for a single forward pass roughly corresponds to a hundred-billion-scale dense model, not a 2.8-trillion one. This is exactly the core idea of MoE—trading parameter capacity for compute sparsity—using a very large parameter total to carry knowledge, and sparse activation to control the compute cost of a single pass.
The cost, however, is this: whether activated or not, all expert weights must reside in addressable storage, because routing is dynamic and data-dependent—you cannot predict which experts the next token will hit. This turns the problem from “insufficient compute” directly into “insufficient capacity.”
3. The Real Bottleneck: Capacity and Bandwidth, Not Compute
If the 2.8 trillion parameters are stored in FP16, the weights alone require about 5.6 TB; even with 4-bit quantization, that is close to 1.4 TB. No single accelerator’s VRAM (typically tens to about a hundred GB) can hold it, and even the combined VRAM of a multi-accelerator cluster is often tight. Two natural paths emerge:
- Add accelerators: stack more accelerators to build up VRAM capacity—high cost, high deployment barrier.
- Use memory: use large-capacity system memory to hold the weights, letting the CPU/memory subsystem act as the “weight warehouse.”
This case takes the second path—using far more memory than usual (via a type of high-capacity persistent memory that has since been discontinued) to fit the model. The essential reason it can run is that it shifts the memory-access bottleneck from “VRAM” to “memory bandwidth.”
The key is this: MoE activates only about 3.7% of parameters per pass (104 billion / 2.8 trillion). Each time it generates a token, it only needs to move the weights of those ~104 billion active parameters from memory to the compute units, rather than moving all 2.8 trillion. This is why such a massive model can still run, albeit at a “snail’s pace”—it is not computing incorrectly, it is simply moving data slowly.
4. 0.85 Tokens per Second: A Quantitative Reading of the Trade-off
Speed is determined by the ratio of memory bandwidth to the number of active parameters. A rough order-of-magnitude estimate:
bytes moved per token ≈ active parameters × bytes per parameterif 104B parameters are active with 4-bit quantization → ~1.04e11 × 0.5 ≈ 52 GBif effective memory bandwidth ≈ 40–45 GB/s (realistic under constrained access patterns)→ tokens per second ≈ bandwidth / bytes moved per token ≈ 45 / 52 ≈ 0.85
(The above is an order-of-magnitude illustration to explain the principle, not a precise measurement.)
This result confirms the characteristics of a memory-bound workload: inference speed is almost entirely determined by “how many bytes can be moved per second,” and the compute units spend much of their time waiting for data. This yields several engineering insights:
- Raising bandwidth or lowering the bytes moved per token are the two main levers for accelerating memory-based inference.
- Quantization (lowering bytes per parameter) delivers far greater benefit for memory-based inference than for dense models.
- The lower the activation ratio, the larger the total parameter scale that the same bandwidth can support.
5. Engineering Techniques to Make “Snail-Pace” Inference Usable
To improve the experience under memory-based deployment, you can start from these directions:
- Weight quantization: compress weights to 4 bits or lower, directly and linearly reducing the memory access per token—the most cost-effective optimization in memory-constrained scenarios.
- Tiering and prefetching: based on the historical hit distribution of routing, cache high-frequency experts in a faster storage tier to reduce access to slower media.
- KV cache management: the longer the context, the larger the KV cache footprint, which competes with weights for bandwidth; cache pruning, quantization, or paged management is needed.
- Batching and pipelining: a single request is already memory-bound, so moderate batching can amortize the fixed cost of moving weights, though it remains capped by the memory bandwidth ceiling.
- Precision and activation strategy trade-offs: under the premise of preserving output quality, explore more aggressive sparse activation to further reduce the bytes moved per token.
It is worth emphasizing: these techniques are general model-deployment and inference-optimization ideas. The feasibility of any trillion-scale model on a single high-memory machine ultimately depends on whether the memory access of the active parameters can be absorbed by memory bandwidth within an acceptable latency.
6. From Single-Machine Experiments to Production: A General Cloud Path
Memory-based inference on a single machine is suitable for validation and experimentation, but真正production-ready use still requires a more complete engineering stack. In a general cloud environment, the following capabilities can be combined to upgrade from “it runs” to “it runs reliably”:
- Object storage: stores model weights and versioned archives, supporting reliable persistence and cold-start pulls for large-scale data.
- Container platform: packages the inference service in containers to enable elastic scaling and fast scale-out/scale-in.
- High-speed networking and parallel file systems: carry weight sharding and Expert Parallelism distribution in multi-node scenarios.
- Monitoring, alerting, and logging services: continuously collect metrics such as throughput, latency, and memory-bandwidth utilization to detect bandwidth bottlenecks and access hotspots in time.
- Message queues: decouple request ingestion from the inference backend to smooth traffic peaks and improve overall system throughput.
Taking Baidu AI Cloud as an example, the above capabilities are largely provided as general cloud product categories (such as object storage, container platforms, and logging/monitoring services), which can help teams progressively evolve a large-memory inference solution from the lab into an operable, observable, and scalable production-grade inference service. Whether to adopt a memory-based single-machine solution, a hybrid sharding solution, or an accelerator-cluster solution should be assessed comprehensively based on model scale, latency requirements, and cost budget.
7. Conclusion
The value of this “0.85 tokens per second” case lies not in how fast it runs, but in how clearly it marks a boundary: when model scale far exceeds VRAM capacity, large-capacity memory combined with MoE sparse activation can turn “cannot run” into “runs slowly.” Understanding the nature of being memory-bound, mastering optimization levers such as quantization, caching, tiering, and batching, and combining them with general cloud capabilities for engineering implementation—this is the true methodology for running ultra-large-scale models on constrained hardware. Speed is only the surface; the “capacity–bandwidth–sparsity” triangle of trade-offs behind it is the technical core most worth learning from this kind of practice.
