Selecting Open-Weight Models: Calculating MoE Memory and Single-GPU Deployment for Huge Contexts
This article explains how to estimate memory and compute requirements for open-weight large language models, focusing on the difference between Mixture-of-Experts (MoE) and dense architectures. Using two anonymized models—a 2.8-trillion-parameter MoE and a 65-billion-parameter dense model—it walks through weight storage, KV cache sizing, communication overhead, and single-GPU feasibility. It also provides practical guidelines for verifying vendor benchmarks and assessing whether a model can actually run within your hardware constraints, especially for million-token contexts.
Introduction: Two Model Profiles
The original source does not reveal the model names, so we refer to them as Model A (a very large MoE) and Model B (a dense model). Before deployment or selection, always verify the official names, release dates, and licenses against vendor announcements.
| Item | Model A (Large MoE) | Model B (Dense) |
|---|---|---|
| Total parameters | ~2.8 trillion (vendor-reported) | ~65 billion (vendor-reported) |
| Active parameters per token | ~104 billion | All parameters active |
| Context length | ~1 million tokens | Million-token class (vendor-reported) |
| Pre-training data | Not mentioned | ~12 trillion tokens (vendor-reported) |
| License | Not mentioned | Permissive open-source license, exact name not disclosed |
| Deployment form | Multi-node, multi-GPU required | 4-bit quantized can run on a single 128 GB VRAM accelerator (vendor-reported) |
| Self-reported benchmarks | 16 of 37 benchmarks beat a proprietary flagship model; list not disclosed | Terminal CLI agent benchmark score 80.5 (vendor-reported) |
All figures above come from the vendors; this article does not independently verify them. The calculations below use public formulas and can be replicated by readers.
Large MoE: Total Parameters vs. Active Parameters
Total parameters determine the storage footprint of the weights. Active parameters determine the computation per token, which directly affects latency and throughput.
For Model A, the active ratio is 104 / 2800 ≈ 3.7%. Using the common rule of thumb, forward computation per token is roughly 2 × active parameters, i.e., 2.08 × 10¹¹ FLOPs. This is close to the compute of a 10-billion-parameter dense model, but the weight storage must be based on total parameters, requiring far more VRAM than a dense model with equivalent compute.
Weight storage = number of parameters × bits per parameter ÷ 8. The function below calculates only the weights:
def weight_only_gb(params_billion, bits):"""Weight-only storage, excluding quantization scales, activations, KV cache, and framework overhead."""return params_billion * 1e9 * bits / 8 / 1e9print(weight_only_gb(2800, 16)) # 5600.0 GBprint(weight_only_gb(2800, 4)) # 1400.0 GBprint(weight_only_gb(65, 4)) # 32.5 GB
Even with 4-bit quantization, Model A’s weights are about 1.4 TB, far exceeding any single-machine VRAM capacity. Multi-node parallelism is mandatory, with different experts distributed across different devices. In practice, additional VRAM (typically 10–30% of the weight size, depending on the inference framework and quantization format) should be reserved.
MoE routers select a few experts per token. When experts are spread across multiple GPUs, tokens must be routed to the device holding the target expert, then the results are returned. A rough single-layer communication estimate is:
Communication ≈ batch tokens × hidden dimension × bytes per element × top-k × cross-device ratio × 2 × number of layers
The following hypothetical parameters (not corresponding to Model A) illustrate: 4096 tokens per batch, hidden dimension 8192, FP16, top-k=8, cross-device ratio 0.75, 60 layers. This gives ~0.8 GB per layer and ~48 GB total for all layers. Communication grows with node count and batch size, potentially offsetting the compute savings from sparse activation.
Expert load imbalance is also a concern. If too many tokens route to a small set of experts, those devices become bottlenecks, degrading tail latency. A common solution is a capacity factor: each expert’s capacity = capacity factor × batch tokens × top-k ÷ number of experts. In the example above, each expert receives an average of 128 tokens; with a capacity factor of 1.25, each expert can hold up to 160 tokens. Excess tokens may be dropped or redirected, requiring a trade-off between accuracy and throughput.
Million-Token Context: KV Cache as the Primary VRAM Pressure
During autoregressive inference, the model must store the key and value vectors for every previous token at every layer—this is the KV cache. The formula is:
KV cache bytes = 2 × layers × KV heads × head dimension × sequence length × bytes per element
def kv_cache_gb(layers, kv_heads, head_dim, seq_len, bytes_per_elem=2):per_token = 2 * layers * kv_heads * head_dim * bytes_per_elemreturn per_token * seq_len / 1e9print(kv_cache_gb(80, 8, 128, 1_000_000)) # 327.68 GB
The example assumes 80 layers, 8 KV heads, head dimension 128, and FP16 precision. Each token requires 2 × 80 × 8 × 128 × 2 = 327,680 bytes (~0.328 MB). For 1 million tokens, this is about 328 GB ≈ 305 GiB. This figure covers only the KV cache; weights, activations, and memory fragmentation are not included.
This demonstrates that a nominal million-token context does not mean a single GPU can handle it. Actual memory needs depend on the model’s layer count and KV head count. You must read these from the model configuration file and not assume generic values.
Common strategies for long-context handling include:
- Input truncation – Most tasks do not need a full context; keep only the relevant history.
- Segment summarization and hierarchical memory – Compress early dialogs or documents into summaries, preserving key information.
- KV cache quantization – For example, storing KV cache in 8-bit reduces VRAM usage, but must be tested for output quality impact.
- Prefix caching – Reuse repeated system prompts or document prefixes to avoid recomputation and save VRAM.
4-bit Single-GPU Deployment: Weights Fit, but Long Context May Not
For Model B, 65B parameters at 4-bit weight quantization require about 32.5 GB, so a 128 GB accelerator can hold the weights. The remaining VRAM is used for KV cache and runtime overhead. The supported context length can be estimated as:
Supported tokens ≈ (available VRAM − weights − runtime overhead) ÷ KV cache bytes per token
Assuming 128 GB available VRAM, subtract 32.5 GB of weights and 20 GB of runtime overhead, leaving about 75.5 GB. Using the hypothetical architecture from the previous section (0.328 MB/token), this supports around 230K tokens—far below the million-token level. The actual layer count and KV head count for Model B are not disclosed in the original text; real results depend on its configuration file.
Before single-GPU deployment, verify the following items:
- Number of layers, KV heads, and head dimension from the model config to compute KV cache.
- Whether the chosen quantization format and inference framework support the model.
- Peak VRAM at the target context length, measured with a real workload rather than theoretical limits.
- Time to first token and tokens per second, to see if they meet business requirements.
- Whether the vendor’s benchmark scores are based on the same quantized version. The original text does not state this.
How to Re-evaluate Vendor Benchmarks
Vendor-published scores often use specific prompt, tool, and sampling settings, so direct horizontal comparisons can be misleading. To verify, follow these steps:
- Record the model version, quantization precision, inference framework version, and tool versions.
- Fix the prompt template, system prompt, and sampling parameters (temperature, top_p, max output tokens) and document them publicly.
- For tasks with randomness, run multiple times and report mean and standard deviation, or use the pass@k metric.
- Check for training data contamination. Use rewritten or new questions and compare the score gap with original questions.
- Compare with independent leaderboards and run small-scale tests on your own business samples.
- Track costs in parallel: input/output token count, latency, and price per call.
- For claims like “exceeds model X in Y benchmarks,” ask for the full list of benchmark names, comparison model, scores, and evaluation dates.
Risk Notes
- Self-reported scores may use different prompts, tools, or compute settings, so they cannot be directly compared with other models.
- The memory estimates in this article are lower bounds. Production environments need headroom for peak load, cache, and fault tolerance.
- “Permissive open-source license” is a broad descriptor. Check the license text for commercial use and redistribution rights.
- Multi-node MoE deployment has higher operational complexity and failure surface than single-GPU dense models. Assess your team’s operational capability.
- If your business requires long context and you cannot accept multi-node deployment, first calculate the achievable context length on a single GPU using the real architecture parameters, then make the selection decision.
