MoE Long-Context Model Deployment: Parameter Budgets, KV Cache, and Cost Planning
This article explains how to plan deployment for long-context MoE models by separating total parameters, active parameters, and context length; estimating weight and KV cache memory; accounting for expert-parallel communication and load balancing; and validating deployments with task-specific evaluation and regression tests. It also compares long-context input with RAG and provides a pre-launch checklist and key references.
MoE Long-Context Model Deployment: Parameter Budgets, KV Cache, and Cost Planning
1. Separate the Three Parameter Scopes First
Large-model announcements commonly cite three numbers: total parameters, active parameters, and context length. They answer different questions.
Total parameters are the total volume of model weights, which determines storage space and the scale of weight loading. Active parameters are the number of parameters that actually participate in the forward computation for a single token. They mainly affect compute volume and per-token inference cost. Context length is the maximum number of tokens a single request can accept. It determines the upper limit of input material capacity, but it does not directly equal memory or compute requirements.
For example, assume a model has 100B total parameters and activates 20B parameters per token. It must store weights for 100B parameters, but the scale of a single computation step is close to that of a dense 20B-parameter model. Both numbers must be calculated clearly; looking at only one of them is not enough.
2. What MoE and Sparse Attention Each Solve
MoE splits the feed-forward network into multiple experts, and a router selects a small number of experts for each token. Its benefit is to expand parameter capacity while controlling per-token computation. The cost is that the weights of all experts must still be placed in an accessible location, and routing results also introduce communication overhead and load imbalance. Shazeer et al. proposed the sparsely gated MoE layer in 2017, and Fedus et al. proposed Switch Transformer in 2021. These two works are common starting points for understanding this approach.
Sparse attention addresses the problem that attention computation grows with sequence length. It allows each query to attend to only part of the historical tokens. Common methods include local windows, global tokens, and content-based selection. Work by Child et al. (2019) and Beltagy et al. (2020) are earlier representative examples.
There is a boundary here that is often overlooked. Sparse attention reduces attention computation and some memory access. The key-value cache (KV cache) for historical tokens usually still needs to be kept in GPU memory or other storage unless it is combined with quantization, compression, or offloading strategies. Therefore, even when sparse attention is used, memory planning must include the KV cache.
3. How to Estimate Deployment Resources
Weight footprint is calculated as parameter count multiplied by bytes per parameter. For a model with 100B total parameters, BF16 uses 2 bytes per parameter, so the weights are about 200 GB; FP8 uses 1 byte per parameter, so the weights are about 100 GB. Actual deployment also requires activation values, communication buffers, and runtime overhead.
The KV cache estimation formula is:
2 × layers × KV heads × head dimension × sequence length × bytes per element
Take 60 layers, 8 KV heads, head dimension 128, BF16 precision, and a single request of 1M tokens as an example. The calculation is 2 × 60 × 8 × 128 × 1,000,000 × 2 bytes, which is about 246 GB in decimal units. If the model uses a compressed attention mechanism such as MLA, the value will be significantly different, and the official implementation should be used as the reference.
| Item | Calculation | Example value |
|---|---|---|
| Weights (BF16) | Parameter count × 2 bytes | About 200 GB |
| Weights (FP8) | Parameter count × 1 byte | About 100 GB |
| Single-request KV cache (1M tokens) | Calculated using the formula above | About 246 GB |
| Activation values and buffers | Determined by stress testing based on batch size and sequence length | Requires measurement |
Concurrent requests cause the KV cache to grow linearly with the number of requests. Therefore, context length and concurrency must be planned together; testing only whether a single request can run is not enough.
4. Communication and Load Balancing
With expert parallelism, tokens need to be dispatched and combined across devices. Communication volume grows with the number of tokens, the number of activated experts, and the hidden dimension. If inter-node bandwidth is insufficient, the time spent on all-to-all communication may offset the time saved by sparse computation.
During deployment, the following metrics should be monitored continuously: the proportion of communication time on each device, the number of tokens received by each expert, the drop rate, and tail latency. If the busiest expert remains significantly above average load for a long time, the routing strategy or capacity factor needs to be adjusted. Specific thresholds should be determined by the team during stress testing.
5. Four Steps for Evaluation and Rollout
Step one: build a task set. Extract a certain number of samples from real business scenarios, covering code, documents, tables, and other types, and annotate standard answers or judgment rules. The number of samples depends on business risk; critical scenarios should have more samples.
Step two: define metrics. Common metrics include key-field accuracy, citation traceability rate, time to first token, end-to-end latency, cost per task, and long-text recall.
Step three: set acceptance thresholds. Thresholds should be determined jointly by the business side and the technical team, such as the lower bound for key-field accuracy and the upper bound for timeout rate. Thresholds should be documented before the pilot begins to avoid adjusting standards after the fact.
Step four: stress testing and regression. Measure throughput and tail latency at the target concurrency. Each time the model version, quantization method, or prompt is changed, rerun the same task set.
Long-context effectiveness also needs to be tested separately. Liu et al. (2023) found that model performance often degrades when relevant information is located in the middle of a long input. Therefore, the test set should include samples where key information appears in different positions.
6. Choosing Between Long Context and Retrieval-Augmented Generation
When the total document volume is much smaller than the context limit and the task requires cross-paragraph reasoning, placing documents directly into a long context is usually simpler. When the total document volume is large, updates are frequent, or access needs to be controlled by permissions, retrieval-augmented generation (RAG) is more suitable. The retrieval-augmented generation approach proposed by Lewis et al. (2020) is a common baseline.
The two can also be combined. First use retrieval to narrow the candidate range, then place relevant passages into the long context and let the model perform summarization and comparison. This approach is usually more controllable and less costly than inserting all documents into the context at once.
7. Risks and Checklist
Before launch, confirm the following items one by one:
- Memory planning includes weights, KV cache, and runtime overhead, rather than estimating only from active parameters.
- Benchmark scores have been validated with the business task set and are not treated as directly equivalent to business results.
- Generated code has been executed and tested, and extracted facts have been checked against the source text.
- The model version, inference framework, quantization method, and hardware configuration are recorded to make issues reproducible.
- Long-context scenarios have been tested for recall when key information appears in different positions.
8. References
- Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, 2017.
- Fedus et al., Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, 2021.
- Child et al., Generating Long Sequences with Sparse Transformers, 2019.
- Beltagy et al., Longformer: The Long-Document Transformer, 2020.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023.
For the parameters, layer counts, and attention implementations of specific models, refer to their official technical reports and release notes.
