Long Context and MoE: How to Understand and Choose Between Two Major LLM Architectures
This article explains long-context attention and mixture-of-experts (MoE) architectures from memory, compute, and engineering perspectives. It covers why KV cache grows with sequence length, how mixed attention reduces full attention work, the main MoE design patterns, the engineering cost of routing, and the metrics and infrastructure checks needed before selecting a model.
Long Context and MoE: How to Understand and Choose Between Two Major LLM Architectures
Why Long Context Consumes GPU Memory
During Transformer inference, the model must store the Key and Value tensors for every layer, known as the KV cache. The cache size is proportional to sequence length, so longer context means more memory.
A rough estimate:
KV cache bytes ≈ 2 × layers × sequence length × KV heads × head dimension × bytes per element × batch size
For example, assume a model has 80 layers, 8 KV heads, a head dimension of 128, BF16 precision (2 bytes per element), a 1 million token sequence, and batch size 1:
KV cache ≈ 328 GB (decimal)
This does not include model weights or intermediate activations. Models with more KV heads or larger batch sizes push memory usage even higher.
So increasing GPU memory alone is not enough to solve long-context problems. The attention mechanism itself has to change.
Mixed Attention: Reducing the Part That Needs Full Computation
Standard attention lets every token interact with all previous tokens. Mixed attention uses different computation scopes for different layers and positions, preserving long-range information while reducing cache and compute.
Common approaches include:
- Compression: Merge the representations of several consecutive tokens into shorter summaries, which later layers read. Compression can be fixed pooling or learned, and the compression ratio can be fixed or content-adaptive.
- Local selection: Use an index or scoring mechanism to find the segments most relevant to the current query, then run full attention only inside those segments.
- Sliding window: Focus only on recent tokens. This is useful when local dependencies are strong; distant information must be propagated layer by layer.
The benefit depends on model scale, training setup, sequence length, and task type. Compression and sparsification both cause some information loss, and long-document QA and cross-paragraph retrieval are the tasks most likely to expose it. Cache-saving percentages reported in technical papers usually apply only to the configuration described there; when applied to your own workload, they need to be measured again.
Major MoE Design Patterns
MoE replaces one large feed-forward network with multiple expert feed-forward networks. A gating network selects which experts each token should use. Total parameter count can be large, while the active parameters actually used per token remain relatively small.
Common implementations can be grouped as follows.
1. Shared Experts plus Routed Experts
This design includes one or a few always-active shared experts and many routed experts. The shared experts handle general semantics and basic knowledge, so routed experts can focus on more specific patterns. This helps reduce the problems of idle experts and concentrated load. The cost is that shared experts participate in every token’s computation and their weights must stay in memory.
2. Pure Routed Experts with Top-K Gating
This is the classic approach. The gating network scores all experts for each token and selects the top K, usually K = 2. It is simple and well supported by mainstream training and inference frameworks. The main difficulty is load balancing: auxiliary losses, capacity limits, or other strategies are needed to prevent a few experts from being occupied by too many tokens.
3. Task-Oriented Routing
Some models expose certain experts to more data of a particular type during training, for example code or math. At inference, the gating network tends to route relevant tokens to those experts. This can improve efficiency in a specific domain, but the benefit depends heavily on training data mix and routing design. Whether it outperforms a dense model of the same compute budget must be verified on the relevant benchmarks; it cannot be inferred from the architecture name alone.
4. Low Activation Ratio
Some models have a large total parameter count but activate only a small fraction per token. This reduces per-token compute and suits high-concurrency, latency-sensitive services. However, fewer active parameters does not mean less memory. At inference, all expert weights usually need to be resident in GPU memory or in fast-accessible storage, so the deployment barrier is not necessarily lower than a small dense model.
| Paradigm | Advantage | Main Risk | Suitable Scenarios |
|---|---|---|---|
| Shared experts + routed experts | Stable general capability; better expert utilization | Shared experts consume fixed compute and memory | General-purpose models; training stability matters |
| Top-K gating | Simple structure; mature ecosystem compatibility | Load imbalance; extra balancing strategies needed | Versioned releases that must run on multiple hardware types |
| Task-oriented routing | High utilization in a specific domain | Depends on data mix; general capability may suffer | Domain-specific models |
| Low activation ratio | Low compute per token; high throughput | Large weight footprint; deployment can be costly | High-concurrency, cost-sensitive online services |
Engineering Costs of Routing
MoE compute is sparse, but communication is not necessarily sparse. Experts are usually distributed across multiple GPUs. Each token must be sent to the devices hosting its selected experts, and the results must be sent back. This is an All-to-All communication pattern, and its cost changes with the number of experts, batch size, and cross-node bandwidth.
The following issues should be evaluated in advance:
- Load imbalance: The device hosting a popular expert becomes a bottleneck while other devices wait.
- Communication latency: If cross-node bandwidth is insufficient, communication time may exceed compute time.
- Memory fragmentation: Dynamically allocated intermediate tensors reduce GPU memory utilization.
- Precision regression: After quantization or inference optimization, outputs from some experts or attention layers may drift from training-time behavior, so they need to be rechecked on real tasks.
Training and inference also have different bottlenecks. Training is more concerned with throughput and overlapping communication with compute. Inference is more concerned with first-token latency, concurrency, and KV cache management. These should be evaluated separately.
Metrics to Check Before Choosing a Model
Before committing to an architecture or model, run through the following checklist:
- Identify whether the main bottleneck is memory, compute, latency, or throughput. Each bottleneck points to a different optimization direction.
- Record total parameters, active parameters, layer count, KV head count, and supported maximum context length.
- Calculate KV cache and weight usage for the target batch size and sequence length, and confirm whether a single GPU or node can hold them.
- Check whether the inference framework supports the architecture and whether the required quantization format is available.
- Test with your own long-text data and business benchmarks. Record accuracy, first-token latency, output tokens per second, and peak GPU memory.
- Compare interconnect bandwidth across hardware options, and confirm that All-to-All communication will not become the main bottleneck at your target scale.
Evaluation conditions should be fixed: same sequence length, batch size, precision, and hardware. Otherwise the results are not comparable.
Infrastructure Considerations
Architecture innovation places specific demands on the underlying platform. Mainstream cloud platforms usually provide GPU container orchestration, object storage, and high-performance networking, which can be used to schedule heterogeneous compute and store model weights and datasets. For MoE training and high-concurrency inference, node-to-node network bandwidth, GPU interconnect method, and memory scheduling deserve particular attention.
Architecture selection and infrastructure configuration should be considered together. The same model can perform very differently on different clusters, so it must be validated in the target environment.
