Choosing an LLM: A Practical Guide to Long Context and Mixture of Experts
Long context windows and mixture-of-experts architectures change how large language models use memory, handle inference, and scale across devices. This guide explains how to evaluate total and active parameters separately, compare latency and cost on real tasks, and test whether a million-token context window is useful in practice.
This guide is for developers selecting or deploying large language models (LLMs). It focuses on how architectural choices affect inference cost, latency, and long-context performance. It does not cover training or make definitive claims about individual models. For model-specific specifications, refer to the official technical reports and documentation.
1. What Has Changed in LLM Architecture?
Mixture of experts (MoE) is a common way to scale up a model’s parameter count. An MoE model can have a large number of total parameters, but each token is routed through only a few experts. As a result, the compute required per token depends on the active parameter count rather than the total count. The deployment setup still needs enough accelerator memory across devices to hold all expert weights. Evaluate these two counts separately: total parameters largely determine the memory required to deploy the model, while active parameters are a major factor in inference speed.
Some models also use shared experts. Every token passes through the shared experts, which handle general-purpose processing, while routed experts specialize. Routing adds overhead of its own. When experts are distributed across multiple accelerators or machines, tokens must move between devices, and communication latency can increase as the cluster grows.
Changes to attention primarily target the memory footprint of the key-value (KV) cache. Grouped-query attention (GQA), introduced by Ainslie et al. in 2023, reduces cache size by letting multiple query heads share key and value heads. Multi-head latent attention (MLA), introduced in the 2024 DeepSeek-V2 technical report, compresses keys and values into lower-dimensional latent vectors, further reducing the cache footprint. Sparse attention takes a different approach: each query attends to only a subset of previous positions, reducing computation.
The benefits of sparse attention depend on the implementation. Position selection, indexing, and kernel efficiency all affect throughput. In some implementations, the overhead of scheduling sparse operations can offset the compute savings, so benchmark the actual implementation. Some sources also describe multiple residual information streams, but publicly available implementation details are not sufficient to verify them, so they are not used as a selection criterion here.
2. How Active Parameters Affect MoE Inference Cost
More active parameters mean more computation per token. Decoding is often limited by accelerator memory bandwidth, so latency typically rises as the active parameter count increases. The size of that increase depends on the hardware, batch size, and parallelism strategy.
Whether the extra compute is worthwhile depends on the workload. Tasks that require cross-document reasoning or multiple tool calls are more likely to benefit from a more capable model. For straightforward classification or extraction, a smaller model’s latency and cost advantages are usually more pronounced. To compare models, run them on the same set of tasks and measure p95 latency and cost per successful task.
3. How to Evaluate a Million-Token Context Window
- Distinguish the advertised window from the effective window. A model’s ability to retrieve information from a long input can vary depending on where that information appears. Use needle-in-a-haystack tests and examples that require multi-hop reasoning across documents to measure performance at different input lengths.
- Measure prefill and decoding separately. Prefill processes the input, and its latency grows with input length. During decoding, the model generates output while retaining a KV cache whose memory footprint grows with sequence length. Record the costs of the two phases separately.
- Document the benchmark conditions. At a minimum, report input length, batch size, accelerator model and count, numerical precision, inference framework version, time to first token (TTFT), output tokens per second, and p50 and p95 latency.
