Large MoE LLM Architecture and Evaluation: Sparse Experts, Latent Attention, and Million-Token Context
Large language models are shifting from dense parameter scaling toward sparse Mixture-of-Experts (MoE) designs that expand total parameters and context length while keeping per-token compute in check. Using an illustrative configuration as a running example (about 770B total parameters, 49B active per token, 1M-token context, 78 layers), this article walks through the key engineering decisions of a new generation of MoE models. These cover routed and shared experts, attention redesign built on latent and sparse attention, KV cache pressure at million-token scale, multi-token prediction (MTP), FP8 quantization, and evaluation. The figures illustrate design trends and tradeoff logic rather than the specs of any particular product. Readers will come away with a deployment- and cost-oriented view of MoE and guidance on evaluating LLMs against their own business workloads instead of relying on potentially contaminated public benchmarks.
Introduction: Why “Sparsity + Long Context” Have Become the Main Direction for LLMs
Over the past two years, competition in large language models has shifted from simply stacking dense parameters toward expanding parameter counts and context lengths while keeping inference costs under control. A dense model activates every parameter for every token it generates. Once parameter counts reach the hundreds of billions, the compute and memory required for both training and inference rise sharply. By contrast, Mixture of Experts (MoE) uses a sparse structure with many total parameters but only a few active per token. This decouples model capacity from the compute required for a single forward pass, which has made MoE a mainstream choice for a new generation of flagship models.
At the same time, agentic applications impose hard requirements for ultra-long context. A single task may need to pack in hundreds of thousands or even a million tokens of conversation history, tool outputs, and knowledge snippets. Context windows of 256K, or even 1M, are no longer marketing numbers; they are real engineering constraints.
This article uses an illustrative configuration as an example: roughly 770B total parameters, about 49B active per token, a 1M-token context window, and around 78 layers. It then unpacks, layer by layer, the key design choices of a new generation of LLMs across sparsity, attention redesign, inference acceleration, quantization, and evaluation. Note that the numbers here only illustrate design trends and the logic behind tradeoffs. They are not a specification commitment for any specific product.
1. The Sparse Route: How MoE Delivers Large Parameters Without Large Compute Costs
The most important MoE metric is the ratio of total parameters to active parameters. In the illustrative configuration, this moves from 295B/21B in the previous generation to 770B/49B. Total parameters grow roughly 2.6×, and active parameters roughly 2.3×. Two clear design principles underpin this shift:
- Knowledge capacity comes mainly from total parameters. More experts give the model room for more granular knowledge and skills, so long-tail knowledge in the training data has more “places to live.”
- Compute per inference depends only on active parameters. Each token passes through only its Top-k experts. As a result, compute (FLOPs) per token stays bounded even when total parameters are very large.
This leads to a conclusion that is often overlooked: for MoE, the deployment bottleneck is often not compute but memory and expert-parallel communication. All expert weights must remain resident in GPU memory. All-to-All routing communication across devices can also become an invisible ceiling on throughput. A typical Top-k routing step can be sketched conceptually as follows:
def moe_forward(x, experts, router, top_k=8):gate = router(x) # score for each expertidx, weight = topk_softmax(gate, top_k)y = 0for i, (e_id, w) in enumerate(zip(idx, weight)):y = y + w * experts[e_id](x) # compute only with the selected expertsreturn y
2. Routed and Shared Experts: Designing Sparse Activation for Stability
The illustrative configuration increases routed experts from 192 to 256. It also retains the “Top-8 routing + 1 shared expert” structure. The shared expert is a key design choice worth highlighting. Regardless of the routing result, the shared expert is activated for every token and carries general-purpose capabilities such as grammar, common sense, and stable representations. Routed experts, in contrast, handle capabilities specialized to particular domains.
This “shared + routed” setup effectively mitigates two problems common in purely routed designs:
- Expert collapse: a few experts are overused while the rest sit nearly idle.
- Capability instability: outputs for the same type of input drift because of fluctuations in routing.
When general-purpose capabilities are consolidated in the shared experts, training becomes smoother and downstream fine-tuning more robust.
3. Attention Redesign: From GQA to Latent Attention and Sparse Attention
3.1 Why Long Context Calls for Sparsity
Standard attention has quadratic complexity in sequence length, O(N²). When context grows from 256K to 1M, attention becomes the dominant cost. New-generation models therefore generally adopt a combination of latent attention and sparse attention.
3.2 Compressing the KV Cache with Latent Attention
Grouped-query attention (GQA) reduces the KV cache by letting multiple query heads share one set of key/value (KV) heads. Latent attention goes a step further. It stores K/V compressed into a low-rank latent space and reconstructs them on demand at inference time, which can sharply reduce KV cache size. This is the first layer of optimization that makes million-token context practical.
3.3 A Lightweight Indexer: Computing Only the Relevant Part
A sparse mechanism is layered on top of latent attention. A lightweight indexer quickly scores historical tokens to identify which positions are most relevant to the current query. The main attention then processes only a fixed number of positions (for example, about 2048). Complexity drops from roughly O(N²) to O(N·k). Conceptually, this looks like:
def sparse_attention(q, k, v, indexer, budget=2048):scores = indexer(q, k) # cheap scoringkeep = topk_indices(scores, budget) # select the most relevant tokensreturn softmax(q @ k[keep].T) @ v[keep]
3.4 Reusing Indexes Across Layers: Cutting Sparse Overhead Further
Index selection itself has a cost. One established approach is to have most of the 78 layers (57 in the illustrative configuration) directly reuse the token indexes selected by a small group of earlier layers (21 layers). Only those earlier layers run the full indexer. This spreads the indexing cost across layers while preserving most of the compute savings from sparsity. In effect, it trades a very small representation error for meaningful gains in inference efficiency.
4. Million-Token Context: The KV Cache Is the Real Bottleneck
The engineering difficulty with long context lies not in whether the input can fit, but in how to read it efficiently once it does. Latent attention compression reduces the cache size per token. Sparse attention reduces the number of tokens computed per step. Together, they make 1M-token context operationally viable. In practice, teams typically combine these with general techniques such as KV cache quantization and paged cache management to balance memory usage against access locality.
5. Multi-Token Prediction (MTP) and Inference Acceleration
In the illustrative configuration, the MTP draft layers scale up from a small size in the previous generation to about 10B total parameters with 0.7B active. The idea behind multi-token prediction is closely related to speculative decoding. A stronger draft layer predicts several subsequent tokens at once. The main model then verifies them in parallel and accepts the correct prefix in one pass.
The stronger the draft layer, and the closer its output distribution is to that of the main model, the higher the acceptance rate and the more throughput improves. MTP does not change output quality. Instead, it raises the curve of generation speed, which directly shapes the user experience.
6. Precision and Quantization: The FP8 and Mixed-Precision Tradeoff
Beyond BF16, the illustrative configuration adds low-precision open formats such as MXFP8. The core value of FP8 is reducing memory usage and memory-bandwidth pressure. This improves throughput and lowers per-GPU concurrency pressure.
The common practice is mixed precision. Sensitive paths such as attention and residual connections retain higher precision. The large matmuls inside MoE experts use low precision. This delivers substantial efficiency gains while keeping accuracy loss within a manageable range.
7. Evaluating LLMs Scientifically: Don’t Rely on Public Benchmarks Alone
This is where teams are most easily misled. Public evaluations today face a widespread data contamination problem. Test set content may already have leaked into training data, inflating scores and distorting rankings. Glancing at a single leaderboard number for curiosity is fine. Using it to make serious model-selection decisions, however, carries considerable risk. A more reliable approach is a multi-dimensional evaluation framework:
- Real-world tasks: private evaluation sets tailored to your own business, covering long-document QA, code, and multi-turn instruction following.
- Long-context capability: needle-in-a-haystack tests, reasoning across passages, and information extraction accuracy.
- Agent capability: success rates for multi-step tool calls, task completion rates, and error recovery.
- Stability: variance in outputs across repeated calls with the same input, and hallucination rates.
- Cost: price per token, end-to-end latency, and concurrent throughput.
8. Deployment and Cost: Balancing the Triangle of Throughput, Latency, and Price
Whether a model works well ultimately comes down to cost efficiency. Evaluate it across three dimensions: price, throughput, and latency. Comparing unit price alone is not enough. A model that wins on benchmarks but loses throughput sharply under long context may produce a higher bill in production.
In practice, general cloud platform capabilities can close the loop. A container platform can host inference services. A monitoring and alerting system can track latency and memory usage. Object storage can hold evaluation data and artifacts. Baidu AI Cloud and similar platforms provide general-purpose cloud products such as model inference, container orchestration, and logging and monitoring that can serve as infrastructure references for deployment and operations. The final choice, however, should still rest on your own workload characteristics and be validated through load testing.
Conclusion
Progress in new-generation LLMs comes down to continually finding a better balance among capacity, context length, and cost. MoE decouples capacity from compute. Latent attention and sparse indexing tame long context. MTP raises generation speed. Low-precision quantization lowers the barrier to deployment.
For developers, what matters is not memorizing a set of parameters. It is understanding the tradeoff logic behind each design choice. It also means building an evaluation approach grounded in real business needs and not swayed by leaderboard numbers. Only then can technical decisions hold up against actual bills in an era of rapidly iterating models.
