Long Context and MoE Architectures: A Deep Dive into Core LLM Technologies

    This article explores the engineering bottlenecks of long-context Large Language Models (LLMs) and the core technologies used to overcome them. It details the mechanisms behind KV cache compression, sparse and hierarchical attention, and the Mixture of Experts (MoE) architecture. Additionally, it provides a practical framework for evaluating long-context solutions and offers actionable engineering deployment recommendations to balance memory, compute, and system scheduling.

    1. Why Long Context Becomes an Engineering Bottleneck

    When LLMs process long texts, the primary cost stems from the attention mechanism. Standard Multi-Head Attention (MHA) scales quadratically with sequence length in terms of computation. In terms of memory, it must store the Key and Value for every layer and every token, which is known as the KV cache. During inference, each newly generated token requires reading these caches; thus, a larger cache means higher memory bandwidth pressure per decoding step.

    The KV cache size can be estimated using the following empirical formula:

    1. KV cache bytes = 2 × num_layers × num_kv_heads × head_dim × seq_length × bytes_per_element

    The coefficient 2 indicates that Key and Value are each stored once. Let’s demonstrate the calculation with a set of hypothetical parameters (these are examples and do not represent any specific model):

    • 32 layers, 32 KV heads per layer, 128 head dimensions, using 16-bit floats (2 bytes per element)
    • Sequence length set to 1 million tokens

    Substituting into the formula: 2 × 32 × 32 × 128 × 1,000,000 × 2 ≈ 5.2 × 10¹¹ bytes, which equals about 488 GiB. This number illustrates that without optimization, the cache for a million-level context alone can exceed the total VRAM of a single machine.

    From this calculation, we can see three approaches to reducing long-context costs: reducing the amount of data saved per token, reducing the number of tokens involved in computation, and changing how computations are organized to improve hardware utilization.

    2. Approaches to Compressing the KV Cache

    2.1 Reducing KV Heads

    In MHA, each query head has its own independent Key and Value heads. Multi-Query Attention (MQA) allows all query heads to share the same set of KV heads, shrinking the cache volume to a fraction of its original size. Grouped-Query Attention (GQA) takes a middle-ground approach, dividing query heads into groups where each group shares a set of KV heads. Both trade off a small amount of expressive power for cache space. In practice, GQA is more common because it offers a better balance between quality and efficiency.

    2.2 Low-Rank Projection

    Another approach is to avoid caching the full Key and Value directly. Instead, they are projected into a low-dimensional latent space, and only the compressed representation is cached. During attention calculation, the projection matrix is either restored or absorbed. The benefit of this method depends on the chosen compression dimension. Too low a dimension loses information; too high a dimension offers limited savings. During design, experimental validation is required between perplexity and memory footprint, rather than just looking at theoretical compression ratios.

    2.3 Quantized Cache

    Compressing the KV cache from 16-bit to 8-bit or even 4-bit can directly halve or reduce memory usage by three-quarters. The risk of quantization lies in outliers. Some channels have a large numerical range, and simple per-tensor quantization will cause a noticeable drop in accuracy. Common practices include per-channel or per-group quantization, and retaining higher precision for critical layers.

    2.4 Paged Memory Management

    In inference services, the sequence lengths of different requests vary greatly. Pre-allocating contiguous VRAM based on maximum length causes significant fragmentation and waste. Paged memory management slices the cache into fixed-size blocks, allocating and reclaiming them on demand, which can significantly improve concurrency. This type of optimization is system-level and independent of model structure, yet it is often the key to whether a long-context service can go live.

    3. Sparse and Hierarchical Attention

    3.1 Sliding Window and Global Tokens

    Sliding window attention allows each token to focus only on a few neighboring tokens, reducing computational complexity from quadratic to linear. Its drawback is that long-distance information cannot be passed directly. A common practice is to retain a small number of global tokens that can see the entire sequence and be accessed by other tokens. The logic of this design can be expressed with the following pseudocode:

    1. def build_sparse_mask(seq_len, window, global_tokens):
    2. mask = zeros(seq_len, seq_len, dtype=bool)
    3. for i in range(seq_len):
    4. lo = max(0, i - window)
    5. mask[i, lo:i + 1] = True # Local window
    6. mask[:, global_tokens] = True # All tokens can access global tokens
    7. mask[global_tokens, :] = True # Global tokens can access all tokens
    8. return mask

    This code only expresses the construction logic of a sparse mask. Actual implementations require efficient kernels; otherwise, sparsity does not necessarily translate to speed improvements.

    3.2 Hierarchical Mixed Design

    A common engineering approach is to use different attention spans across different layers. Shallow layers focus on local patterns, such as lexical and phrase structures, and can use narrow windows. Deep layers focus on semantic correlations, requiring a larger receptive field, and can use compressed global representations or sparser but broader patterns. Such hierarchical arrangements maintain a good balance between computational load and expressive power.

    3.3 Adaptive Compression

    With a fixed compression ratio, simple and complex texts receive the same processing intensity, which is clearly unreasonable. Adaptive schemes attempt to dynamically adjust the compression degree based on the information density of the input content—for example, increasing compression for highly repetitive paragraphs while retaining original precision for key information. The difficulty with these methods lies in training stability and how to make the model learn to judge “which information can be compressed.” Without sufficient ablation studies, adaptive strategies should be treated as unverified hypotheses rather than guaranteed benefits.

    4. Mixture of Experts Architecture: Shared and Routed Experts

    The Mixture of Experts (MoE) architecture replaces the single Feed-Forward Network (FFN) layer with multiple FFNs (experts), where only a small fraction of experts are activated per token. This allows the model to have a very large total parameter count, while the inference computation cost relates only to the activated parameters.

    4.1 Routing Mechanism

    The router is a small linear layer that calculates scores for each expert per token, then selects the top-scoring experts:

    1. def moe_forward(x, router, experts, shared_expert, top_k=2):
    2. scores = softmax(router(x)) # Score for each expert
    3. topk_idx = argsort(scores)[-top_k:] # Select top_k routed experts
    4. weights = scores[topk_idx] / sum(scores[topk_idx])
    5. out = shared_expert(x) # Shared expert always participates in computation
    6. for idx, w in zip(topk_idx, weights):
    7. out += w * experts[idx](x)
    8. return out

    This pseudocode illustrates a common structure: the shared expert handles general knowledge needed by all tokens, while the routed experts are responsible for more specialized patterns.

    4.2 Tradeoffs of Fixed-Quantity Configuration

    A common configuration is “several routed experts plus one shared expert,” such as selecting two out of multiple routed experts each time. This structure has several distinct characteristics:

    • The shared expert ensures each token has a stable baseline representation, reducing duplicate learning among routed experts.
    • When there are many routed experts, each can be more specialized, but more VRAM is needed to store parameters.
    • Routing results must be constrained by a load balancing mechanism; otherwise, a few experts will be overused while the rest are undertrained.

    Load balancing generally uses two approaches: first, adding an auxiliary term to the loss function to penalize uneven distribution; second, setting an expert capacity limit during inference or training, where excess tokens are dropped or forwarded. Both have costs: the former may interfere with main task optimization, while the latter affects output quality.

    4.3 Comparison with Dense Models

    In dense models, every token passes through all parameters. The structure is simple, and the engineering complexity of training and deployment is lower. MoE models can have larger parameter capacities at the same inference cost, but they must handle expert parallelism, communication overhead, and load imbalance. Selection should be based on target throughput, latency, and hardware conditions, rather than simply pursuing parameter scale.

    5. How to Evaluate a Long-Context Architecture Solution

    When evaluating, it is recommended to start from four dimensions and ensure consistent testing criteria:

    1. Memory and Throughput: Record peak VRAM, Time To First Token (TTFT), and tokens generated per second at different sequence lengths.
    2. Long-Range Information Utilization: Use retrieval-based tasks, placing key information at different positions in the context, and observe whether recall decays based on position.
    3. General Capability Regression: After long-context optimization, test whether short-text reasoning, coding, and knowledge QA capabilities decline.
    4. Stability: Check for memory fragmentation, abnormal outputs during long-term operation, and truncation behavior for ultra-long inputs.

    Public benchmark scores should only serve as a reference. Different evaluation sets vary greatly in topic distribution, prompt templates, and sampling parameters. Directly comparing numbers can easily lead to wrong conclusions. The internal team should standardize a set of evaluation scripts to compare different solutions using the same batch of data.

    6. Engineering Deployment Recommendations

    1. Calculate Before Selecting: Based on target concurrency, average context length, and peak context length, estimate cache requirements using the previous formula to determine hardware and batching strategies.
    2. Introduce Optimization in Phases: Prioritize enabling paged memory management and quantization before considering model structural changes, as the former carries lower risk and is easier to roll back.
    3. Establish an Observability System: Set up monitoring and alerts for memory usage, request queue time, and output quality to promptly detect anomalies caused by long-tail requests.
    4. Leverage General Cloud Capabilities: When deploying in a cloud environment, you can use object storage to save model weights and evaluation data, a container platform to manage inference instances, and logging and monitoring services to track service status and perform elastic scaling based on traffic.

    Overall, the improvement of long-context capabilities is the result of the synergy between memory, compute, and system scheduling. Understanding the cost source of the KV cache, distinguishing the applicable conditions of different compression and sparse methods, and then verifying the effects with a unified evaluation approach is the fundamental path to judging whether a solution is reliable.