Large Language Model Technical Routes: Long Context, Sparsity, and Multimodality
This article covers the main technical trade-offs in large language model design. It explains how attention cost limits long-context processing, how Mixture-of-Experts routing and load balancing add sparsity, and how distillation, quantization, and pruning reduce model size. It also covers aligning visual and text tokens in multimodal models, controllability through preference data and principle-based self-correction, and how to choose a solution for common scenarios.
Background: One Architecture, Different Engineering Constraints
Large language models based on the Transformer have converged on similar training objectives, but their constraints in production differ widely. Latency, GPU memory, throughput, training data composition, and safety requirements together determine where a team puts its resources. As a result, general-purpose large models built on the same foundation follow clearly different technical routes. This article examines three areas of trade-off: how to handle long inputs, how to reduce the computation per inference, and how to help models understand images and other non-text information.
Long Context: Attention Cost Is the Core Bottleneck
The computation of standard self-attention scales with the square of the sequence length. When the sequence grows from 4K to 128K tokens, the attention matrix grows by roughly 1,000 times ((128/4)² = 1,024). Beyond computation, the KV cache used during inference grows linearly with sequence length, and memory often runs out before compute does.
Common engineering approaches include:
- Sparse attention: computes attention only over local windows or specific patterns, which lowers the complexity.
- KV cache compression: reduces memory usage through grouped-query attention, cache quantization, and similar methods.
- Position encoding extrapolation: adjusts how rotary position embeddings are scaled so the model produces stable outputs beyond its training length.
- Retrieval augmentation: splits very long documents into chunks and retrieves the segments relevant to the question before sending them to the model, instead of placing all the content in the context.
A long context window does not guarantee that the model uses all of the information in it. Research has observed that information in the middle of long texts is more likely to be overlooked. Therefore, teams should run targeted evaluations to measure actual recall, rather than relying only on the advertised window length.
Sparsity: Routing in Mixture-of-Experts
The core idea of Mixture-of-Experts (MoE) architectures is to split the feed-forward layer into multiple expert sub-networks. A gating network selects a small number of experts for each token. The computation per token depends mainly on the number of activated parameters, not the total parameter count. This means the overall model can be very large while inference cost stays manageable.
A typical routing process works as follows:
- The gating network outputs a score (logit) for each expert, for each token.
- Softmax is applied to the scores to produce a probability distribution.
- The K experts with the highest probabilities are selected.
- The outputs of the selected experts are combined as a weighted sum, using the normalized weights.
A common misconception is that the temperature coefficient determines how many experts are activated. It does not. Softmax is a monotonic function, so temperature scaling does not change the ordering of the scores. In pure top-K routing, the choice of which K experts are selected is independent of temperature. Temperature only affects how weight is distributed among the selected experts. A higher temperature pushes the weights toward uniform, while a lower temperature concentrates the weight on the highest-ranked experts. The number of activated experts is set by K.
import numpy as npdef top_k_route(hidden, w_gate, k=2, temperature=1.0):"""hidden: hidden vector of a single token, shape [d]w_gate: gating weights, shape [d, num_experts]Returns: selected expert indices and normalized weights"""logits = hidden @ w_gate # expert scoresscaled = logits / temperatureprobs = np.exp(scaled - scaled.max()) # numerically stable softmaxprobs /= probs.sum()experts = np.argsort(logits)[-k:] # selection depends only on the original rankingweights = probs[experts] / probs[experts].sum() # temperature only affects the weightsreturn experts, weights
In practice, training must also address load imbalance. If the gate consistently favors a few experts, those experts receive most of the training signal while others rarely get updated, and the favored experts become bottlenecks. Common remedies include adding a load-balancing term to the training loss and setting a capacity limit for each expert. Tokens that exceed an expert’s capacity are either routed to other experts or skip that layer’s computation.
Lightweighting: Trading Off Quality and Cost
The main lightweighting techniques fall into three categories:
- Knowledge distillation: uses the output distribution of a large model as soft labels to train a smaller model, preserving part of its reasoning ability.
- Quantization: compresses weights from 16-bit floating point to 8-bit or 4-bit integers. Memory usage drops roughly in proportion to the bit width, but accuracy loss increases as the bit width decreases, so results must be evaluated on the target task.
- Pruning: removes redundant channels, layers, or attention heads. It usually requires fine-tuning to recover performance.
Clarify the task characteristics before choosing a technique. Tasks such as structured extraction and text classification generally suit smaller models well. Multi-step reasoning and complex code generation are more sensitive to parameter count, and performance degradation after compression is often more noticeable.
Multimodality: Aligning a Unified Representation Space
A common multimodal design works as follows. A vision encoder divides an image into patches and maps each patch to a vector. A projection layer then connects these vectors to the language model, so visual information is processed in the same form as text tokens. Training typically has two stages: image-text alignment pretraining, followed by instruction tuning.
Two factors are critical for this approach. First, the quality of the semantic alignment between visual tokens and text tokens directly affects accuracy in chart understanding and text recognition. Second, the balance between input resolution and token count. Higher resolution captures more detail, but it increases the number of visual tokens, which raises inference latency and memory use. Scenarios such as document parsing and long-image understanding usually rely on tiling or dynamic resolution strategies to balance the two.
Controllability: Rule Constraints and Data-Level Alignment
Model controllability follows two main paths.
The first path is training on preference data. Human annotators rank multiple responses to the same prompt. These rankings train a reward model, which is then used to tune the policy model through reinforcement learning or direct preference optimization.
The second path is principle-based self-correction. The model receives a set of explicit behavioral principles and uses them to critique and rewrite its own outputs. The revised outputs are then used for training. This approach reduces reliance on large-scale human annotation, but the wording of the principles directly affects the results.
Data-level work is equally important. It includes cleaning training corpora, covering samples in the target languages and cultural contexts, and using structured knowledge bases to provide verifiable factual grounding for answers. The first two paths mainly constrain the model’s behavioral boundaries, while data-level methods mainly improve factual accuracy. The approaches complement each other.
Selection Guidelines
| Scenario | Priority metrics | Common approaches |
|---|---|---|
| Long document Q&A, contract review | Effective context length, recall rate | Retrieval augmentation, long-context models |
| High-concurrency online services | Cost per token, time to first token | Sparse architectures, quantized deployment |
| Image understanding, document parsing | Visual resolution, number of visual tokens | Multimodal models, chunked processing |
| Edge and on-device deployment | Memory footprint, power consumption | Knowledge distillation, low-bit quantization |
When selecting a solution, first build an evaluation set that matches the distribution of real business inputs. Then compare candidates on quality, latency, and unit cost. Judgments based only on public leaderboards or parameter counts often diverge significantly from real business performance.
