Architecture Design and Engineering Practices for Ultra-Scale Mixture-of-Experts Large Language Models

    This article explains the engineering principles behind ultra-scale Mixture-of-Experts (MoE) language models, including sparse activation, long-context processing, native multimodal input, model evaluation, cost accounting, deployment considerations, and training efficiency. It focuses on how to balance model capacity, compute cost, latency, memory usage, and task performance in real-world systems.

    1. Background: The Tension Between Model Scale and Cost

    The parameter counts of large language models have continued to grow over the past several years. Larger models generally offer greater knowledge capacity and stronger reasoning ability, but their training and inference costs also increase. If every computation activates the full parameter set, the compute cost grows approximately in proportion to the model size, making ultra-scale models difficult to deploy in practice.

    Engineering teams therefore need to balance model capability, response latency, and per-task cost, and identify a trade-off that fits their specific workloads.

    This article uses a typical class of ultra-scale open-source models as its background and summarizes several broadly applicable engineering concepts, including sparse expert architectures, long-context processing, multimodal input, evaluation methodology, cost accounting, and deployment strategies. These descriptions apply to similar architectures in general and do not represent the official specifications of any particular product.

    2. Mixture-of-Experts: Using Sparse Activation to Increase Model Capacity

    The core idea of Mixture-of-Experts (MoE) is to divide the feed-forward network into multiple expert sub-networks. A router then selects a small number of experts for each input token. The model can contain a large number of experts, while each token passes through only a subset of them. As a result, compute requirements are no longer strictly tied to the model’s total capacity.

    When analyzing an MoE model, it is important to distinguish between two types of parameters:

    1. Total parameters: The combined parameters of all experts, shared layers, and embedding layers. This determines the model’s knowledge capacity and storage requirements.
    2. Activated parameters: The parameters that actually participate in a single forward pass. This determines the compute required per token and strongly affects inference latency.

    For example, a model may contain hundreds of experts but activate only a dozen or so for each token. Its per-token compute can therefore be far lower than that of a dense model with a similar total parameter count. GPU memory requirements should be estimated based on total parameters, while throughput and latency should be evaluated based on activated parameters.

    The computation flow of an expert layer can be represented by the following pseudocode:

    1. Input: token hidden vector x
    2. routing scores = routing_network(x) # One score for each expert
    3. Select the top-k experts using top_k(scores)
    4. Normalize the scores of the selected experts to obtain weights w
    5. Output y = sum of w_i * expert_i(x) for the selected experts

    This structure presents three major engineering challenges.

    Load Balancing

    If the router consistently favors a small number of experts, the compute units hosting those experts may become overloaded while other units remain underutilized. Common approaches include adding a load-balancing term to the training loss and introducing a capacity factor that limits the number of tokens each expert can process in a batch.

    Communication Overhead

    Experts are usually distributed across multiple devices. Tokens must be dispatched across devices and gathered again after computation. This communication pattern has a direct impact on scaling efficiency, so MoE systems require suitable communication kernels, scheduling strategies, and carefully designed parallelism strategies.

    Training Stability

    Routing involves a discrete selection operation, which requires special treatment for gradient propagation. The learning rate, routing temperature, and load-balancing coefficient must be tuned jointly. Adjusting only one of these factors can easily cause training instability or oscillation.

    3. Long Context: From Attention Complexity to State Compression

    The computational cost of standard self-attention grows quadratically with sequence length. Increasing a sequence from 100,000 tokens to 1 million tokens can increase the attention computation by approximately 100 times. During inference, the memory required by the key-value cache (KV cache) grows linearly with context length. As the number of concurrent requests increases, GPU memory can quickly become the main bottleneck.

    Common approaches to long-context processing include the following:

    1. Sparse attention: Compute attention only between selected positions, such as a local window combined with a small number of global anchor points. This reduces computation, but it is necessary to verify that important information is located within the positions being attended to.
    2. Linear attention: Use kernel approximations or recurrent states to make computational complexity approximately linear with sequence length. This approach is efficient, but it may underperform full attention on retrieval-oriented tasks that require precise localization of a particular passage.
    3. Hybrid attention: Alternate linear-attention layers and full-attention layers at different depths of the network. This provides a compromise between long-sequence efficiency and precise retrieval capability and has become an area of significant interest.
    4. Cache compression: Quantize or reduce the dimensionality of key-value tensors to lower memory consumption directly. This is useful for increasing concurrency without changing the network architecture.

    The organization of residual paths also affects information flow in deep networks. Some research has explored improved methods for aggregating information across layers, allowing deep networks to reuse shallow features more effectively and potentially improving scaling efficiency.

    The appropriate design depends on the task. Document question answering and codebase retrieval require precise information localization and therefore rely more heavily on full-attention layers or retrieval-augmented methods. Long-conversation summarization and log trend analysis are often better suited to linear or sparse approaches.

    4. Native Multimodality: Unified Modeling of Visual Inputs

    There are two common approaches to adding multimodal capabilities. One is to attach a vision encoder to an already trained language model and then perform alignment fine-tuning. The other is to train the model from the pretraining stage to accept both text and images, allowing the two types of signals to be learned in a shared representation space.

    The latter approach is commonly called native multimodality. It can help a model develop cross-modal reasoning capabilities, such as answering numerical questions based on a chart or locating a user-interface control from a screenshot.

    Key engineering considerations for native multimodal models include the following:

    1. Image tokenization: An image is divided into multiple visual patches, and each patch is mapped to a group of visual tokens. These tokens are then concatenated with text tokens into a unified sequence. Higher resolution produces more visual tokens and therefore increases compute and memory requirements.
    2. Data mixture: The ratio of pure-text data to image-text data affects performance on both types of tasks. This ratio must be determined through experimentation rather than judged from a single metric.
    3. Staged evolution: A common engineering trade-off is to strengthen text capabilities first and then extend the model to multimodal inputs. This approach stabilizes the language foundation before adding visual capabilities within the same framework, reducing training risk.

    For business applications that involve only text, adding visual capabilities increases cost without necessarily providing benefits. Teams should first identify the types of inputs their workloads require and then decide whether a multimodal model is necessary.

    5. Evaluation: How to Interpret Leaderboard Scores Correctly

    Public leaderboards provide a convenient starting point for comparing models, but score comparability depends on many conditions. When comparing two models, teams should at least verify the following:

    1. Whether the evaluation datasets are the same, and whether their difficulty and domain distributions are consistent.
    2. Whether scoring is automated or based on human preferences, and whether tool use is permitted.
    3. Whether inference settings such as prompt templates, sampling temperature, and context length are consistent.
    4. Whether the leaderboard update date corresponds to the model version being evaluated.

    For programming tasks, common metrics include pass rates such as pass@k, unit-test success, the degree to which code changes are minimized, and human preference scores for interface quality. Front-end tasks also require evaluation of rendering results and interaction details, which are difficult to judge fully through automation.

    Organizations should build internal evaluation sets based on real business data. These sets should cover common requests, edge cases, and known failure cases, and should be updated regularly.

    A simplified evaluation process is as follows:

    1. For each sample:
    2. Construct a prompt and call the model
    3. Record the output and latency
    4. Run automated checks, such as format validation,
    5. unit tests, or comparison of key fields
    6. For samples that cannot be judged automatically,
    7. record a human score
    8. Aggregate:
    9. pass rate
    10. average latency
    11. cost per request
    12. distribution of failure types

    Internal evaluations help teams understand how a model performs on their own workloads instead of relying solely on general-purpose rankings.

    6. Training Efficiency and Scalability

    Scaling efficiency describes how much model performance improves for a given compute investment, or how much additional value is obtained from each unit of compute as the model grows.

    Improving scaling efficiency generally requires progress in several areas:

    1. Data quality: Deduplication, filtering low-quality content, and designing an appropriate data curriculum often matter more than simply increasing the amount of data.