Ultra-Large Multimodal MoE Models: Weight Analysis and Local Deployment

    This article explains how to interpret a 2.8-trillion-parameter multimodal Mixture-of-Experts model, why total parameters differ from activated parameters per token, and how routed and shared experts affect computation. It also examines million-token sparse attention, native multimodal input costs, 1.56 TB weight files, resource assessment before local deployment, practical deployment strategies, and the real thresholds for running ultra-large models locally.

    1. What “2.8 Trillion Parameters” Actually Means

    When describing ultra-large models, the most easily misunderstood metric is the total parameter count. For models that use a Mixture-of-Experts (MoE) architecture, the total number of parameters does not mean that every input token passes through all parameters.

    These models typically consist of many expert networks. After an input enters the model, the routing module selects a small number of experts from the expert pool based on the semantic features of the token. Experts that are not selected do not execute during the computation for that token. Therefore, the model’s “total capacity” and its “per-token computation” are two different concepts.

    Consider the public structural description of the target model:

    • Total parameters: approximately 2.8 trillion;
    • Activated parameters per token: approximately 104 billion;
    • Routable experts: 896;
    • Experts selected per token: typically 16;
    • In addition to routed experts, 2 shared experts are configured;
    • Network depth: 93 layers;
    • Maximum context length: approximately 1 million tokens;
    • Input support: both text and images.

    Thus, “2.8 trillion parameters” is better understood as the model’s total capacity, while “104 billion activated parameters” is closer to the computation scale of a single inference pass. The former determines how much knowledge and specialization capacity the model can hold, while the latter more directly affects the computation cost per token.

    2. How the MoE Architecture Performs Computation

    A traditional dense model sends every token through the same complete feed-forward network. As model size increases, the number of matrix multiplications required for a single inference pass also increases. MoE models split the feed-forward network into multiple experts, and each expert can develop different parameter specializations during training.

    A simplified processing flow is as follows:

    1. An input token enters the current network layer;
    2. The router calculates matching scores between the token and each expert;
    3. A small number of experts are selected according to the routing strategy;
    4. The token is dispatched to the selected experts;
    5. The expert outputs are combined with weights;
    6. The result continues into subsequent attention layers and network layers.

    If each token uses only 16 routed experts, plus shared experts, the parameters actually involved in computation are far fewer than the full set of expert parameters. This design can expand model capacity while controlling per-token computation.

    However, sparse activation does not mean hardware requirements are low. The inference system still needs to store all expert weights, because different tokens may access different experts. In other words, computation is reduced, but VRAM capacity and weight-loading pressure remain substantial.

    3. The Role of Shared Experts

    Shared experts are an important part of the MoE architecture. Regular routed experts handle more differentiated patterns, such as complex reasoning, code understanding, visual description, or long-text organization. Shared experts provide a relatively stable basic transformation for all tokens.

    Shared experts generally provide three kinds of value:

    First, they improve the stability of basic capabilities. Regardless of which experts the router assigns a token to, the shared portion can provide a consistent underlying representation.

    Second, they help mitigate routing imbalance. If some experts are accessed by too many tokens, computational hotspots and device load imbalance may occur. A shared path can carry part of the general computation and reduce the fluctuation caused by relying entirely on routed experts.

    Third, they improve training stability. Routing mechanisms are prone to expert collapse, where a large number of tokens concentrate in a few experts. Shared experts provide a stable common pathway, but they cannot fully replace load-balancing losses, capacity limits, and dynamic scheduling strategies.

    4. Million-Token Context and Sparse Attention

    The core bottlenecks of long-context models are attention computation and cache size. Standard full attention must process a large number of pairwise relationships between tokens. As sequence length grows, computation and VRAM overhead rise rapidly.

    Compared with its predecessor architecture, the main change in the target model is that some full-attention layers are replaced with sparse-attention layers. Sparse attention does not let every token attend to the entire sequence. Instead, it uses local windows, block connections, selective global connections, and other mechanisms to preserve key dependencies while reducing unnecessary computation.

    The benefits of this approach include:

    • Reducing attention computation for very long inputs;
    • Slowing the growth of the key-value cache;
    • Increasing the manageable length of long documents, code repositories, and multi-turn conversations;
    • Reducing, to some extent, the latency and VRAM pressure of long-context inference.

    However, sparse attention does not mean that inputs of any length will maintain exactly the same quality as short text. For information associations that span extremely long distances, the model may still be affected by connection patterns, positional encoding, retrieval capability, and training data coverage. In practice, it is still advisable to combine segmentation, summary compression, external retrieval, and key-information indexing.

    5. Changes Brought by Native Multimodal Capabilities

    The model processes not only text but also image input. Multimodal models usually include a visual encoding module that converts images into visual tokens the model can understand, which are then sent into the backbone network together with text tokens.

    From an engineering perspective, image input adds three types of cost:

    First, visual encoding cost. The higher the resolution and the more images there are, the greater the number of visual tokens, and the more preprocessing time and VRAM are required.

    Second, context usage. After images are converted into visual tokens, they occupy the model’s context window. Inputting multiple high-resolution images at once can quickly consume the available context.

    Third, cross-modal reasoning cost. The model must associate image regions, textual descriptions, and user instructions at the same time, which usually requires more computation than pure text tasks.

    Therefore, in actual deployment, image resolution should be selected according to the task, and uploading oversized images without a clear purpose should be avoided. For scenarios such as document recognition and table analysis, image cropping, region detection, and structured preprocessing can also be performed first.

    6. What 1.56 TB of Weights Means

    If the model weight file is approximately 1.56 TB, local deployment first faces not a compute problem but storage, VRAM, and weight-loading problems.

    Several capacity concepts need to be distinguished:

    • Weight file size: the disk space required to store model parameters at a given precision;
    • VRAM usage: device memory occupied after the model is loaded, usually including runtime buffers;
    • KV cache: a cache that continues to grow during long-context inference;
    • Communication and scheduling overhead: additional space and bandwidth requirements generated during multi-device collaborative inference.

    Even if the total device VRAM is theoretically larger than 1.56 TB, it cannot simply be assumed that the model will run. The system also needs to reserve space for activations, routing caches, communication buffers, framework runtime, and input/output data.

    The statement that “at least 8 GPUs are required locally” is usually only a rough resource estimate, not an absolute conclusion. The actual number of GPUs depends on per-GPU VRAM, weight precision, whether quantization is used, context length, the number of concurrent requests, and the memory management method of the inference framework.

    For example, lower-precision representations can reduce weight usage but may cause precision loss. Weight sharding can split the model across multiple devices but increases inter-device communication. Shortening context can reduce KV cache pressure but limits long-document processing capability.

    7. Assessments to Complete Before Local Deployment

    Before actual deployment, it is advisable to perform estimates in the following order.

    Step 1: Confirm weight precision. Different numerical formats correspond to different storage requirements. Hardware scale cannot be judged only by parameter count.

    Step 2: Calculate baseline weight capacity. A rough estimate can be made using “number of parameters × bytes per parameter,” then format metadata and shard-file overhead should be added.

    Step 3: Reserve runtime space. Activations, temporary tensors, and communication buffers during inference all require additional VRAM.

    Step 4: Evaluate the context budget. Million-token context significantly increases KV cache size, and short inputs and long inputs may correspond to completely different device requirements.

    Step 5: Determine concurrency goals. The resource model for single-user, low-concurrency use differs from that for multi-user service deployment. As concurrency increases, cache and scheduling pressure usually rise noticeably.

    Step 6: Validate the communication topology. Multi-GPU inference depends not only on total VRAM but also on interconnect bandwidth between devices. If communication efficiency is insufficient, throughput may drop and latency may rise even when VRAM is sufficient.

    8. Suitable Deployment Strategies

    For developers with limited resources, offline inference, short context, and low-concurrency configurations can be prioritized to first verify whether the model meets task requirements.

    For enterprise internal services, mechanisms such as multi-device sharding, request queues, dynamic batching, and VRAM monitoring are recommended. Long-document splitting, image preprocessing, and result caching can be handled before and after the model.

    For high-concurrency scenarios, attention should also be paid to:

    • Whether expert load is balanced;
    • Whether some devices remain hotspots for long periods;
    • Whether long requests block short requests;
    • Whether KV cache supports reclamation and reuse;
    • Whether the system can automatically recover after a model loading failure;
    • Whether size and quantity limits are set for multimodal input.

    9. How to Properly View the Local Runtime Threshold for Ultra-Large Models

    The advantage of ultra-large MoE models lies in their large capacity and broad task coverage, and they can control part of the computation cost through sparse activation. However, they do not automatically become models suitable for ordinary workstations.

    To judge whether local deployment is appropriate, four questions should be considered together:

    1. Is there enough VRAM to hold all weights?
    2. Can the system withstand the cache growth caused by long context?
    3. Do multiple GPUs have stable high-bandwidth communication?
    4. Does the target task really require a model of this scale?

    If the task is only ordinary text question answering, summarization, or lightweight code completion, a smaller model may offer better cost efficiency. The capacity advantages of an ultra-large model are more likely to become apparent only in complex reasoning, very long documents, multimodal understanding, or highly difficult agent tasks.

    Overall, the key value of this type of model is not only parameter scale. It is the combination of “massive total parameters + limited activated parameters + multimodal input + sparse attention” that balances model capability, inference cost, and long-context processing. For developers, understanding these structural differences is more important than focusing only on parameter count or weight file size.