Inside a 2.8T-Parameter Open-Source MoE Model: Architecture and Local Deployment

    This article breaks down the architecture of a 2.8T-parameter native multimodal agent model with about 104B activated parameters, 896 experts, 93 layers, and a 1M-token context. It then examines the engineering challenges of deploying 1.56TB of weights locally, including memory and storage requirements, parallel strategies, quantization, expert offloading, and the practical implications for enterprises and developers.

    Background: Open-Source Models Are Entering the Trillion-Parameter Era

    Over the past few years, open-source community discussions around large models have mainly focused on scales ranging from billions to tens of billions of parameters. Recently, a vendor released the weights of a native multimodal agent model with 2.8T total parameters, with weight files totaling 1.56TB. This marks a point where open weights at trillion-parameter scale have become a reality, while also raising a set of engineering questions for developers and enterprise users:

    • How is a trillion-parameter model organized internally?
    • Why can inference costs remain acceptable despite such a huge total parameter count?
    • What does it actually take to deploy 1.56TB of weights locally?

    This article addresses these questions across three layers: model architecture, compute design, and deployment practice.

    Architecture Breakdown: How 2.8T Parameters Are Composed

    MoE Structure: Large Total Parameters, Small Activated Parameters

    According to the official model card, this model is a native multimodal agent model further trained from a previous-generation version. Its core structure is a sparse mixture-of-experts architecture. Key configurations include:

    • Total parameters: 2.8T
    • Activated parameters per pass: about 104B (the portion actually involved in computation for each token)
    • Expert scale: 896 experts
    • Routing strategy: each token selects 16 experts through the router
    • Shared experts: 2 additional shared experts that are always activated for all tokens

    The value of this design can be understood from two perspectives. First, in terms of representational capacity, 896 experts allow the model to distribute “knowledge” across different subnetworks. Different inputs activate different expert combinations, and the overall parameter scale determines the upper bound of the model’s knowledge capacity. Second, in terms of compute cost, each token only passes through 16 experts plus 2 shared experts, so the actual forward computation is roughly on the order of a 104B dense model. In other words, the model gains knowledge capacity far beyond a 104B dense model while paying inference compute close to that of a hundred-billion-scale dense model. This is the fundamental reason current ultra-large models generally adopt MoE.

    Depth and Context: 93 Layers and a 1M-Token Context

    The model has 93 layers and a context length of 1 million tokens. Ultra-long context is especially important for agent applications: when an agent executes complex tasks, it often needs to include multi-turn tool-call results, long-document retrieval content, and multimodal observations in the context. A million-token window means the model can “remember” the entire task process within a single session without relying on frequent external summarization and compression.

    Multimodal Capability: Vision Encoding Module

    The model natively supports image input and includes an independent vision encoding module that converts images into vector representations the model can understand, which are then fused with the language model backbone. Compared with post-hoc multimodal approaches that bolt on additional modules later, this native multimodal architecture of “vision encoder + language backbone” generally performs better on joint image-text reasoning and is better suited to agent scenarios that require “reading screens, charts, and documents.”

    Key Architectural Change: Sparse Attention in 24 Layers

    Compared with the previous-generation version, the model makes only one structural change: replacing full attention in 24 layers with sparse attention. The motivation deserves elaboration.

    Under full attention, when the sequence length is N, attention computation and memory overhead grow with N squared. When the context reaches one million tokens, quadratic overhead becomes almost unaffordable in engineering terms. The core idea of sparse attention is to let each token compute attention only with a filtered subset of key tokens, rather than pairwise with all historical tokens. This preserves the ability to model long-range dependencies while substantially reducing compute overhead in long-context scenarios. For a model whose core capabilities center on long-document processing and ultra-long sessions, this is a change that must be made for practical deployment.

    Deployment Challenges: What 1.56TB of Weights Means

    Memory and Storage Barriers

    With 1.56TB of total weights and a common per-GPU memory ceiling of 80GB, loading a full copy of the weights alone requires about 20 GPUs. When KV cache, activations, and system overhead are considered, the officially suggested 8-GPU reference configuration generally refers to a multi-node, multi-GPU environment, such as a multi-node cluster. For individual developers, “running locally” at this stage is closer to “running on a cluster.” Practical paths include:

    1. Multi-node inference clusters: connect multiple GPU nodes through high-speed interconnect networks and split the model using a combination of tensor parallelism, pipeline parallelism, and expert parallelism.
    2. Quantization compression: quantize weights from original precision to lower bit widths such as INT8 or INT4. This can significantly reduce memory usage, but precision loss must be evaluated. Quantizing router parameters requires particular caution, because small deviations in routing decisions can affect expert selection.
    3. Hierarchical loading: use high-speed storage and memory tiering to load inactive experts on demand. This is the so-called “expert offloading” strategy and is suitable for offline scenarios that are not latency-sensitive.

    Choosing Parallel Strategies

    For MoE models, parallel strategies are more complex than for dense models and usually require combining multiple partitioning methods:

    • Tensor parallelism: split weight matrices within a single layer. This requires frequent intra-layer communication and depends on high-bandwidth interconnect.
    • Expert parallelism: distribute 896 experts across different devices. After routing, tokens may be transmitted across devices, and the communication pattern depends on the expert distribution of tokens.
    • Pipeline parallelism: split vertically across 93 layers to reduce per-machine memory pressure.

    A simplified deployment configuration example is shown below:

    1. # Illustrative configuration; not real parameters for any specific framework
    2. deployment:
    3. world_size: 16 # Total number of GPUs
    4. tensor_parallel: 2 # Intra-layer partitioning
    5. pipeline_parallel: 2 # Inter-layer partitioning
    6. expert_parallel: 4 # Expert partitioning
    7. precision: "bf16"
    8. kv_cache_policy: "paged" # Paged KV cache management

    In actual deployment, expert load balancing also needs attention: if certain experts are activated frequently, the corresponding devices will face much higher compute pressure than others and may become the bottleneck for overall throughput. Mainstream inference frameworks generally mitigate this through routing load monitoring and capacity factor tuning.

    Implications for Enterprises and Developers

    First, MoE architecture has become a mainstream form for ultra-large models. “Total parameters” and “activated parameters” need to be evaluated separately; activated parameter count is the key metric determining inference cost.

    Second, sparse attention is becoming a standard solution for ultra-long-context scenarios. When selecting models, developers should pay attention to memory behavior on long sequences, not just the claimed upper limit of context length.

    Third, local deployment of trillion-scale open weights remains a high-barrier engineering task. For most enterprises, accessing model capabilities through cloud services or using a managed inference platform is a more practical path. Only teams with strong requirements for data isolation and customization are likely to justify investing in multi-node cluster construction.

    Overall, this release sends a clear signal: the technical details of trillion-parameter models are rapidly becoming public, and deployment, quantization, and inference optimization for very large MoE models will be core topics for the engineering community for some time to come.