Local Deployment of a 2.8T MoE Model: GPU and Compute Planning

    This article explains the engineering considerations for deploying a 2.8T-parameter Mixture-of-Experts model locally. It covers the model’s 2.8T total parameters and 104B active parameters, the practical impact of 1.56TB weight files, MXFP4/MXFP8 quantization, sparse attention, KV-cache growth, GPU memory and interconnect requirements, license review, and a self-hosted versus cloud decision framework. Readers can use it to estimate hardware needs, identify deployment bottlenecks, and prepare for compliance review before production use.

    1. Introduction: Open Weights Are Not Plug-and-Play

    According to publicly available information, the trillion-parameter-class MoE model discussed here, referred to below as “the model,” has a parameter and weight footprint among the largest currently available in the open-weight model space. Because the weights have been openly released, organizations can evaluate local deployment, private inference, or domain fine-tuning on top of the model.

    However, obtaining the weights does not mean the model is immediately ready for use. There are at least two major thresholds:

    1. Compute threshold: The weight size is about 1.56TB. Local inference typically starts from eight accelerators, and GPU memory, interconnect, inference frameworks, and operations all need to be re-evaluated.
    2. License threshold: The model uses a custom license, usually marked as an “Other” license on the repository page rather than a standard open-source license. Commercial terms need to be reviewed separately.

    To avoid misleading readers, this article does not elaborate on release-day claims or fabricate official links. Model names, repository addresses, page dates, license text, and clause numbers should be verified against the official model card, repository page, or follow-up materials. Where official links cannot be included, the wording here should be treated as based on publicly available information and subject to later official updates.

    This article focuses on three engineering questions:

    • What is inside the released weights?
    • What does the license mean for commercial use?
    • What should be calculated before deployment?

    2. Model Structure: 2.8T Total Parameters and 104B Active Parameters

    According to the model’s publicly available model card and repository page, with final details subject to the official pages, the core structure is as follows:

    Dimension Configuration Engineering implication
    Total parameters 2.8T Determines storage and GPU-memory pressure
    Active parameters per token 104B Determines per-token compute cost and inference speed
    Architecture MoE mixture of experts 896 routed experts, 16 selected per token, plus 2 shared experts
    Layers 93 layers 24 layers use sparse attention
    Context length 1 million tokens Supports long documents, long conversations, and multi-turn agent tasks
    Vision capability Proprietary visual encoding module Supports image input and native multimodal capability

    The most important distinction is this: the 2.8T total parameter count determines “memory and storage pressure,” while the 104B active parameter count determines “single-step compute pressure.”

    MoE models have a key characteristic: they save compute, but they do not save memory. Although only about 104B parameters are activated per token, all expert weights may still need to reside in GPU memory or be loaded quickly during inference.

    A rough estimate can illustrate the relationship. Ignoring additional attention-related overhead, the floating-point operations required to process one token are approximately twice the number of active parameters:

    1. 2 × 104B ≈ 208 GFLOPs/token

    If eight GPUs together provide about 8 PFLOP/s of theoretical compute, and actual utilization is 30%, the theoretical throughput would be approximately:

    1. 8 × 1 PFLOP/s × 30% ÷ 208×10⁹ ≈ 11.5k tokens/s

    This number should only be treated as an order-of-magnitude reference. Actual throughput is affected by GPU model, interconnect bandwidth, framework optimization, batching strategy, and context length. The key point is that a MoE model is not necessarily easy to deploy simply because its active parameter count is small. GPU memory and communication are often the real bottlenecks for local deployment.


    3. Sparse Attention and Quantization: Benefits and Costs of Long Context

    3.1 Sparse Attention: Not Every Layer Needs Full Attention

    Of the model’s 93 layers, 24 layers use sparse attention. Standard attention has computation that grows quadratically with sequence length. If the context is expanded from 4K to 128K, the sequence length becomes 32 times larger, and the attention computation may grow by roughly 1000 times.

    Sparse attention reduces the number of tokens each token needs to attend to. For example, a token may only attend to a local window, a small number of global tokens, or tokens selected by the model as important. This can reduce long-context inference pressure, but it may also introduce tradeoffs:

    • Long-range recall capability may decrease.
    • Key information spread across different parts of a document may be missed.
    • Precise question answering, citation location, and multi-hop reasoning may require additional validation.
    • For high-reliability scenarios, retrieval-augmented generation, RAG, or engineering-side chunking strategies may be needed.

    Sparse attention can be beneficial for long documents, multi-turn agents, and code repository understanding. However, it should not be treated as equivalent to “no quality loss for long context.” Before deployment, teams should prioritize validating long-range recall and key-fact retrieval using their own business data.

    3.2 Quantized Weights: MXFP4 Weights and MXFP8 Activations

    The model uses 4-bit weights and 8-bit activations in micro-scaling floating-point formats, specifically MXFP4 weights and MXFP8 activations. MX formats typically share one exponent across a group of values, while values within the group retain only the mantissa. This trades a small amount of precision overhead for storage compression.

    Such models are usually produced through quantization-aware training, meaning quantization error is included in the optimization objective during training rather than applied only after training is complete. As a result, they usually preserve original capabilities better than post-training quantization approaches.

    However, two issues need attention:

    1. The precision ceiling is locked: If the official release only provides quantized weights, then fine-tuning, distillation, or auditing at higher precision would require dequantization, and information from training may not be fully recoverable.
    2. Local deployment becomes harder: Consumer-grade GPUs have limited hardware support for 4-bit floating-point formats. If the inference framework does not support the native format, it may fall back to a software dequantization path, which can increase memory usage and latency.

    4. Capability Profile: How to Interpret Vendor-Reported Scores

    From publicly available comparison data, the model performs strongly on coding, command-line tasks, software engineering, and agent-style tasks, but still lags behind on difficult cognitive tasks such as scientific knowledge and PhD-level reasoning. The data below is vendor-reported. Actual model selection should be based on evaluations using your own tasks.

    Capability dimension Comparison benchmark, example grouping Model performance
    Difficult command-line task execution A difficult command-line task benchmark, version 4.0 65.7%, slightly below the strongest closed-source flagship at 66.4% and above another closed-source flagship at 58.2%
    Long-horizon software engineering A marathon-style software engineering benchmark 51.7%, about 5 percentage points ahead of the strongest closed-source flagship at 46.3%
    Scientific subtasks Science sub-items in the command-line benchmark above 38.1%, below the comparison value of 63.3%
    PhD-level knowledge and reasoning A difficult knowledge-reasoning benchmark, Diamond subset 39.4%, compared with 60.9%

    The vendor claims that the model outperforms the comparison model on 16 of 37 benchmarks, while the remaining results are comparable or worse. An independent evaluation organization also ranked it fourth globally on a composite intelligence index, but the organization name, report link, and evaluation date should be verified against the original evaluation page.

    Overall, the model’s capability profile can be summarized as follows:

    • Strong at execution: code generation, command execution, multi-step operation, and complex engineering tasks.
    • Weak at deep reasoning: scientific knowledge question answering, high-difficulty logic, and mathematical reasoning.
    • Above average under vendor-reported conditions, but not leading in every dimension.

    For business use, the practical implication is:

    • If your scenario is a coding assistant, automated operations, process automation agents, or developer productivity tools, the model is worth serious evaluation.
    • If your core requirement is rigorous academic reasoning, medical or financial risk control, or complex scientific question answering, it should not be the default choice. Stronger human review and knowledge-enhancement mechanisms should be introduced.

    5. License Interpretation: Open Weights Are Not Open Source

    The model’s license is one of the most important concerns for enterprise adoption. The repository page marks it as an “Other” license, not a mature open-source license. The license text, version date, and clause numbers should be verified against the official repository LICENSE, model card, or release page. This article only interprets the license based on publicly visible labeling. Commercial compliance should be reviewed by legal counsel against the original license text.

    Two common misunderstandings need clarification:

    1. Assuming that downloadable weights mean free commercial use.
    2. Treating open weights as equivalent to open source.

    Under common open-source definitions, open-source licenses cannot restrict the field of use. For example, they cannot state “commercial use is prohibited” or “certain use cases require additional authorization.” In contrast, the model’s license retains restrictions related to commercial scale or specific service models.

    The main grant is broadly MIT-like: use, copying, modification, distribution, fine-tuning, and commercial use are generally permitted in principle. This is relatively friendly for many enterprise scenarios, such as internal tools, data analysis, model secondary development, and private knowledge bases.

    The additional restrictions mainly target two types of large-scale business:

    • Companies that provide model-as-a-service, MaaS, externally and whose cumulative revenue over the past 12 months exceeds a threshold defined in the license may need to sign a separate agreement before commercial use.
    • Commercial products with a monthly active user base reaching a certain scale may also fall within the additional terms.

    The exact thresholds, measurement methods, applicable entities, and exceptions should be confirmed from the original license text. In other words, the “brake” in the license mainly targets a small number of large-scale players, but it still reminds all users that a custom license is not a one-time permanent grant. It is a set of dynamic terms.

    Practical recommendations:

    • Legal pre-review: Include the license file in compliance review and clarify revenue measurement, user-scale measurement, and responsibility boundaries.
    • Preserve notices: When redistributing the model or distributing derivative products trained on it, retain the original license and copyright notices.
    • Maintain a registry: Record model version, download source, checksum, deployment scope, and using departments.
    • Monitor changes: If later versions change the license terms, re-evaluate the compliance of existing deployments.

    6. Deployment Threshold: Why Local Inference Starts at Eight GPUs

    6.1 Converting 1.56TB of Weights into GPU Memory

    The officially published weight size is about 1.56TB, including 96 weight shard files. Before deployment, two concepts need to be distinguished:

    • Disk space: Download, extraction, caching, and keeping multiple versions usually require about 2x space, meaning more than 3TB.
    • GPU-memory usage: After loading onto accelerators, actual usage may exceed the file size. Shards may contain indexes, metadata, and alignment overhead, and the runtime framework may also retain communication buffers and resident memory.

    If each GPU has 80GB of memory, loading 1.56TB would require at least:

    1. 1.56 TB ÷ 80 GB ≈ 20 GPUs

    The “eight-GPU starting point” does not mean that eight 80GB GPUs can easily handle the model. Instead, if 1.56TB is spread across eight GPUs, each GPU needs to carry about 195GB of weight share. Therefore, an eight-GPU setup will most likely require high-end accelerators with 192GiB or more per card, such as 192GiB or 288GB-class cards.

    In addition, 195GB is only the lower bound for weight distribution. Actual GPU memory must also include:

    • Inference-framework resident overhead.
    • Communication buffers.
    • Activations.
    • KV cache.
    • Peak memory usage from concurrent batching.
    • Safety margin for memory fragmentation.

    Therefore, “eight GPUs” is closer to the minimum configuration for getting the model started, not a production configuration for smooth serving.

    6.2 GPU-Memory Estimation Formula

    A simplified formula can estimate the minimum number of GPUs:

    1. Per-GPU memory requirement ≈ weight shard + KV cache + activation and framework overhead + communication buffer
    2. Minimum GPU count ≈ ceil(total requirement ÷ (per-GPU memory × 0.8))

    The 0.8 factor is a recommended safety coefficient used to avoid out-of-memory errors caused by memory fragmentation, sudden peaks, and long-sequence inputs.

    For a rough KV-cache estimate, you can use:

    1. KV cache bytes ≈ 2 × layers × KV heads × head dimension × precision bytes × sequence length

    This article uses 8 KV heads as an illustrative assumption. This is only an example and should be verified against the model card and inference-framework implementation. Assuming 93 layers, 8 KV heads, a head dimension of 128, and an FP16 KV cache:

    • Per token: approximately 2 × 93 × 8 × 128 × 2 ≈ 380KB
    • 32K context: approximately 380KB × 32768 ≈ 12GB
    • 128K context: approximately 380KB × 131072 ≈ 48GB
    • 1M context: approximately 380KB × 1048576 ≈ 380GB

    These values will vary depending on the actual model structure, parallel sharding method, and framework caching strategy. However, they are enough to show that as context length and concurrency increase, KV-cache usage can approach or even exceed the weight footprint itself.

    6.3 Minimum Startup Configuration Examples

    The following configurations are engineering examples, not official recommendations:

    Goal Example GPU count Example per-GPU memory Interconnect requirement Suggested context Example KV cache Notes
    Load and validate 8 GPUs 192GiB or more, or 288GB-class Prefer high-speed single-node interconnect 4K to 32K About 12GB total at 32K Suitable for validating loading and short-context inference
    Stable inference 16 GPUs 141GB or more is more stable High-speed interconnect such as NVLink or InfiniBand 32K to 128K About 48GB total at 128K Requires measured throughput and long-text stability testing
    Long context / high concurrency 24+ GPUs 141GB or more Multi-node high-speed interconnect and parallel strategies 128K to 1M About 380GB total at 1M Higher requirements for TCO, communication, and framework tuning

    If the business requires a 1M-token context or high-concurrency long sessions, KV cache becomes a core cost item. At that point, the question is no longer only “Can the weights be loaded?” but also “Can the model maintain stable throughput under long-context conditions?”

    6.4 Download, Storage, and Distribution

    A 1.56TB weight set is itself an engineering task:

    • Bandwidth: Downloading over a 100 Mbps connection at full speed may take more than 30 hours. A gigabit network may take about 3.5 hours.
    • Checksum verification: After downloading, verify the hash of each file to avoid load failures caused by corrupted shards.
    • Storage: High-speed NVMe or enterprise network storage is recommended, with at least 2x space reserved.
    • Distribution: For internal enterprise distribution, use object storage, internal file services, or caching mirrors to reduce repeated downloads.

    6.5 Inference Framework and Hardware Requirements

    For 4-bit weights to run effectively, the inference framework must support the corresponding micro-scaling format and MoE structure. Otherwise, it may fall back to a dequantization path, increasing memory usage and latency. When selecting a framework, confirm the following:

    1. Native support for MXFP4 weights and MXFP8 activations.
    2. Support for hybrid expert parallelism, EP, and tensor parallelism, TP.
    3. Support for paged KV-cache management similar to PagedAttention.
    4. Support for continuous batching.
    5. Support for multi-node communication and fault isolation.

    On the hardware side, an eight-GPU starting point means that a single machine requires at least eight high-end accelerators, or a multi-node inference cluster must be built. MoE models are sensitive to all-to-all communication. In cross-node deployments, high-speed interconnect directly affects efficiency.


    7. Self-Hosted or Cloud: A Decision Framework

    7.1 Three Decision Paths

    Scenario Recommendation Reason
    Sensitive data and high compliance requirements, such as finance, healthcare, or government Private self-hosting Data stays inside the private network, making auditing and access control easier
    Stable long-term usage with existing GPU resources Self-hosted inference service Per-token cost may be more controllable, and customization can be deeper
    Highly variable demand or rapid validation phase Elastic cloud inference Pay as you go and avoid large upfront capital investment

    7.2 Cost-Oriented Comparison

    Comparison dimension Local self-hosting Cloud deployment
    Initial investment High; requires purchasing eight or more GPUs Low; pay-as-you-go pricing
    Elastic scaling Weaker; longer expansion cycles Stronger; can scale in minutes
    Data security Strong; data can remain inside the private network Depends on cloud security configuration
    Operations complexity High; involves drivers, frameworks, and fault handling Lower; managed services may cover more
    Suitable scenarios Stable high-frequency use and strong compliance Variable traffic and rapid launch

    7.3 TCO for Self-Hosting and Cloud Billing Inputs

    Self-hosting should not be evaluated by GPU purchase price alone. A total cost of ownership, TCO, model should be built:

    1. Monthly self-hosting cost ≈ GPU depreciation + electricity + data-center and network costs + operations labor + fault-redundancy costs

    Where:

    1. GPU depreciation ≈ total purchase cost ÷ depreciation months
    2. Cost per token ≈ monthly self-hosting cost ÷ daily effective token volume ÷ 30

    Cloud costs can be broken down as:

    1. Monthly cloud cost ≈ base compute/model-service fee + peak elasticity premium + storage and network fees + data migration costs
    2. Cost per token ≈ monthly cloud cost ÷ daily effective token volume ÷ 30

    When comparing self-hosting with cloud deployment, at least the following variables should be included:

    • Daily token volume.
    • Peak QPS.
    • Average input and output length.
    • Context length.
    • GPU utilization.
    • Whether 7×24 availability is required.
    • Data compliance requirements.
    • Team operations capability.
    • Whether the license triggers additional agreement requirements.

    7.4 A Practical Recommendation

    If the business scale has not reached the commercial threshold in the license terms and the team lacks GPU-operations experience, it may be practical to first validate effectiveness through mainstream cloud platforms or model services before deciding whether to self-host. Mainstream cloud platforms usually offer multiple forms, from model services to GPU compute instances, which can help teams reduce upfront investment risk. This article only describes deployment forms and does not constitute a commercial recommendation for any single brand.


    8. Conclusion: Calculate First, Deploy Second

    The value of these weights is not simply whether they can be downloaded. It is that they put a practical question in front of every team: when model scale enters the trillion-parameter class, open weights do not mean everyone can use the model immediately. They mean the model is usable by teams that can measure the requirements, deploy it compliantly, and operate it continuously.

    Returning to the original questions:

    • What is it? A MoE model with 2.8T total parameters and 104B active parameters. It is relatively strong in coding and agent tasks, but weaker in scientific knowledge.
    • Can it be used commercially? The main license is broadly friendly, but additional terms apply to large-scale businesses and require legal review of the original text.
    • Can it run locally? The weights are about 1.56TB. Eight GPUs are closer to the minimum startup threshold. Per-GPU memory generally needs to be 192GiB or more, or 288GB-class, along with high-speed interconnect and a mature inference framework.

    For most teams, a more rational path is to first evaluate license and business fit, then validate effectiveness with small-scale real data, and finally weigh cost, compliance, operations capability, and elasticity requirements before deciding whether to self-host. Models will continue to evolve, but the engineering habit of “calculate first, deploy second” is the fundamental way to handle compute thresholds.