Large Open-Weight MoE Model: A Practical Guide to Local Deployment and Licensing

    This guide walks through what comes after you download a 1.56TB open-weight mixture-of-experts (MoE) model: what is inside the weights, how the license works, and whether you can run it on your own hardware. It breaks down the 2.8-trillion-parameter architecture (sparse experts, sparse attention, and MXFP4 weights), explains how the custom license permits most use cases while imposing revenue and MAU thresholds on large-scale commercial services, and shows how to estimate VRAM, choose a parallelism strategy, and decide between cloud and self-hosted deployment.

    Large Open-Weight MoE Model: A Practical Guide to Local Deployment and Licensing

    Overview

    A model team recently launched its next-generation flagship model first as an API and a web app. At launch, the team promised to release the full weights before the start of November. On the agreed date, the weights appeared on the major model-hosting communities as promised: no application required, no approval gate.

    The weight release consists of more than 100 files, of which over 90 are weight shards, totaling about 1.56TB. The community and vendors quickly began offering online demos, and the open-source buzz took off.

    By sheer size, this is the largest open-weight model released to date; both its parameter count and file size far exceed any comparable model before it. But there is a gap between “the weights can be downloaded” and “you can run it yourself and use it with confidence,” and that gap spans architecture, licensing, and compute. This article focuses only on three questions that come after the weights land:

    • What is inside?
    • How does the license work?
    • Can an ordinary user run it?

    How the 2.8T parameters are built

    Sparse experts: you compute fewer, but you store them all

    The model card describes this as a natively multimodal agent model, continued-trained from the previous generation of the same family. The core parameters are:

    • Total parameters of about 2.8 trillion (2.8T), with only about 104 billion (104B) parameters activated per token;
    • A mixture-of-experts (MoE) structure with 896 routed experts per layer, from which each token selects 16, plus 2 shared experts that are always active;
    • 93 layers with a context length of 1 million tokens;
    • A built-in vision encoder that provides image understanding.

    In other words, a single token’s forward pass touches only about 18 experts (16 routed plus 2 shared). By parameter count, the activation ratio is roughly 104B ÷ 2.8T ≈ 3.7%.

    There is a point that is easy to confuse here: MoE saves compute, not memory. Routing depends on the input, and any expert can be selected, so all experts must be loaded into memory during inference.

    Think of a hospital with 896 departments where each patient is referred to only 16 of them, yet the hospital cannot open just those 16 departments. The model is the same: not a single one of the 2.8T parameters can be dropped.

    Attention sparsification: saving compute for long context

    Compared with the previous generation, the architecture changes in only one place: 24 of the layers switch from full attention to sparse attention (DSA), while the rest keep full attention unchanged.

    This “local replacement” is highly targeted. In regular attention, every token computes associations with all previous tokens, and the cost grows quadratically with sequence length. After converting part of the layers to sparse attention, the computation for long-context scenarios drops noticeably while the main structure of the model stays stable. The tradeoff is that the sparse pattern introduces extra routing overhead, so the real benefit must be measured at the target context length.

    Weight format: only a 4-bit version

    The official release ships only one version: MXFP4 weights (a microscaled 4-bit float) with MXFP8 activations, both produced through quantization-aware training. Common higher-precision formats such as BF16 and FP8 are not released. This directly determines the deployment approach:

    • VRAM usage stays relatively manageable, but almost every step runs at 4-bit precision;
    • The hardware and inference framework must natively support the microscaled format; otherwise you must dequantize online, which raises both VRAM and time costs significantly;
    • To run at higher precision, you can only dequantize the already-quantized weights. This neither restores the original precision nor reduces the VRAM demand, which grows severalfold.

    A quick mental check: a microscaled format typically shares one 8-bit scale factor per 32 weight elements, which works out to about 4.25 bits per parameter. 2.8T × 4.25 ÷ 8 ≈ 1.49TB, and adding the unquantized parts roughly matches the 1.56TB file size. You can start your VRAM planning from this number.

    Capability profile: the strengths and the weaknesses matter equally

    Start with the comparison data from the official model card. All of it is vendor-reported:

    Evaluation dimension This model Comparison top closed-source model
    Terminal-task comprehensive evaluation 65.7% 66.4%
    Long-horizon software engineering tasks 51.7% 46.3%
    Scientific terminal tasks 38.1% 63.3%
    High-difficulty knowledge Q&A 39.4% 60.9%

    The vendor states that 16 of 37 benchmarks beat the comparison model. A third-party evaluation platform also gives an overall intelligence index of 55, ranking fourth globally.

    Read this table in three steps:

    1. The strengths are in agents and engineering. The lead on long-horizon software engineering is about 5.4 percentage points, and terminal tasks are roughly on par with the top closed-source model, showing an edge at “chaining tool calls together to break down and execute multi-step tasks.”
    2. The weaknesses are in science and reasoning. Scientific terminal tasks trail by 25.2 percentage points, and high-difficulty knowledge Q&A trails by 21.5 percentage points, both clear gaps.
    3. The win/loss counts line up. The vendor’s “16 wins out of 37” is consistent with the split between a strengths region and a weaknesses region, so the conclusion has to be read per dimension.

    Keep in mind that all of this data is vendor-reported. Benchmark selection, prompt wording, and evaluation settings all affect the results, so it should not be treated as an independent conclusion. When choosing a model, sample from your own real tasks and build a small evaluation set of 100 to 300 items, comparing with unified prompts and criteria, with a focus on these scenarios:

    • Long-horizon tool-call tasks: whether the model keeps its goal from drifting across many consecutive steps;
    • Private-domain Q&A: whether it answers professional questions accurately;
    • Multimodal input: whether it meets the bar on chart, screenshot, and other image understanding.

    Licensing: open weights are not the same as open source

    This is the point most worth discussing on its own. The weight repository carries a custom license, marked as “other” (license: other) on the hosting platform, rather than a standard open-source license like MIT or Apache 2.0. This license is identical word for word to the previous generation’s, with only the name changed, which indicates a long-term strategy rather than a temporary arrangement.

    Structurally, the main body is indeed a permissive, MIT-style grant: you are free to use, copy, modify, distribute, and use commercially, and you can fine-tune it and build secondary development on top of it. On top of that, two restrictions apply only to large-scale external business:

    1. A company that offers the model as a service (MaaS) externally and has cumulative revenue exceeding $20 million over the last 12 months must sign a separate agreement with the model owner before commercial use;
    2. Once a commercial product’s monthly active users exceed a certain scale (on the order of millions), a separate authorization is also required.

    Two readings:

    • Most teams are unaffected. Individual research, internal enterprise use, and small- to mid-scale commercial products all fall within the permissive grant. Both restrictions have clear thresholds and target ultra-large-scale commercial services.
    • Top vendors must communicate early. Cloud or SaaS vendors with larger revenue, if they plan to offer model services externally, should start business discussions early and not leave the license review until the eve of launch.

    For implementation, three things are recommended:

    • Distinguish “open source” from “open weights” in your documentation and marketing to avoid compliance risk;
    • Pin the model version and the license text in your dependency list, and re-run the review when the model is upgraded;
    • Before offering services externally, check both thresholds, including the revenue measure (12-month cumulative) and the MAU measure (whether it is counted per product or per model service). If the wording is ambiguous, confirm directly with the model owner.

    Local deployment: 8 GPUs are just the starting point

    The VRAM math: why “8 GPUs” is not generous

    The phrase “you need 8 GPUs to run it locally” makes the barrier sound low, but the real constraint is how much VRAM each card has. Using the rough script in section 5.2 (weights only, no KV cache):

    Per-card VRAM 8-card total Can it hold the ~1.4TB of weights?
    80GB class ~0.6TB No
    141GB class ~1.05TB No
    192GB class ~1.43TB Barely, with almost no headroom
    • With the most common 80GB-class accelerators, you need at least 16 cards to start;
    • With 141GB-class cards, a single card cannot hold it, so you need multi-card parallelism or expert offloading;
    • 192GB-class high-end accelerators are a scarce resource; 8 cards can run it, but very little room is left for KV cache and activations.

    A VRAM estimation script

    Before deploying, a simple script to work out the math can help you avoid buying or renting compute blindly. The parameters in the code below can be adjusted to match the actual model card:

    1. # Note: This script estimates from the parameter scale in public information; rely on the model card and testing for actual values.
    2. # Output unit is GiB, where 1 GiB = 1024^3 bytes.
    3. TOTAL_PARAMS = 2.8e12 # Total parameter count
    4. BITS_PER_PARAM = 4.25 # 4-bit weights + an 8-bit scale factor per 32 elements
    5. GiB = 1024 ** 3
    6. # KV cache uses the common "GQA shared KV heads" structure for illustration:
    7. # 93 layers, 8 KV heads, head_dim 128, KV stored in 8-bit
    8. kv_bytes_per_token = 2 * 93 * 8 * 128 * 1
    9. weight_gib = TOTAL_PARAMS * BITS_PER_PARAM / 8 / GiB
    10. print(f"Weight VRAM estimate: ~{weight_gib:.0f} GiB")
    11. for gpus, per_card_gb in [(8, 80), (8, 141), (8, 192), (16, 80)]:
    12. capacity = gpus * per_card_gb * 1e9 / GiB
    13. status = "fits" if capacity >= weight_gib else "insufficient"
    14. print(f"{gpus} GPUs x {per_card_gb} GB: total ~{capacity:.0f} GiB, {status}")
    15. for ctx in (131072, 1048576):
    16. kv_gib = kv_bytes_per_token * ctx / GiB
    17. print(f"Context {ctx:>7} tokens: KV cache ~{kv_gib:.0f} GiB")

    The output looks like this:

    1. Weight VRAM estimate: ~1385 GiB
    2. 8 GPUs x 80 GB: total ~596 GiB, insufficient
    3. 8 GPUs x 141 GB: total ~1050 GiB, insufficient
    4. 8 GPUs x 192 GB: total ~1430 GiB, fits
    5. 16 GPUs x 80 GB: total ~1192 GiB, insufficient
    6. Context 131072 tokens: KV cache ~23 GiB
    7. Context 1048576 tokens: KV cache ~183 GiB

    Three notes:

    • VRAM is labeled in decimal. Vendors usually quote VRAM in GB (10^9 bytes), while the script converts everything to GiB before comparing, avoiding a misjudgment of about 7%;
    • KV cache assumptions. The KV cache in the script is only a structural illustration. Whether a given model uses shared KV heads (such as GQA) or low-rank compressed KV can change the result by several times, so go by the model card;
    • Reserve runtime overhead. Leave another 10% to 15% for activations, temporary buffers, and VRAM fragmentation. An 8-card 192GB configuration has a theoretical capacity of about 1430 GiB, only about 45 GiB more than the weights; after KV cache and runtime overhead, it is essentially at full load, so concurrency and context are both limited.

    So “8 GPUs” is closer to a lower bound: it can start, but it is hard to use comfortably.

    Parallelism strategy

    When a single card cannot hold all the weights, you combine several parallelism approaches:

    • Tensor parallelism (TP): splits each layer’s matrices across cards by row or column, with frequent intra-layer synchronization; it suits the high-speed interconnect inside a single machine.
    • Expert parallelism (EP): distributes the 896 experts across different cards so each card stores only some of them. This is a common approach for MoE models.
    • Pipeline parallelism (PP): splits by layer segments across different nodes with low communication, but it introduces pipeline bubbles that need enough batching to fill.

    MoE expert routing is essentially a “dispatch tokens” collective-communication operation. Which card a token goes to is decided by the routing result, so inter-card interconnect bandwidth directly becomes a throughput bottleneck. A common setup uses EP together with TP inside a node and PP across nodes, and tries to keep one inference instance within the same high-speed interconnect domain.

    There is also a “fewer cards” route: expert offloading. Keep inactive experts in host memory and swap them in on demand to run inference with fewer cards. Host memory should be significantly larger than the weight size and paired with high-bandwidth memory channels; the cost is higher latency and lower throughput. It suits low-frequency, offline, or internal-tooling scenarios.

    Context, concurrency, and performance acceptance

    • Context. The official figure is 1 million tokens, but the usable length is constrained by the KV cache. The table shows about 23 GiB for a 128K context and about 183 GiB for a 1M context, and in multi-card setups this is multiplied by the tensor-parallelism degree. Business rarely needs a million-token context, and setting the service window to 128K or 256K often saves a lot of VRAM.
    • Quantization format. The inference framework must natively support microscaled 4-bit weights, and on load each card’s share of parameters should be an integer multiple of the quantization block size (usually 32); otherwise padding is needed at the boundaries, which hurts efficiency.
    • Acceptance metrics. Don’t just check whether it “can start.” Fix the input length and concurrency, record first-token latency, per-token generation speed, output tokens per second, and cost per token, then compare against a cloud API.

    Cloud or self-hosted

    There are two routes at the compute level, and the choice depends on the shape of your business:

    • Cloud deployment. Flexibly rent GPU cloud servers by card count, keep the model files in object storage, and let a container platform handle startup and scaling, with logging and monitoring to track VRAM, queuing, and errors. Baidu AI Cloud, for example, offers this kind of capability as a combination of GPU cloud servers, object storage, containers, and monitoring. Pay-as-you-go pricing suits teams with spiky traffic, in a validation phase, or needing fast trial and error.
    • Self-hosted deployment. Consider this for long-term, stable workloads where data cannot leave the internal network. Beyond accelerators, you also need to plan high-speed internal networking, host memory on the order of thousands of GB or even TB, and data-center power and cooling. Fixed costs are higher, but the data stays under your control and the unit cost falls as utilization rises.

    A simplified rule of thumb: when average GPU utilization stays steadily above 40% to 50% and the business is long-term, self-hosting is usually more cost-effective; when utilization is chronically low, on-demand cloud is more economical. A real decision should fold in card-hour cost, network and storage fees, and operations labor.

    Conclusion

    This 1.56TB weight release has real value and a real barrier to entry:

    1. Capability. It is close to, and in places ahead of, the top closed-source models on agent and engineering tasks, but has clear weaknesses in science and reasoning. It fits code, operations automation, and multimodal understanding, but is not a replacement for research-oriented models.
    2. Licensing. The main body is permissive, but two thresholds, revenue and MAU, apply to ultra-large-scale commercial services. Top vendors should communicate early, and small- to mid-sized teams can use it with confidence.
    3. Compute. “8 GPUs to start” applies only to 192GB-class high-end accelerators. A more realistic configuration is 16 or more cards, or accepting the performance loss from expert offloading; elastic cloud compute is often the safest choice for a validation phase.

    Open weights put the capability in your hands, but using it well depends on planning ahead for architecture, licensing, and compute. Verify the results on a small evaluation set first, then use a script to work out the VRAM, and finally decide between self-hosting and the cloud. That is the safest path for deploying a model of this scale.