Local Deployment of Ultra-Scale Open-Weight MoE Models: Architecture, Memory Budget, and License Compliance
A practical guide to locally deploying a 2.8T-parameter (104B active) open-weight MoE model. It breaks down the model architecture, shows how to budget VRAM across 8, 16, or 32 GPUs, compares deployment options, and walks through license compliance and engineering checklists for teams evaluating local deployment.
In early November, an AI company delivered on a promise made a month earlier by publicly releasing an open-weight large model with 2.8T total parameters and 104B active parameters, with weights totaling roughly 1.56TB. The officially stated threshold for local deployment is “8 GPUs minimum.” Now that the weights are available, three questions deserve serious attention: How is the model built internally? Is the license a standard open-source license or custom terms? And how should developers and enterprises plan their local deployment? This article covers five dimensions—architecture, VRAM budgeting, deployment options, license compliance, and engineering practice—and provides reusable estimation methods and checklists.
1. Event Recap: From “API First” to “Weights Land”
1.1 Timeline
The model’s release unfolded in three stages.
- Capabilities first: On October 24, the model went live as an API and web chat. The team also promised that the full weights would be released before November 4.
- Preparation: On November 2, the official repository for hosting the weights was created, but it remained closed as of the morning of November 4.
- Weights land: On the evening of November 4, the repository officially opened for download, with no application or review required.
The weights consist of 112 files, of which 96 are sharded weight files, totaling about 1.56TB. Only 11 days passed between the promise and delivery, keeping the wait within an acceptable range.
Releasing weights in sharded form is favorable for post-download integrity verification. Running a hash check on each shard lets you catch transfer corruption before loading, avoiding hidden failures like “the model starts but produces abnormal output.”
1.2 Why This Release Is Worth Discussing
Most models with strong general capability are currently offered as closed-source APIs. There are plenty of open-weight models, but the largest among them typically top out in the hundreds of billions of parameters. Releasing weights of 1.56TB and 2.8T total parameters turns local deployment of a trillion-scale model from a thought experiment into a task that demands real calculation.
A note on framing: based on publicly available vendor information, this is one of the largest models by parameter count currently available as open weights. This conclusion should be confirmed against the official release notes.
2. Architecture Breakdown: How 2.8T Parameters Are Structured
2.1 Total vs. Active Parameters: The Core Logic of MoE
The model has 2.8T total parameters, but processes each token by activating only 104B—about 3.7% of the total. This “large total, small active” characteristic is the signature of a Mixture-of-Experts (MoE) architecture. Its compute logic can be understood as a division of labor across two layers.
- Routing layer: A lightweight gating network selects the appropriate experts for each token. This model picks 16 from 896 routing experts, plus 2 shared experts that handle the universal computation every token passes through.
- Expert layer: Large in number but mutually independent, each expert can be understood as a branch of a feed-forward network. The 18 experts that ultimately participate (16 routing plus 2 shared) account for the bulk of the parameter computation.
MoE has two direct consequences.
- Compute–memory decoupling: Compute cost depends mainly on the 104B active parameters, while memory footprint depends on all 2.8T parameters. This is the key to understanding “8 GPUs minimum,” and Section 4 works through the full derivation.
- Load-balancing challenge: Whether routing is even determines how well the experts on each device are utilized. In an open-weight scenario, training-time information such as routing statistics is usually not shipped with the weights. The deployer can only rely on benchmarks and stress tests to observe whether long-tail experts become a bottleneck.
2.2 Layer Structure and Sparse Attention
The model has 93 layers. Compared with the previous generation, 24 layers switched their attention computation from full attention to sparse attention (DSA).
- The full-attention bottleneck: Each token must attend to all previous tokens, so attention computation grows quadratically with sequence length. When context expands from thousands of tokens to a million, the attention share rises sharply.
- The sparse-attention idea: Each token attends only to a small selected subset of history—for example, a recent window, globally important tokens, or segments chosen by scoring—bringing computation down to roughly linear.
- The hybrid-attention tradeoff: Using sparse attention everywhere would sacrifice the ability to capture distant information. Keeping some layers full-attention and others sparse-attention is a common way to balance efficiency and capability. For the exact inter-layer arrangement and sparsity patterns, refer to the official model card.
Note: Sparse attention primarily reduces attention computation. How much VRAM the KV cache can save depends on the specific sparse implementation and cannot be assumed in general.
2.3 Native Multimodality and Million-Token Context
The model natively supports multimodality, with a built-in vision encoder that can understand image inputs directly, without attaching a separate vision component. Its context window is 1 million tokens.
Keep in mind that on consumer-grade hardware, a million-token context can rarely be fully utilized. The longer the context, the higher the attention-related VRAM usage and compute latency.
2.4 Low-Bit Weights: MXFP4 Weights and MXFP8 Activations
Weights use a 4-bit microscaling floating-point format (MXFP4) and activations use an 8-bit format (MXFP8)—the product of quantization-aware training (QAT). “Microscaling” means data is grouped into small blocks that share a scaling factor each, trading a very small overhead for a wider dynamic range.
- Why this combination: Weights are static and can be compressed more aggressively; activations are dynamic, and 8-bit preserves more numerical precision.
In practice, at a 1.56TB storage footprint there is only a single officially released weight version, with no higher-precision variant. For evaluators, this means it is not possible to compare quality differences across precisions on the same model. Only a release with richer per-element types would make fine-tuning and research reproduction more convenient. This is an important consideration during selection.
3. How 1.56TB Fits on 8 Cards: The Correct Way to Budget VRAM
3.1 Order-of-Magnitude Estimation of Weight Size
Before deployment, you can do a rough calculation first. At 4 bits, each parameter takes 0.5 bytes, so the theoretical size of 2.8T parameters is about:
2.8×10^12 × 0.5 Byte = 1.4×10^12 Byte ≈ 1.4 TB
Add the overhead of one scale-factor byte shared per 32 elements, plus parameters stored at higher precision—attention, vocabulary embeddings, the vision encoder, and normalization layers—and a total in the 1.5–1.6TB range is reasonable. This matches the officially reported 1.56TB.
3.2 VRAM Must Be Budgeted in Fast Memory
A common mistake when estimating VRAM is to count only the weights and ignore runtime overhead. A complete VRAM budget should include four parts.
- Weights: all model parameters.
- KV cache: the Key and Value vectors for every processed token at each layer, growing linearly with context length and concurrent requests, commonly managed in pages.
- Activations and temporary buffers: intermediate results during the forward pass, tied to batch size, sequence length, and the parallelism strategy.
- Framework and communication overhead: the runtime’s own footprint, plus the communication buffers allocated for tensor-parallel all-reduce and expert-parallel all-to-all.
Of these four, only the weights can be solved by “spreading thin.” The rest grow with concurrency and context length.
3.3 Parallelism Strategies: How to Spread the Weights Across GPUs
| Strategy | What is sharded | Main communication | Suitable scenario | Main risk |
|---|---|---|---|---|
| Tensor Parallelism (TP) | Weight matrices within a single layer, split by row or column | Multiple all-reduces per layer; frequent communication | High-speed interconnect within a node | High communication share; diminishing returns as scale grows |
| Expert Parallelism (EP) | Expert groups distributed across devices | Two all-to-alls (dispatch/combine) | Naturally suited to MoE models | Severe load skew when routing is uneven |
| Pipeline Parallelism (PP) | Layers split into stages | Point-to-point transfer between adjacent stages | Reduces cross-machine communication pressure in multi-node deployments | Pipeline bubbles lower utilization |
| Data Parallelism (DP) | Replicate the full model; split requests | Essentially no model communication | Scaling throughput once a single machine can host the model | VRAM overhead grows proportionally |
In the “8 GPUs minimum” configuration, the usual combination is tensor parallelism plus expert parallelism: tensor parallelism splits the computation inside each expert, while expert parallelism distributes the 896 experts evenly across cards. Each increment of parallelism adds another round of communication that requires global synchronization.
3.4 Where the Remaining Budget Goes After Spreading the Weights
| Config | Per-card weight usage | Remaining VRAM per card (assuming 200GB/card) | Assessment |
|---|---|---|---|
| 8 cards | ~195GB | ~5GB | Barely boots; weak long-context and concurrency capability |
| 16 cards | ~97.5GB | ~102GB | Ample VRAM headroom; supports longer contexts |
| 32 cards | ~48.8GB | ~151GB | Supports high concurrency and long contexts; cost rises significantly |
This table explains two key conclusions.
- 8 cards is the floor for “getting it running”: it meets the minimum requirement for loading the weights, not the threshold for “running comfortably.”
- The cost structure changes as scale grows: 16- or 32-card configurations have ample VRAM, but cross-machine communication costs rise noticeably. The investment is only worthwhile when the business genuinely needs high concurrency or ultra-long contexts.
3.5 A Reusable Estimation Script
# VRAM budget estimate: weight spreading + runtime headroomTOTAL_PARAMS = 2.8e12 # total parameter countBITS_PER_WEIGHT = 4 # weight bits (4-bit)SCALE_OVERHEAD = 1.10 # empirical margin for scale factors and higher-precision paramsweight_bytes = TOTAL_PARAMS * BITS_PER_WEIGHT / 8 * SCALE_OVERHEADprint(f"Weight size ~ {weight_bytes / 1e12:.2f} TB") # ~1.54 TBfor cards in (8, 16, 32):per_card = weight_bytes / cardsprint(f"{cards:>2} cards: weight per card ~ {per_card / 1e9:.1f} GB")# With 200GB per card, remaining VRAM is used for KV cache and activationscapacity = 200e9for cards in (8, 16, 32):rest = capacity - weight_bytes / cardsprint(f"{cards:>2} cards: headroom per card ~ {rest / 1e9:.1f} GB")# Empirical runtime margin: varies with context length, concurrency, and parallelism; calibrate via testingRUNTIME_HEADROOM = 0.15need = weight_bytes / 8 * (1 + RUNTIME_HEADROOM)print(f"8-card recommended per-card capacity >= {need / 1e9:.0f} GB") # ~222 GB
3.6 Two Hidden Bottlenecks Beyond VRAM
KV cache: With 8 cards, each has only a few GB of headroom. It is barely usable at limited concurrency and moderate context lengths; once you extend the context or raise concurrency, it quickly becomes the bottleneck.
# KV cache estimate: per-request usage, in bytes# kv_heads and head_dim come from the model card; bytes_per_elem is the KV storage precisionkv_per_token = 2 * 93 * kv_heads * head_dim * bytes_per_elem # 2 for K and Vtotal = kv_per_token * seq_len * batch_sizeprint(f"Context {seq_len} x concurrency {batch_size}: KV cache ~ {total / 1e9:.1f} GB")
Storage and network: A 1.56TB weight set has to be downloaded and loaded repeatedly, and bandwidth directly determines startup speed. A single instance’s memory cannot hold all the weights, and multi-node deployment must also account for parameter synchronization and failure recovery. It is advisable to place the weights in centralized storage such as object storage, combined with mirror synchronization and multi-replica distribution, to avoid re-downloading on every scale-out.
4. What “Running Locally” Actually Means: Four Deployment Options
4.1 Four Tiers by Level of Investment
| Option | Typical hardware | Who it suits | Core limitation |
|---|---|---|---|
| Public inference service | No own hardware | Individual developers, prototype validation | Data must leave your environment; cost scales linearly with usage |
| Single or dual GPU | Consumer or professional graphics cards | Personal study, small-scale validation | VRAM usually 24–80GB; only small-parameter models fit |
| 8-card single node | 8 high-capacity-VRAM cards in one node | Enterprise intranet private deployment | Minimal VRAM headroom; weak concurrency |
| 16 cards and above | Multi-node, multi-card high-speed interconnect | Teams with large-scale inference needs | Cross-machine communication and operational complexity rise sharply |
4.2 Decision Criteria: Three Accounts to Calculate
- The VRAM account: 2.8T total parameters is a hard constraint that cannot be bypassed by “shrinking the model”; it can only be spread thin by adding cards or lowering precision.
- The concurrency account: The real concurrency of an 8-card setup is often far below that of a dense model with equivalent VRAM. In long-context scenarios, the KV cache consumes the headroom first.
- The staffing account: Multi-node deployment requires solving parallel sharding, communication tuning, failure recovery, and monitoring/alerting. This effort is often severely underestimated.
4.3 Compromise Options: Choosing an Option by Scenario
If you are concerned about data leaving your environment but do not want to bear the operational burden of an 8-card machine room, private deployment is an option. Place the weights in object storage, spin up inference services elastically through a container platform, and add logging plus monitoring/alerting for end-to-end observability. This satisfies compliance requirements that keep data in-house while avoiding permanently occupying your own hardware.
5. License Compliance: Open Weights ≠ Open Source
This is a part of the event that must be discussed on its own. The model’s weights are released under a custom license, and the license label on the hosting platform is marked “other”—neither a standard open-source license like MIT or Apache-2.0, nor a common standard template.
5.1 The Typical “MIT-Style Grant + Additional Terms” Structure
According to the official statement, the license’s main body is an MIT-style grant: it permits free use, copying, modification, distribution, fine-tuning, and commercial use. At the same time, there are several additional terms. In the open-weight domain, these terms usually focus on two directions.
- Attribution and naming requirements: When distributing or publishing a derivative model, you must retain the copyright and license notices; some terms also restrict a derivative from reusing the original name or brand identity.
- Acceptable use policy: Constraints on the legality and safety of outputs, requiring the deployer to establish content-safety mechanisms.
Note: For the exact content of the additional terms, refer to the official license text and verify item by item; do not infer from experience.
5.2 Four Key Differences from Standard Open-Source Licenses
| Dimension | Standard open-source license (e.g., MIT, Apache-2.0) | Custom license |
|---|---|---|
| Term certainty | Legal text tested over long-term practice | Requires item-by-item interpretation; boundaries depend on exact wording |
| Automated compliance scanning | Accurately recognized by common tools | Often flagged as “other”; needs manual review |
| Additional obligations | Usually attribution, notice retention, patent grants, etc. | May include usage-scenario restrictions, naming restrictions, feedback requirements, etc. |
| Derivative-model distribution | Generally no extra restrictions | May require labeling “based on [model]” or similar statements |
5.3 Four Checks Before Going Live
- Commercial-use scope: Confirm whether commercial scenarios are permitted and whether there are limits on user scale, industry, or output purpose.
- Distribution and attribution obligations: Internal deployment usually does not constitute “distribution,” but packaging the model into a product for external delivery—or letting customers download the weights—requires fully fulfilling attribution and notice-retention obligations.
- Derivative-model naming: If a fine-tuned model is released externally, comply with naming and labeling requirements to avoid confusion with the official model.
- Acceptable use policy: Clarify which outputs are prohibited, implement input/output review and blocking at the gateway layer, and retain audit logs.
5.4 An Easily Overlooked Detail
The “other” license label means common automated license-scanning workflows cannot produce a definitive conclusion. If an enterprise already has a mature license-allowlist mechanism, it will typically need to run a separate manual legal review for such models and record the review conclusion in the artifact-admission checklist, rather than re-judging it in every project.
6. Engineering Checklist: From “Can It Run” to “How to Run It Well”
6.1 Pre-Launch Checklist
- Verify the hash of each shard after download to confirm the weights are complete and uncorrupted
- Run the minimal case with the official inference configuration and confirm output matches the official examples
- Do a quality comparison on real business samples, not just benchmark scores
- Stress-test different concurrency levels and context lengths; record throughput, time-to-first-token, and VRAM usage
- Clarify the boundaries of license obligations; complete the legal review and keep it on file
- Configure input/output review, audit logging, and call quotas
- Define degradation and circuit-breaking strategies for VRAM overflow, node failure, and request queuing
6.2 Key Parameters for the Inference Service
Below is an illustrative inference configuration; actual values should follow the official documentation:
model:name: open-weight-moe-2.8tweight_bits: 4 # 4-bit weightsactivation_bits: 8 # 8-bit activationsmax_context: 1048576 # model's upper limit; scale down as needed at deploy timeparallel:tensor: 8 # tensor parallelism degreeexpert: 8 # expert parallelism degreepipeline: 1serving:max_num_seqs: 8 # max concurrent sequences; keep low on an 8-card setupgpu_memory_utilization: 0.90enable_chunked_prefill: true # reduce TTFT jitter in long-context scenarioskv_cache_dtype: auto # may quantize to lower bits to save VRAM as needed
On an 8-card setup, concurrency should not be set too high. With only single-digit GB of VRAM headroom, excessive concurrency will directly trigger VRAM overflow or request queuing. A safer approach is to explicitly set a concurrency limit and queue timeout, and to rate-limit long-context requests separately.
6.3 Notes on Evaluation
Officially published comparison data should be viewed rationally. A few suggestions.
- Scores are self-reported by the vendor: Prefer third-party independent evaluations or your own business samples for verification.
- Single-task vs. aggregate differences: This model’s results show a clear structural pattern—coding and task-execution tests are near or slightly below leading closed-source models, with leads on some tasks, while gaps are pronounced on scientific and some reasoning tests. This shows the capability distribution is uneven, so select by matching business scenarios rather than looking only at an overall ranking.
- Limits of leaderboard rankings: Composite indices like third-party intelligence indices typically weight and aggregate multiple capabilities, so their reference value is limited. A more reliable approach is to build an evaluation set close to real business needs and compare models using the same inputs.
7. Conclusion: Three Judgments
Placing this weight release back in an engineering context, three judgments can be distilled.
- The value of open weights lies in controllability: Data stays in-house, costs are predictable, and behavior is auditable—things an API form cannot replace. But “open weights” does not mean “works out of the box”; the hardware threshold and operational complexity are real costs.
- The license is a prerequisite for selection, not an afterthought: A custom license may impose additional obligations around commercial use and distribution. Legal review should proceed alongside technical selection, not be patched in before launch.
- 8 cards is a starting point, not the finish line: 2.8T total parameters means the weights must be spread thin across multiple cards; an 8-card setup meets the minimum requirement to “get it running,” while concurrency and long-context capability depend on the remaining VRAM headroom. VRAM budgeting and concurrency stress testing should become mandatory steps for deploying this class of model.
For developers, the real work lies not in “whether it can be downloaded” but in “how to calculate the budget, run the service stably, and hold the line on compliance after downloading.”
