Deploying a 1.56 TB Open-Weight Large Model: GPU Memory, Parallelism, and License Compliance
This article examines the public release of a 1.56 TB open-weight multimodal agent model and turns it into a practical deployment guide. It explains MoE total versus activated parameters, GPU memory sizing, KV cache planning, tensor/expert/data parallelism, MXFP4/MXFP8 quantization, sparse attention for 1M-token contexts, benchmark interpretation, and custom license compliance. It also includes a Baidu AI Cloud deployment route and a production readiness checklist.
1. What Happened: A 1.56 TB Weight File
In late October, a model team first released a new model through an API and a web interface, promising that the full weights would be released before November 4. The weights were published as scheduled on a mainstream model hosting platform and could be downloaded without an application process.
The repository was created on November 2. It remained closed on the morning of November 4 and was officially opened in the evening. The repository contained 112 files, including 96 weight shards, totaling about 1.56 TB.
The model card states the following:
- It is a native multimodal agent model, continued from the previous generation.
- It has 2.8T total parameters, with only 104B parameters activated per token.
- The MoE structure uses 896 routed experts, with 16 selected per token, plus 2 shared experts.
- It has 93 layers and supports a 1 million-token context.
- The visual encoder is MoonViT-V2.
- Compared with the previous generation, the structure has only one change: 24 layers of full attention have been replaced with sparse attention, referred to as DSA.
- The weights are in MXFP4 format and the activations are in MXFP8 format, produced through quantization-aware training.
- No BF16 or FP8 versions have been officially released.
According to public data available at the time, this is the largest open-weight model by size to date. For engineering teams, these weights introduce two practical challenges:
- Can we run it? How to fit 1.56 TB of weights into GPU memory.
- Can we use it compliantly? How to interpret the boundaries of a custom model license with additional conditions.
2. First Understand the Parameter Structure: 2.8T Does Not Mean Inference Compute
MoE, or mixture-of-experts, models have two key metrics:
- Total parameters: determine the scale of weights that must be resident in GPU memory or host memory.
- Activated parameters per token: determine the actual compute volume for a single forward pass.
| Metric | Value | Meaning |
|---|---|---|
| Total parameters | 2.8T | Baseline for weight size and GPU memory usage |
| Activated parameters per token | 104B | Matrix multiplication scale for one forward pass |
| Routed experts | 896 choose 16 | Each token touches only about 1.8% of routed experts |
| Shared experts | 2 | All tokens pass through these |
| Layers | 93 | Amplification factor for KV cache and communication frequency |
2.1 Activated Parameters Determine Compute, Total Parameters Determine Memory
Each token activates about 104B parameters, roughly 3.7% of the total parameters. That means that for every generated token, less than 4% of the total weights actually participate in computation.
However, all experts must remain resident in GPU memory, or be swappable into memory quickly. The router cannot know in advance which experts the next token will select. Once an expert is swapped out, latency can deteriorate sharply. For autoregressive decoding, where tokens are generated one by one, swapping is almost equivalent to making the model unusable.
2.2 Memory Bandwidth Is the Bottleneck for Decode Throughput
Autoregressive decoding is serial. To generate token t+1, the model must traverse all weights selected by that token. The theoretical upper bound of single-stream decoding can be approximated as:
Decode speed, tokens/s ≈ GPU memory bandwidth ÷ weight bytes read per token
This is why single-stream decoding is limited by GPU memory bandwidth: compute units are often waiting for data. Therefore:
- GPU memory bandwidth and memory capacity are equally important. Hardware selection should not focus only on compute metrics.
- Batching is an effective way to improve throughput. Multiple requests can share one forward pass, so the weights are read once and the cost is amortized across all requests.
- However, larger batches increase per-request latency. Concurrency limits should be set based on business requirements for time to first token and generation speed.
3. The “8-GPU Starting Point” Behind the Memory Arithmetic
Inference engines such as MindIE-Turbo, SGLang, and vLLM can recognize information like “104B activated per token” and automatically infer the model’s compute characteristics. However, the engine cannot determine whether your hardware is sufficient. Capacity planning must be done by yourself.
3.1 Weights Are the Hard Threshold
Whether the weights fit depends on the relationship between:
GPUs × memory per GPU
and:
weight size
If we assume the weights are evenly distributed across GPUs:
Weight usage per GPU ≈ weight size ÷ number of GPUs
Using 1.56 TB as an example:
| Configuration | Weight Usage per GPU | Assessment |
|---|---|---|
| 8 × 80 GB, 640 GB total | About 195 GB per GPU | Weights do not fit per GPU. They must be offloaded to host memory, causing time to first token to rise sharply. |
| 8 × 192 GB, 1,536 GB total | About 195 GB per GPU | Even in a best-case estimate, headroom is below about 7 GB. This is not suitable for inference. |
| 8 × 256 GB, 2,048 GB total | About 195 GB per GPU | About 61 GB per GPU remains, which can support a meaningful KV cache and runtime overhead. |
“8 GPUs” describes the number of GPUs. “It can run” also depends on memory per GPU.
As a vertical comparison, 1.56 TB is roughly the aggregate memory of a large number of consumer 24 GB GPUs. A workstation-class host usually cannot fit this many GPUs. Inference also requires frequent cross-GPU communication, so multi-machine communication is hard to avoid. Therefore, the realistic form of “local deployment” is usually a high-performance server node equipped with large-memory accelerators.
3.2 KV Cache: The Hidden Burden of Long Context
Fitting the weights is only the passing line. For every generated token, the model must refer to all previous context. These intermediate results are stored in GPU memory as the KV cache, whose size can be approximated as:
KV cache bytes = 2 × KV heads × head dimension × layers × bytes per token × context length × concurrent requests
If the model uses a low-rank KV compression structure such as MLA, only the compressed latent vectors need to be cached, and the footprint can be much smaller.
A practical approach is to run one million-token-long request as a calibration test: measure the KV cache usage of a single request, then multiply it by the target concurrency. That gives the real GPU memory budget needed for production serving.
3.3 Cloud or Local: How to Choose Among Three Deployment Forms
| Form | Hardware Requirements | Suitable Scenarios | Main Cost |
|---|---|---|---|
| On-premises deployment in a self-built data center | At least 8 large-memory accelerators per node | Data must stay inside the intranet, with stable long-term workloads | High upfront investment and complex operations |
| Elastic GPU instances plus container services | Rent on-demand single-node 8-GPU instances; use container images preconfigured with the engine and weight mounts | Medium-scale workloads with peak and trough patterns | Requires careful instance lifecycle management |
| Directly use a managed model service | No hardware preparation needed | Rapid validation or workloads with highly variable compute demand | Requires review of terms and data compliance |
4. Parallel Strategy: How Fast It Runs
After the weights fit, the next question is how to split the model across 8 GPUs. Common partitioning methods are as follows:
| Parallelism | What Is Partitioned | Benefit | Main Cost |
|---|---|---|---|
| Tensor parallelism, TP | Weight matrices in each layer are split by rows or columns | Per-layer compute is spread out and memory is evenly distributed | Every layer has collective communication, requiring high inter-GPU bandwidth |
| Expert parallelism, EP | The 896 experts are grouped across GPUs | Naturally matches sparse routing in MoE models | Frequent all-to-all communication; load skew when routing is uneven |
| Pipeline parallelism, PP | The model is split into layer stages | Low communication volume; suitable for cross-node deployment | Pipeline bubbles reduce utilization |
The mainstream approach for MoE models is hybrid parallelism: use TP to split dense parts such as attention and shared experts, and use EP to split routed experts.
On this basis, many engineering practices also use attention data parallelism: each GPU holds a complete copy of the attention computation, handles different requests, and maintains KV cache only for its own requests. When computing experts, the system switches to expert parallelism and dispatches tokens through all-to-all communication.
This approach is especially important for million-token contexts:
- Attention computation grows with the square of context length.
- Data parallelism spreads this load across multiple GPUs and distributes KV cache by request.
- If tensor parallelism is used entirely, every GPU must store KV cache for every request, causing obvious memory waste under long contexts.
4.1 Single-Node 8 GPUs Is the Sweet Spot
If the 8 GPUs are in the same node and connected by high-bandwidth inter-GPU interconnect, all-to-all and all-reduce operations can stay entirely within the node. This gives the highest communication efficiency and is the hardware basis that makes “starting with 8 GPUs” reasonable.
If the deployment is split across multiple nodes, GPUs must rely on network interconnect. Every generated token requires cross-node data exchange, and both latency and bandwidth become bottlenecks.
A practical principle: if communication can be completed inside a node, do not cross nodes.
4.2 If Communication Becomes the Bottleneck, Consider Increasing the TP Share
During single-stream decoding, each GPU receives very few tokens. Each expert may receive only a few tokens, or even less than one. The matrix multiplication degenerates into matrix-vector multiplication, leaving compute units largely idle. In this case, the latency proportion of all-to-all communication becomes more prominent.
At this point, increasing the TP share can help:
- Use finer weight partitioning to make matrix multiplications larger.
- Keep communication on higher-bandwidth inter-GPU links.
- The cost is additional all-reduce operations, which should be evaluated through testing.
4.3 Synchronous Implementations and Disaggregated Architectures
There are two mainstream implementation approaches:
- Synchronous implementation, such as SGLang and vLLM: split one forward pass into four stages: attention, all-to-all dispatch, expert computation, and all-to-all gather. Each GPU executes only its own computation. Collective communication in each stage is naturally aligned, making the implementation straightforward and easy to debug.
- Cross-GPU asynchronous scheduling, such as DeepSeek’s open-source DualPipe: allows computation and communication of different microbatches to overlap, hiding all-to-all communication as much as possible under computation and improving utilization. The cost is more complex scheduling and deeper code changes.
In addition, disaggregated architectures that deploy prefill and decode on different instances are increasingly common:
- Prefill is compute-intensive and consumes compute resources.
- Decode is bandwidth-intensive and consumes GPU memory bandwidth.
- Separating them allows independent hardware selection and scaling for each workload, with especially clear benefits in long-context scenarios.
4.4 Parallelism Example
The following example uses the model definition configuration of the open-source framework MindIE-Turbo to show how TP, EP, and data parallelism can be combined:
actor_name: mindie_turbo_0_1_0model_type: npu_agentic_modelmodel_path: /path/to/weightsquantization: wmxa8 # Weights MXFP4 + activations MXFP8quant_config: {}expert_tensor_parallel_size: 1world_size: 8data_parallel_size: 8 # Attention uses data parallelismn_routed_experts: 896n_shared_experts: 2num_experts_per_tok: 16n_group: 1num_hidden_layers: 93hidden_size: 7168intermediate_size: 2048moe_intermediate_size: 1024block_size: 128sparse: attention # Enable sparse attentionmax_position_embeddings: 1000000enable_decode_sink: true # Preserve attention sinks at the beginning of the sequence
data_parallel_size: 8indicates that attention is partitioned with data parallelism, while the expert part uses an EP layout.enable_decode_sinkpreserves tokens at the beginning of the sequence that are frequently attended to, known as attention sinks. In long contexts, these tokens carry a large amount of attention weight. Removing them may noticeably degrade output quality.
5. MXFP4 Weights and MXFP8 Activations: The Chain Reaction of Low Bitwidth
5.1 This Is a Model Designed for Low Bitwidth
MXFP4 and MXFP8 belong to the Microscaling, or MX, numeric formats. Their core idea is “small blocks plus a shared scaling factor”:
- Every 32 elements form a small block and share one 8-bit exponent-format scaling factor.
- Weights are represented with 4 bits, and activations with 8 bits.
- Block-wise scaling better matches the blocked computation style of tensor instructions.
More importantly, these weights come from quantization-aware training, or QAT. During training, quantization error is simulated, and the model learns to maintain performance under quantization noise. Compared with post-training quantization, where a model is trained first and quantized later, QAT usually has smaller accuracy loss.
5.2 Higher Requirements for Hardware and Software Stacks
These formats involve a clear tradeoff between “high-throughput decoding” and “compatibility with older hardware”:
- Native MX format hardware decoding: On newer GPUs and NPUs that support this format, weights can be read directly at 4 bits, dequantized on-chip, and then used in matrix multiplication. This uses the least GPU memory and gives the highest decode throughput. It is the official intended execution mode.
- MX dequantization to FP8 for FP8 matrix multiplication: On NPUs without native MX hardware decoding, weights can first be dequantized block by block into FP8 and then run through FP8 matrix multiplication. Weights remain stored in GPU memory at 4 bits, but additional dequantization compute is required. This path is suitable for newer platforms.
- FP conversion to BF16 path: On hardware that only supports BF16 matrix multiplication, weights must first be converted to BF16 before computation. This conversion may need to be repeated for each token and can only rely on cache reuse. It is suitable for low-concurrency scenarios.
5.3 Only One MXFP4 Version Means You Cannot Afford Precision
The team released only this one weight version. If users dequantize the weights to BF16 themselves, each parameter takes 2 bytes, and 2.8T parameters require about 5.6 TB of storage. The total memory of 8 GPUs with 512 GB each is far from enough. The only option is to fall back to weight offloading or swapping in and out, which severely reduces performance.
For teams pursuing extreme throughput, the community also uses a more aggressive approach: use high-precision computation in the prefill stage and low-precision weights in the decode stage, with independent quantization parameters for each stage. This is an empirical optimization, and accuracy loss must be validated on business data.
A low-bit model is not a “discounted model”. Two out-of-distribution scenarios require caution:
- Extrapolation beyond the training context length.
- Private-domain data that differs significantly from the training data distribution.
In these scenarios, quantization noise is more likely to be amplified. Before launch, be sure to run comparative evaluations using your own data.
6. Million-Token Context: What Sparse Attention Solves
6.1 The Quadratic Bottleneck of Standard Full Attention
In standard full attention, every token computes relevance with all tokens before it. The computation grows quadratically with context length. Using the 93-layer K3.1 Pro as an example:
- Increasing sequence length from 8K to 1 million expands the context length by about 128 times.
- Attention computation expands by about 16,000 times.
If every layer retained full attention, long-context compute costs would become extremely high.
6.2 DSA: Dynamically Decide What to Look At
K3.1 Pro changes 24 of its layers to sparse attention. According to public technical materials, the overall idea of DSA is “compressed selection plus sparse computation”:
- Compress each token’s key vector into a latent vector, similar to low-rank compression in MLA.
- A lightweight scorer computes similarity scores between the current query and all historical latent vectors, quickly selecting the top-k most relevant tokens.
- Attention is computed precisely only on the selected tokens.
The effect is that attention computation drops from quadratic to near-linear, while attempting to preserve long-range dependency capability.
This also brings two implications:
- When precise attention is combined with compressed representations, both latent vectors and exact key-values may need to be cached. Actual GPU memory usage should be based on the framework’s real behavior.
- The scorer does not use strict top-k. It uses a combined strategy of “sorting threshold plus a top-k cap.” It behaves more like “filter as much as possible” rather than hard truncation.
6.3 Key Engineering Switches for Long Context
| Configuration | Role | Suitable Scenario |
|---|---|---|
sparse: attention |
Enables sparse attention | Should always be enabled for long-context scenarios |
enable_decode_sink |
Protects attention sinks at the beginning of the sequence | Recommended for long-context scenarios |
| Chunked prefill | Splits very long prompts into smaller chunks for batched computation | Reduces peak GPU memory per iteration and long-tail latency |
For long-context serving, two additional common strategies are:
- Prefix caching: Requests that share the same system prompt or multi-turn dialogue history can reuse already computed KV cache.
- KV eviction and sparsification: During decoding, selectively retain KV cache for important tokens.
In addition, the cost of the prefill stage, which processes input, is much higher than the decode stage. Some measurements indicate that one request with 1 million input tokens can cost roughly as much as generating about 100,000 output tokens, with a difference of about 10 times. Therefore, compressing input and reusing cache are extremely valuable in long-context scenarios.
7. How to Read Benchmarks: Vendor-Reported Data
The comparison table included in the model card shows that this product does not lead on every dimension:
- Coding and agent tasks: On a terminal-operation benchmark, it scored 65.7%, slightly below a leading closed-source flagship model at 66.4%, but higher than another flagship model at 58.2%. On a long-task benchmark requiring continuous multi-step operations, it scored 51.7%, exceeding the flagship’s 46.3%.
- Science and some reasoning tasks: On a science benchmark, it scored only 38.1%, while a leading flagship scored 63.3%. On a high-difficulty multidisciplinary reasoning benchmark, it scored 39.4%, while the flagship scored 60.9%.
The vendor claims that 16 of 37 benchmarks outperform the flagship model. A case-by-case comparison shows that it leads in coding and agent tasks but clearly lags in science and some reasoning tasks.
A third-party independent evaluation organization gave it a composite intelligence index of about 55, ranking fourth globally, which is broadly consistent with the above judgment.
7.1 Three Principles for Reading Benchmarks
- Distinguish capability dimensions: Benchmark scores are often projections of a capability profile. Similar total scores do not mean balanced performance across dimensions.
- Watch for self-reporting bias: Vendor-reported data may use different baseline versions, prompts, and inference configurations from your scenario. Treat it only as an initial screening reference.
- Lock down the test configuration: The quality of sparse-attention block selection changes with configuration. For example, for short-context scenarios within 8K, many practices recommend disabling sink protection or increasing the top-k cap to reduce accuracy loss caused by sparsification.
7.2 Use Your Own Evaluation Set for Model Selection
A lightweight but effective evaluation process is recommended:
# Private evaluation set for model selection, illustrativetasks = {"code_gen": ["Internal API calls", "Legacy system modernization"], # Coding"agent": ["Multi-step tool calls", "Long-workflow task recovery"], # Agent"long_context": ["Very long contract comparison", "Million-token codebase Q&A"], # Long context"science": ["Domain formula derivation", "Experiment data analysis"], # Science"reasoning": ["Complex logical reasoning", "Multi-constraint planning"], # Reasoning}for name, prompts in tasks.items():report = run_eval(model="candidate_model", dataset=prompts)print(name, report.pass_rate, report.p95_latency, report.cost_per_1k)
This provides conclusions directly relevant to your business across three dimensions: quality, latency, and cost.
8. Open Weights Does Not Mean Open Source: How to Read Additional Terms
This is one of the most important points in this release. The repository uses a custom model license. The hosting platform labels it as an “Other” license, not a standard open-source license such as MIT or Apache 2.0.
According to the model card, this license is word-for-word identical to the previous generation’s license, with only the name changed. The main body is MIT-like: free use, copying, modification, distribution, fine-tuning, and commercial use. However, two additional restrictions are layered on top, targeting only “large-scale external businesses”:
- A company that provides the model as a service, meaning model-as-a-service, MaaS, and whose cumulative revenue over 12 months exceeds $20 million must sign a separate agreement with the copyright holder before commercial use.
- A commercial product with more than 100 million monthly active users, or more than $20 million in monthly revenue, must prominently display the model name in its interface.
8.1 Concept Clarification: Open Weights Is Not Open Source
The authoritative definition of “open source” in the industry is the Open Source Initiative’s Open Source Definition, or OSD. Its core principles include:
- Free redistribution.
- Permission for derivative works.
- No discrimination against specific persons, groups, or fields of use.
The requirement that “users above a certain usage threshold must sign an additional agreement” is an additional usage restriction. Therefore, this type of “open-weights” model is more accurately described as “openly usable with additional terms,” rather than open source in the strict sense.
Another interpretation of the open-source spirit is “process openness.” Anthropic’s “Constitutional Openness” argues for publicly sharing training methods, data mixtures, and the model spec. From this perspective, the release made the model card and license public, but training data, code, and detailed methods have not been made public.
8.2 Enterprise Compliance Checklist
| Check Item | Key Question | Recommended Action |
|---|---|---|
| Business type | Does it involve providing the model as a service externally? | If yes, evaluate whether 12-month revenue may exceed the threshold |
| Revenue threshold | Will cumulative 12-month revenue exceed $20 million? | Contact the copyright holder in advance if approaching the threshold |
| User scale | Will monthly active users exceed 100 million? | Products above the threshold need an attribution plan |
| Revenue scale | Will monthly revenue exceed $20 million? | Either monthly active users or monthly revenue can trigger the condition |
| Attribution requirement | Is the model name prominently displayed in the interface? | Fix it in UI standards and include it in release checks |
| Ownership of model outputs | Who owns generated code and content? | Usually the user; in important scenarios, confirm with the service terms |
| License-change risk | Custom terms may be adjusted | Archive the weights and a snapshot of the license version in use |
| Patent grant | Custom MIT-like licenses usually do not specify patent grants | Products with higher patent risk need legal review |
For most teams with smaller revenue, these terms usually do not create an obstacle. The model can be freely used for commercial purposes, fine-tuning, and secondary distribution. However, for enterprises with larger revenue, legal evaluation should be moved earlier in the model selection process to avoid rework after launch.
8.3 Cloud Deployment Route: Running Open-Weight Models on Baidu AI Cloud
For teams using Baidu AI Cloud, this type of open-weight model can be deployed in four steps:
Weight hosting
- Store the 1.56 TB weights in object storage and download shards with multithreaded concurrency.
- Enable versioning and integrity checks, such as comparing per-file hashes, to prevent corrupted shards.
- For teams that need to move weights from an external hosting platform into the cloud, data transfer services can be used to efficiently synchronize weights into cloud storage and then mount them to inference instances.
Inference service
- Deploy the inference engine using a container service and choose single-node 8-GPU instances with large per-GPU memory, ensuring that communication for expert parallelism and attention data parallelism stays within one node.
- Enable prefix caching and chunked prefill for long-context workloads.
- If a managed service is provided through a DeepSeek all-in-one appliance deployed by Baidu AI Cloud on customer-owned GPU or NPU resources, users can call the API directly without building their own cluster.
Elasticity and cost
- Set concurrency limits based on predicted decode throughput and configure elastic scaling according to business peaks and valleys.
- Run offline batch jobs during off-peak periods.
- Continuously track GPU utilization and cost per token through monitoring services.
Observability and security
- Use logging services to collect request logs and error logs.
- Use monitoring and alerting for GPU memory usage, time to first token, generation speed, and error rate.
- Use an API gateway for authentication and rate limiting.
- Use security posture tools for vulnerability and compliance checks.
9. Production Launch Checklist
| Check Item | Key Point |
|---|---|
| Weight integrity | The number of shards and file hashes match the official manifest |
| GPU memory headroom | About 195 GB per GPU for weights; remaining memory must support KV cache for the target concurrency |
| Inference engine | Use an engine that supports MXFP4 weights plus MXFP8 activations, or a validated dequantization approach |
| Parallel configuration | The main communication for the TP/EP/DP combination is completed inside one single-node 8-GPU system |
| Long context | Enable sparse attention, attention sink, chunked prefill, and prefix caching based on actual measurements |
| Quality validation | Compare against a closed-source flagship model on your own evaluation set across coding, agent, long-context, and scientific reasoning |
| License compliance | Confirm whether revenue and user scale trigger additional terms, prepare an attribution plan, and archive a license snapshot |
| Observability | Metrics and alerts exist for GPU memory, latency, throughput, and error rate |
10. Conclusion
The release of this kind of very large open-weight model hands both model capability and engineering difficulty to developers. The 1.56 TB weight size corresponds to about 195 GB per GPU, which means 8 GPUs are only the starting point. MXFP4 and MXFP8 determine the space for hardware and engine selection. Sparse attention under a million-token context determines whether long-text workloads can run efficiently. And a custom license with additional conditions determines how far you can use the model.
For developers, the rational path is:
- First use your own evaluation set to confirm quality.
- Then use the GPU memory budget and parallel strategy to confirm hardware.
- Finally use the license terms to confirm boundaries.
On this basis, cloud elastic compute and managed services can turn an open-weight release into stable, controllable, and sustainable product capability.
