Open-Weights 2.8T Parameter Model: Architecture Breakdown and 8-GPU Deployment
This article reviews the release of a 2.8-trillion-parameter open-weights model and breaks down its MoE architecture, sparse attention, 4-bit quantization, reported benchmarks, licensing constraints, and practical 8-GPU deployment considerations. It covers the 1.56 TB weight files, expert routing, long-context behavior, verification steps, and selection trade-offs for teams evaluating large open-weights models for private deployment.
1. Event Overview: An Open-Weights Release with Conditions
After the weight files were made available, discussion exploded. A model with 2.8 trillion parameters, roughly 1.56 TB of weight files, and a practical starting point of eight GPUs for local execution has pushed nearly every number associated with “open-weights models” to a new level.
A recurring observation in the discussion is that this marks a shift for open-weights models from “comparable capability” to “comparable scale.” However, “downloadable” does not mean “freely usable,” and it certainly does not mean “ready to run out of the box.”
1.1 Timeline
| Date | Key action |
|---|---|
| Oct. 24 | The model first launched via API and a web interface. The publisher promised to release the full weights by Nov. 4. |
| Nov. 2 | The weight distribution repository was created, but remained closed. |
| Nov. 4 | The repository was closed in the morning and officially opened in the evening. All files were publicly downloadable without an application. |
Only 11 days passed between the promise and the release.
It is also worth noting that launching the model first as a hosted service and then releasing the weights has become a common pattern for major teams shipping flagship models. The online service can accumulate real traffic first, while the open weights later serve teams that need private deployment.
1.2 File Structure
The repository contains 112 files. Of these, 96 are sharded weight files totaling about 1.56 TB. The remaining files include model configuration, tokenizer files, chat templates, generation parameters, and other supporting assets.
This “multi-shard weights plus index file” layout is the standard form for extremely large model weights. During loading, each shard can be mapped independently, supporting parallel reads. If a single shard is damaged, only that shard needs to be downloaded again.
1.3 Two Conditions Behind “Open”
The term “open weights” actually involves two layers of constraints.
- License constraints: Open weights usually mean that the weights can be downloaded, modified, and redistributed. This is not the same as a permissive open-source license. Some community licenses include clauses requiring attribution, capping user scale, or prohibiting the use of model outputs to train competing products. These terms need to be reviewed one by one before use.
- Hardware constraints: Whether the weights can fit in a local environment and run at a usable speed is itself a barrier.
For most teams, the second condition often becomes a practical problem earlier than the first.
2. Parameter Structure Breakdown: How 2.8T and 104B Relate
2.1 Basic Logic of MoE
The model uses a Mixture of Experts architecture.
At a high level, each feed-forward network layer can be viewed as a row of “experts.” The router, or gating mechanism, selects a small number of the most relevant experts for each token and computes only those experts.
Total parameters determine the model’s knowledge “capacity,” while activated parameters determine the “compute cost” of a single inference step. This decouples scale from per-token compute.
The model’s configuration is as follows:
- Total parameters and activated parameters: 2.8 trillion total parameters; only 104 billion parameters are activated per token, about 3.7% of the total.
- Expert scale: 896 routed experts. Each token selects 16 of them. There are also 2 shared experts that are always involved in computation.
- Network depth: 93 layers.
- Context length: 1 million tokens.
- Multimodal capability: The model is natively multimodal and includes a built-in visual encoder. It can understand images directly, rather than attaching a separate vision module to a text-only model.
2.2 Design Trade-offs Behind the Configuration
Fine-grained experts with fewer activated experts: With 896 experts and 16 selected per token, each expert carries about 1/56 of the routed expert capacity. Finer granularity increases the combination space and makes each expert more specialized.
The trade-off is that routing decisions become more sensitive. If a small number of tokens are concentrated on a few experts, those experts become “hot experts,” slowing overall inference.
Shared experts provide general capability: The 2 shared experts apply to every token and handle common computation such as general knowledge and syntax. Routed experts focus on narrower domains. This division of labor can alleviate the problem of all tokens competing for a small number of general-purpose experts.
In inference implementations, shared experts can be fused directly into local computation and do not necessarily need to participate in expert-parallel communication.
Deep network with low activation ratio: With an activation ratio of about 3.7%, the model is in a typical range for today’s very large MoE models. One direct result is:
- During decoding, generating each token requires reading about 52 GB of activated weights from GPU memory, calculated as 104 billion × 0.5 bytes.
- Under the same memory bandwidth, theoretical decoding speed can approach that of small- or medium-sized dense models.
This is an ideal estimate. Actual speed depends on the parallelism strategy, expert-parallel communication overhead, and batching efficiency, which are discussed later.
Native multimodal design: Visual information and text are jointly modeled inside the same model. This avoids the interface loss and context fragmentation that can occur in a “vision model plus language model” pipeline.
The cost is that the visual encoder’s weights, GPU memory footprint, and operators must also be included in the deployment plan.
3. Key Architectural Change: Attention Designed for Million-Token Contexts
Compared with the previous generation, the official description says the structure changed in one main place: 24 of the 93 layers switched from full attention to sparse attention, while the remaining 69 layers kept full attention.
3.1 Why Sparsity Is Necessary
In standard self-attention, every token attends to all previous tokens, so compute grows quadratically with sequence length.
Expanding from 4K to 1M tokens increases sequence length by 250×. Attention compute theoretically increases by about 62,500×. Relying only on more compute is not practical.
Sparse attention allows each token to attend only to a selected set of important context, substantially reducing that overhead.
3.2 Why Only 24 Layers Were Changed
Representations in deep networks become increasingly abstract:
- Earlier layers often handle local feature aggregation, where local attention may be sufficient.
- Later layers rely more on global information integration, where full attention is more valuable.
The compromise is to introduce sparse attention in intermediate layers to control cost, while preserving full attention in important layers to maintain quality.
One notable detail is that sparse attention mainly saves compute, but it may also reduce KV-cache access if the implementation reads KV cache selectively. If that holds in practice, memory-bandwidth pressure during long-context decoding may also decrease.
3.3 New Engineering Problems from Sparsity
- Higher implementation complexity: Different sparse patterns may require custom kernels, placing higher demands on inference framework support.
- Behavior must be validated: For tasks that rely heavily on global information, such as long-document question answering or cross-file code understanding, evaluation should confirm that information recall does not degrade significantly.
- Context-tier validation: 1M is the model’s theoretical upper limit. For deployment, it is best to run staged tests at 32K, 128K, 512K, and 1M context lengths, verifying quality, latency, and GPU memory usage at each level before gradually opening up larger contexts.
4. Quantization and Training: Only a 4-Bit Version
4.1 Precision Scheme
The weights use a 4-bit floating-point format, while activations use an 8-bit floating-point format. Both come from quantization-aware training.
In simple terms, the model is trained to adapt to lower-precision computation during the later stages of training, reducing accuracy loss from quantization as much as possible.
4.2 File Size Estimate
If all 2.8 trillion parameters were stored at 4-bit precision, the theoretical size would be about 1.4 TB, calculated as 2.8 × 10¹² × 0.5 bytes.
The actual files are about 1.56 TB, a difference of roughly 12%. This difference may come from tensors still stored at higher precision, such as token embeddings, normalization-layer parameters, or precision-sensitive modules.
One reminder: file managers may display binary units such as GiB, so the displayed value may not exactly match “1.56 TB.” When planning disk space, runtime files beyond the weights themselves should also be included.
4.3 Three Implications of a Single Quantization Version
- No alternate compute precision: Without BF16 or FP8 variants, you cannot choose a different compute precision to trade for speed, and you may not be able to run the model on accelerators that only support standard floating-point formats. This raises the hardware threshold, because deployers must choose accelerators that natively support the relevant low-precision formats.
- Limited fine-tuning flexibility: Performing full-parameter fine-tuning directly on low-precision weights is difficult to control in both effect and risk. For domain adaptation, safer options include:
- Applying low-rank adaptation such as LoRA on top of the quantized weights, keeping fine-tuning parameters at higher precision during training and merging them with the base weights during inference;
- Using prompt engineering or retrieval-augmented generation without modifying the weights.
- Low-precision MoE is more friendly to fine-grained quantization: Each token activates only a small number of experts, so quantization error is less likely to be amplified across many parameters at once. Compared with dense models, MoE models are generally less sensitive to weight quantization. This is one reason a model of this type may release a 4-bit version directly.
5. How to Read the Reported Benchmarks
The model card provides the following comparison data. All figures are self-reported by the publisher. The comparison targets are closed-source flagship models from a similar period, referred to generically below as “closed-source flagship.”
| Evaluation area | This model | Closed-source flagship | Relative performance |
|---|---|---|---|
| Terminal-based general programming tasks | 65.7% | 66.4% | 0.7 pp lower |
| Terminal-based general programming tasks, vs. another closed-source model | 65.7% | 58.2% | 7.5 pp higher |
| Long-horizon software engineering tasks | 51.7% | 46.3% | 5.4 pp higher |
| Research-oriented terminal tasks | 38.1% | 63.3% | 25.2 pp lower |
| Hard knowledge evaluation | 39.4% | 60.9% | 21.5 pp lower |
5.1 Capability Profile: Strong on Tool Use, Weaker on Knowledge
Overall, the profile is “at parity or ahead on tool-oriented tasks, with a clear gap on knowledge-oriented tasks.”
- General programming is roughly at parity with the closed-source flagship and about 7.5 percentage points ahead of another closed-source model.
- Long-horizon software engineering tasks lead by 5.4 percentage points. These tasks require the model to call tools over multiple turns, read and write files, run tests, and iteratively fix issues. This aligns well with the model’s 1M context and agent-oriented positioning.
- Research-oriented terminal tasks trail by 25.2 percentage points.
- Hard knowledge evaluation trails by 21.5 percentage points.
The knowledge weakness has a direct impact on selection. For scenarios requiring deep domain knowledge, such as scientific analysis or complex reasoning, candidate models should be retested on the same question set rather than selected based only on an overall ranking.
5.2 Four Reservations About Self-Reported Data
- Comparability: The numbers are vendor-reported and lack independent third-party reproduction.
- Evaluation protocol: Prompt design, tool permissions, number of runs, and voting strategy, such as pass@1 versus best-of-multiple-sampling, can all affect results.
- Score semantics: “65.7%” may be a task pass rate, accuracy, or another metric. Results from different evaluations cannot be directly compared horizontally.
- Timeliness: Models continue to evolve quickly, and leaderboards change over time.
The recommended approach is to build a small evaluation set on your own business data, fix the evaluation scripts and random seeds, and use “running the model on your own tasks” as the final step in the selection decision.
6. Licensing and Compliance: Open Weights Are Not Unrestricted
6.1 License Review Checklist
- License type: Permissive or community-defined.
- Commercial use and scale limits: Whether commercial use requires additional authorization, and whether there are user-scale or revenue limits.
- Attribution requirements: Whether attribution is required when distributing the model or offering a service based on it.
- Derivative model restrictions: Whether model outputs can be used to train other models.
- Liability terms: Who bears responsibility for the accuracy and compliance of generated content.
6.2 Deployment Compliance Considerations
Depending on the jurisdiction and deployment scenario, teams may need to address local requirements for service registration, safety assessment, content moderation, data protection, and labeling of generated images, audio, or video.
Private internal deployments also require data security and access-control mechanisms.
6.3 Compliance Implications of the Release Cadence
The “API first, weights later” pattern means that the service terms and the weight license may not be the same document, and they may apply to different subjects:
- When using the API, usage is governed by the service agreement, and data handling is determined by the service provider.
- After downloading the weights, usage is governed by the weight license. Data stays local, but compliance responsibility shifts to the deploying party.
Teams working with sensitive data should review the license text as soon as the weights are released and retain a version record.
7. Deployment Threshold: Fitting 1.56 TB into 8 GPUs
7.1 GPU Memory Estimation Formula
Total GPU memory usage can be broken down into five parts:
Total GPU memory ≈ weights + KV cache + activations and temporary buffers + framework overhead + safety margin
A simple estimate of per-GPU weight usage is shown below:
# Example: per-GPU weight footprint; GB is a decimal unitWEIGHT_BYTES = 1.56e12 # Weight files are about 1.56 TBGPU_COUNT = 8per_gpu_gb = WEIGHT_BYTES / GPU_COUNT / 1e9print(f"Per-GPU weights: ~{per_gpu_gb:.1f} GB") # ~195 GB# Example: single-sequence KV cache usage at 128K context with FP8 storage# (head count and head dimension are illustrative only)kv_per_token = 2 * 93 * 8 * 128 * 1 # K and V: 2 × layers × KV heads × head dim × 1 bytectx = 128 * 1024print(f"128K context single-sequence KV cache: ~{kv_per_token * ctx / 1e9:.1f} GB")
7.2 8 GPUs Is the Baseline, Not the Comfortable Configuration
Even if all 8 GPUs are used to hold weights, each GPU still needs about 195 GB. Adding KV cache, activations, and framework overhead, 8 GPUs are mostly enough for the scenario where the weights just fit.
If you want to support contexts above 128K with a certain concurrency level, an 8-GPU deployment will need to reduce KV-cache precision, reduce the per-GPU context budget, or enable expert-weight offloading, which temporarily moves some experts to host memory. The latter will noticeably increase decoding latency.
A more practical approach is to treat 8 GPUs as a first-round validation configuration. First confirm that the service can start, perform inference, and produce correct results. Then scale linearly according to business throughput requirements.
Parallelism often needs to double in powers of two. A 128K-context concurrency scenario may require 16, 32, or more GPUs.
7.3 Parallelism Strategy: Tensor Parallelism and Expert Parallelism
| Parallelism dimension | What is split | Per-GPU load | Main communication |
|---|---|---|---|
| Tensor parallelism | Attention layers and dense layers | 1/N of each layer’s weights | All-reduce per layer |
| Expert parallelism | 896 routed experts | About 112 experts per GPU with 8 GPUs | All-to-all dispatch and combine per layer |
MoE deployments usually combine both:
- Use tensor parallelism to split the attention portion.
- Use expert parallelism to distribute the experts.
Two practical points:
- Keep routing balanced: With 896 experts across 8 GPUs, each GPU handles about 112 experts. This provides relatively balanced compute per GPU, assuming routing is not overly concentrated.
- Prioritize high-speed intra-node interconnect: Expert parallelism relies heavily on all-to-all communication, which is demanding on the network. Try to keep all GPUs within the same high-speed interconnect domain of one node. Before cross-node deployment, measure the share of communication overhead in total decoding time.
7.4 Example Startup Configuration
The following configuration is illustrative only. Exact fields and values depend on the inference framework being used.
# First-round validation: 8-GPU single node, 128K context# Other capabilities are enabled per request as neededlaunch_config = {"weight_dtype": "low-precision floating point", # Matches the official 4-bit weight format"tensor_parallel_size": 8, # Tensor parallelism across 8 GPUs for attention and dense layers"expert_parallel_size": 8, # Expert parallelism across 8 GPUs for the 896 experts"max_model_len": 131072, # Start with 128K, then gradually increase toward 1M"kv_cache_dtype": "fp8", # Use KV-cache quantization to gain long-context headroom"enable_prefix_caching": True, # Reuse shared system prompts and tool descriptions"gpu_memory_utilization": 0.9, # Reserve headroom to avoid OOM from fragmentation}
7.5 Weight File Verification
Multi-shard weights are very sensitive to file integrity. Before loading, check the shard count, total size, and hash digests:
import hashlibfrom pathlib import Pathdef sha256_of(path: Path, chunk_size: int = 1 << 20) -> str:h = hashlib.sha256()with path.open("rb") as f:for chunk in iter(lambda: f.read(chunk_size), b""):h.update(chunk)return h.hexdigest()root = Path("model_dir")shards = sorted(root.glob("*.safetensors"))print(f"Shard count: {len(shards)}") # Expected: 96print(f"Total size: {sum(p.stat().st_size for p in shards) / 1e12:.2f} TB")
Note that computing hashes shard by shard requires reading all weights once. With 1.56 TB of data, this can take hours over a 10GbE network. It is best to run the check after download and before formal deployment.
7.6 GPU Memory Planning for the Multimodal Branch
The visual encoder also consumes GPU memory. High-resolution images can produce many visual tokens after encoding, which is especially noticeable for long-document parsing and multi-image comparison tasks.
It is advisable to define the maximum allowed image resolution and number of images per request. For very long image inputs, use truncation or chunking to avoid pushing a single request to the memory limit.
7.7 Private Deployment Architecture
- Storage layer: Object storage can be used to store weight files, then pulled through the internal network to local disks on compute nodes. This avoids reading weights directly across the network during inference.
- Compute layer: A container platform can orchestrate multi-GPU inference services. A single-node, multi-GPU setup can be treated as one scheduling unit to keep expert-parallel communication on the high-speed interconnect.
- Service layer: A load balancer can expose a unified model API and route requests to backend inference instances.
- Observability layer: Logging and monitoring services can collect metrics such as latency, throughput, GPU memory, and temperature, with tiered alerts configured.
For cloud-based deployments, the general capabilities of Baidu AI Cloud can be considered as an infrastructure option for this type of deployment.
8. Long Context and Agentic Inference: 1M Context Does Not Mean Unlimited Use
8.1 Context-Tier Validation
For 1M-token contexts, input tokens are often billed as the full sequence, and attention overhead grows with length. It is better to include context budgeting in request-level policies:
- Allocate budgets by task: Keep retrieval QA under 32K, cross-file code understanding under 128K, and only relax to higher tiers for tasks that truly need longer documents.
- Rate-limit long requests separately: Use a dedicated queue and concurrency limit for long-context requests so they do not slow down short requests.
8.2 Prefix Reuse for Agentic Inference
Agent workloads often have highly repetitive system prompts, tool definitions, and prior trajectories. Prefix caching can reuse that computation and significantly reduce first-token latency. This is also one reason the model performs well on long-horizon software engineering tasks.
8.3 Notes on Speculative Decoding
Speculative decoding uses a small draft model to generate candidate tokens, then has the large model verify them in parallel.
For a large model with a substantial number of activated parameters, the size gap between the draft model and the target model makes it harder to guarantee a high draft-model hit rate. Before production rollout, validate the actual benefit through A/B testing rather than enabling it by default.
9. Common Misconceptions and Selection Decisions
| Common misconception | Actual situation |
|---|---|
| “2.8 trillion parameters must be stronger than smaller models.” | Knowledge-oriented evaluations still trail closed-source flagship models. The capability profile is different from dense small models. |
| “8 GPUs can run it locally.” | 8 GPUs are the minimum to fit the weights. Long context and concurrent throughput require more GPUs or precision trade-offs. |
| “Open weights mean open-source freedom.” | License terms, local compliance requirements, and content-safety obligations still apply. |
| “The official 4-bit version is enough.” | The absence of higher-precision versions limits fine-tuning options and hardware choices. |
For selection, consider the following order:
- Task type: Prioritize tool-calling and long-horizon coding tasks. Independently evaluate knowledge-intensive tasks.
- Data sensitivity: Only consider private deployment when data cannot leave the domain.
- Throughput requirements: Estimate the number of GPUs from target QPS and context length, rather than using “it starts successfully” as the standard.
- Team capability: Multi-GPU parallel deployment, monitoring, and operations require dedicated engineering capability.
- Total cost of ownership: Compare the combined cost of accelerators, storage, power, and personnel against the cost of pay-as-you-go services.
For most teams, validating model capability and business fit through the API first, then gradually migrating to private deployment once data compliance and cost are under control, is the lower-risk path.
10. Conclusion
2.8 trillion parameters, 1.56 TB of weights, and an 8-GPU starting point represent an important step for open-weights models in terms of scale. They also place the boundaries of “open” in a clearer position: license, hardware, context, and operations all require engineering judgment.
Take reported leaderboards rationally. Focus on three questions: does the task match, is the data compliant, and is the cost controllable. That is the more reliable approach when evaluating very large open-weights models.
