Deploying a 2.8T-Parameter Open-Weights Model: 1.56TB of Weights, 8 GPUs to Start

    A leading model team has made good on a long-standing promise and published the full weights of an ultra-large open-weights model: 2.8T total parameters, 104B activated per token, 1.56TB on disk, MXFP4-only quantization. This article walks through what the release actually contains, why a sparse mixture-of-experts architecture makes compute cheap but memory expensive, how to estimate GPU count and KV cache requirements, how to read the vendor-reported benchmark numbers, and why "open weights" is not the same as open source. You will learn the practical trade-offs between self-hosting on multiple nodes, renting dedicated cloud instances, and calling a hosted API.

    1. The release: an overdue set of weights finally lands

    Recently a leading model team delivered on a promise made weeks earlier and published the complete weights of an ultra-large model. Parameter count, weight volume, activation configuration and the local deployment bar all set new records for the open-weights ecosystem to date. For readability, this article refers to it as “the model” or “the target model.”

    1.1 Timeline: from API to weights

    • Late October: the model went live in two forms, an API and a web interface, giving developers a way to call it. At the same time the team promised that full weights would be released by early November.
    • Early November: the weights appeared as promised on the main international model-hosting hubs. The repositories were public and downloadable, with no application or approval step.
    • Release day: the repository was still closed that morning and opened that evening, a gap of less than a day.
    • Mirrors: mirrors made the weights available through other channels as well, so the region-restricted download problems that affected earlier releases were largely a non-issue.

    Roughly ten days separated “usable” from “obtainable.” Those ten days are exactly the window in which an organization moves from technical demonstration to accountable delivery. An API service can contain risk server-side with traffic controls, quotas and rate limits; once full weights are public, both the risk and the responsibility shift to whoever runs them.

    1.2 What is actually in the repository

    • File layout: 112 files in total. The core weights are split into 96 safetensors shards, alongside the model card, license text, configuration files and tokenizer and other supporting assets.
    • Download size: roughly 1.56TB in full. On a gigabit connection at an ideal ~125MB/s, pure transfer time alone exceeds 3.5 hours; on a 10Gbps link the theoretical time is about 21 minutes. Actual time also depends on mirror synchronization strategy, the number of concurrent connections and server-side throttling.
    • Deployment shape: the full weights are a single whole, not an on-demand subset. Any single inference touches most of the shards where experts live, a point covered in section 3.
    • Where it sits historically: it is one of the largest open-weight releases to date and is in a different size class from previous generations at comparable scale. “Can it be downloaded” is no longer the question. The question is “what do you do with it once you have it, and who pays for that.”

    2. Inside 2.8T: how the sparse expert model is built

    2.1 Parameter composition: total parameters are not compute

    According to the model card, the model is a natively multimodal agent model, continued from the previous generation. Its core structure is:

    Dimension Configuration Engineering implication
    Parameter count 2.8T total, 104B activated per token Sparse activation ratio of roughly 37:1; size memory for 2.8T, size compute for 104B
    Expert structure 896 routed experts, 16 selected per token, plus 2 shared experts Routing itself is cheap; expert scheduling and communication are the bottleneck
    Depth 93 layers Layer count multiplies both compute and KV cache growth
    Context window 1M tokens Large long-context upside, with rising memory and attention cost
    Vision Built-in vision encoder Native multimodality, at the price of a longer deployment chain

    This is a conventional sparse mixture-of-experts (Sparse MoE) architecture. Each token passes through 16 routed experts (about 1.8% of all routed experts) and the 2 shared experts; the remaining experts do not participate in that forward pass. Both the benefit and the cost are clear:

    • Benefit: compute cost grows with activated parameters, not with total parameters. Forward compute is roughly determined by the 104B activated parameters, which is in the range of a small-to-mid-sized dense model, so the compute cost of a single inference is not extreme.
    • Cost: memory occupancy is still determined by total parameters, and not one of the 2.8T can be dropped. That produces a “cheap compute, expensive memory” cost structure, and it is the origin of the “8 GPUs to start” claim.

    The relationship between total parameters, activated parameters, memory and compute can be built up with the sketch below:

    1. # Illustrative only; the actual layout depends on the inference engine's
    2. # parallelization strategy and routing implementation.
    3. def estimate_model_memory_and_compute(total_params, active_params, bits=4):
    4. weight_bytes = total_params * bits / 8 # Weight memory ~= 2.8T x 0.5 B ~= 1.4 TB
    5. bytes_per_token = 2 * active_params * bits / 8 # Parameter bytes read for one forward pass
    6. return {
    7. "weight_bytes": weight_bytes,
    8. "bytes_per_token": bytes_per_token,
    9. "bytes_per_1k_tokens": bytes_per_token * 1000,
    10. }

    At 4-bit precision, processing 1,000 tokens requires reading roughly 26GB of weight data out of memory. A few numbers side by side show where the bottleneck sits:

    • Memory bandwidth on mainstream data-center GPUs is in the multi-terabyte-per-second range;
    • ideally, reading the weights for 1,000 tokens takes about 4–8 ms;
    • in practice this is stacked on top of routing, all-to-all expert communication and attention compute;
    • the densification of a single expert’s computation adds extra FLOPs, and the GPU compute ceiling is typically more than an order of magnitude below the bandwidth ceiling.

    So on this class of model, throughput is constrained first by bandwidth and communication, not by floating-point compute. Exact numbers depend on hardware and engine implementation.

    The routing logic of a sparse MoE can be reduced to:

    1. # Pseudocode: token-level sparse routing
    2. def moe_layer(x, router, experts, shared_experts, num_experts, top_k=16):
    3. # x: [T, H], where T is the token count and H the hidden size
    4. logits = router(x) # [T, num_experts], num_experts = 896
    5. scores, idx = topk(logits, k=top_k) # each token picks 16 experts
    6. weights = softmax(scores, dim=-1) # normalize within the selected experts
    7. y = zeros_like(x)
    8. for slot in range(top_k):
    9. for eid in range(num_experts): # group by expert for batched compute and parallel dispatch
    10. mask = idx[:, slot] == eid
    11. if mask.any():
    12. y[mask] += weights[mask, slot, None] * experts[eid](x[mask])
    13. for expert in shared_experts: # 2 shared experts: every token passes through
    14. y += expert(x)
    15. return y

    The 2 shared experts carry the general-purpose capability every token needs. They are selected far more often than any individual routed expert and account for a meaningful share of actual compute. For that reason, 104B activated parameters does not mean “routed experts only computed 1.8% of expert parameters.”

    2.2 One key structural change from the previous generation

    Compared with the previous generation, the structural change is concentrated in one place: 24 of the 93 layers switch from full attention to sparse attention (DSA), while the other 69 keep full attention. The goal is to save compute on extremely long contexts.

    • Cost of full attention: both compute and KV cache reads grow with the square of sequence length, which is the main cost source in long-context scenarios.
    • What sparse attention buys: each query attends only to a small number of selected tokens, bringing attention complexity down to roughly linear. On inputs in the 100K-token range the gain is substantial.
    • Worth noting: the change covers only 24 layers, and the remaining 69 are still full attention. Total KV cache size and long-sequence attention cost are still dominated by those 69 layers.

    2.3 Native multimodality: capability boundary and deployment cost

    The vision encoder and the language backbone are trained end to end, so visual understanding is a native capability rather than a bolted-on module:

    • images are encoded into visual tokens that enter the same context as text tokens;
    • a single image typically maps to hundreds or thousands of visual tokens, depending on resolution and tiling strategy;
    • in a 1M-token window, a large number of high-resolution images quickly crowd out room for text and tool-call results.

    Deployment carries knock-on effects: you need a multimodal preprocessing chain, additional memory for the vision encoder, and image parsing and validation overhead on the input side.

    3. Memory is the real barrier: why “only 104B per pass” is still expensive

    3.1 Compute can be sharded; memory must be resident

    Inference has two phases:

    • Prefill: processes all input tokens at once. It is compute-intensive with high parallelism.
    • Decode: generates token by token. Every step must read the weights of whichever experts the current token routes to, which makes it memory-access-intensive.

    MoE routing is decided dynamically per token, so in theory each token can land on any 16 of the 896 experts. Engineering practice typically pre-groups tokens by expert (grouped GEMM) or uses all-to-all to dispatch tokens to the devices holding the relevant experts. Either way, nearly all experts must be schedulable at any moment, otherwise you get frequent “waiting for expert to load” bubbles.

    The conclusion is direct: weights can be sharded across GPUs, but they cannot be discarded on demand. The complete set must be resident in memory at all times.

    3.2 Back-of-envelope: what 1.56TB means

    Start with internal consistency. 2.8T parameters at 4 bits is about 1.4TB in theory (2.8×10¹² × 0.5 bytes). The actual download is around 1.56TB, roughly 10% higher, which comes from three sources:

    • unquantized modules such as token embeddings, attention projections, normalization layers, routing gates and the vision encoder;
    • per-shard headers and checksum information;
    • a small amount of padding from tensor-shape alignment.

    That in turn confirms that the vast majority of parameters really are stored at 4-bit precision. Extrapolating further:

    • spread across 8 GPUs, each card must hold about 195GB of weights;
    • that is before KV cache, activations, communication buffers and framework overhead;
    • even using high-end compute cards with more than 140GB per card, 8 cards in total still cannot hold 1.56TB of weights.

    So “8 GPUs” is not a universal bar. It is much closer to a figure that depends heavily on per-card memory. The safer approach is to work backwards: cards needed ≈ total weight size ÷ (per-card memory × 0.7). At mainstream 80GB and 140GB per-card memory, that implies at least about 28 cards and about 16 cards respectively. Both point to multi-node deployment, and the actual count has to include KV cache as well.

    3.3 KV cache: the hidden cost of long context

    KV cache grows linearly with context length and can be estimated with:

    1. KV memory (GB) ~= layers x KV heads x head dim x K/V factor x context length x bytes per element / 1024^3

    For MLA-style latent caches the K/V factor is about 1 (one low-rank vector per token per layer), while ordinary multi-head and multi-query attention typically uses 2. Because the model card does not publish attention head configuration, here are two orders-of-magnitude references (assuming 93 layers, latent vector dimension 512, FP8 storage):

    Context length Theoretical KV cache (excluding extra cost of the 69 full-attention layers) Practical meaning
    20K tokens about 1.8GB Routine long-text work; negligible
    100K tokens about 9GB Roughly one mainstream card’s entire memory
    1M tokens about 90GB Exceeds most single cards; requires memory tiering and distributed caching

    The table estimates 512 elements of latent cache per token per layer and excludes any extra cache the 69 full-attention layers may retain. Real KV overhead will be higher and stacks up multiplicatively with concurrent requests. For agent workloads this matters: one task can involve dozens of tool calls and result feedback loops, and the context easily climbs into the six-figure token range.

    3.4 Summary: what each resource class means

    Resource Main cost source How to scale Typical bottleneck signal
    Memory capacity Resident 2.8T weights plus KV cache Add cards (constrained by interconnect and parallel strategy) Load failures, concurrency ceiling
    Bandwidth and interconnect Continuous expert-weight reads during decode, all-to-all communication High-bandwidth in-node interconnect, expert-affinity scheduling High per-request latency, low GPU utilization
    Compute Prefill over long contexts Add compute cards or improve batch efficiency Long time to first token (TTFT)

    4. Quantization is the finished product, not an option: what MXFP4-only means

    The team published exactly one variant: MXFP4 weights with MXFP8 activations. There is no BF16 or FP8 release. MX (Microscaling) is a family of block-level scaled floating-point formats in which each value shares a scale factor with a small block of neighbors; common widths are FP4 and FP8. It is the output of quantization-aware training (QAT): the model simulates low-precision computation during training or continued training, so the weights actively adapt to quantization noise instead of being compressed after the fact. The upside is less accuracy loss at the same bit width; the downside is that the accuracy ceiling is locked in during training.

    4.1 Three direct consequences

    1. The accuracy ceiling is fixed. Dequantizing MXFP4 back to BF16 cannot recover information the quantization discarded. If accuracy falls short for your use case, you can only compensate with fine-tuning, distillation or RAG, not “swap in a higher-precision set of weights.”
    2. The hardware bar moves to 4-bit. Weights and activations both run at 4-bit/8-bit precision, so the compute path needs hardware support for those low-precision formats and block-level scaling. Hardware that only supports higher precision has to dequantize and compute at higher precision, which noticeably cuts throughput. The inference engine also needs to support reading and dequantizing MX formats.
    3. Fine-tuning room is limited. Full-parameter fine-tuning on a 4-bit base is expensive. The more realistic routes are parameter-efficient fine-tuning such as LoRA, or continued-pretraining-style domain adaptation, and you need to evaluate how trainable a 4-bit base actually is.

    4.2 Quantized accuracy needs validation

    Open weights do not mean zero risk. A low-precision quantized model can show degraded long-text consistency, subtle errors in generated code, or numerical drift. Before shipping, run a regression pass on your own business data set, with particular attention to long context, tool calling and structured output.

    5. How to read the numbers: interpreting vendor-reported results

    All comparative results published in the model card are vendor-reported. A few representative figures follow (unit: %):

    Benchmark area The model (self-reported) Competitor A Competitor B
    Terminal operation benchmark 65.7 66.4 58.2
    Long-horizon software engineering benchmark 51.7 46.3 —
    Terminal scientific reasoning benchmark 38.1 63.3 —
    Hard knowledge reasoning benchmark (HLE-style) 39.4 60.9 —
    • Overall picture: the vendor claims to beat Competitor A on 16 of 37 benchmarks, just under half.
    • Clear split: coding, terminal operation and agent tasks are strong, and the model beats Competitor A by about 5 points on long-horizon software engineering. Scientific reasoning and hard knowledge reasoning lag clearly, with the HLE-style gap exceeding 20 points.
    • Third-party reference: one third-party evaluator puts its intelligence index at about 55, fourth globally.

    That supports a rough read: the model is better suited to “hands-on” coding and agent workloads than to “paper-style” scientific Q&A. Selection still has to be validated on your own task distribution.

    Four cautions when reading self-reported numbers

    1. Separate benchmark from scaffold. Agent benchmarks are extremely sensitive to scaffolding (prompt templates, tool sets, retry and backtracking strategy, maximum step count). The same model can move by ten-plus points with a different scaffold.
    2. Check the scoring convention. pass@1 versus pass@k, whether multiple attempts are allowed, and whether a test-verification stage is included all affect comparability.
    3. Look at confidence intervals. Most benchmarks have limited sample sizes, and a few points of difference are often inside the noise.
    4. Test it yourself. A small evaluation on 20–50 real tasks close to your business is worth more than any leaderboard.

    6. “Open weights” is not “open source”: four key license questions

    This is the most easily misread part of the release. The repository carries the team’s own custom model license, tagged on hosting platforms as “other” rather than a standard open-source license such as MIT or Apache. The text is word-for-word identical to the previous generation’s license, with only the name changed. The main terms are indeed permissive in the MIT style: free to use, copy, modify, distribute, fine-tune and use commercially. But “permissive” is not “unconditional,” and at least four questions need answering before you build on it.

    6.1 What this license actually grants

    • Body of the text: the grant itself is quite permissive and covers research, product development and commercial services.
    • Additional obligations: custom licenses of this kind usually carry a few extra conditions, typically including:
      • a copy of the license must accompany redistribution of the model;
      • renaming or modifying the model before redistribution triggers naming or attribution requirements;
      • commercial services above a certain daily-active-user scale require separate authorization;
      • using model outputs to train a competing model is prohibited, if such a clause is present.
    • Does it count as “open source”? Under the widely used “Open Source AI Definition” thinking, something must open up weights, code, training data and training process sufficiently, under a license permitting research, modification, sharing and commercial use, before it qualifies as open source. A permissive body of text alone does not satisfy every condition. The more accurate term is “open weights”: the core difference is that what is open is the trained artifact, not a fully reproducible training process.

    6.2 A checklist for the four key questions

    Check item What to confirm Common risk
    Commercial boundary Whether direct commercial use is allowed; whether reaching a specific user scale requires separate authorization Product growth after launch triggers additional obligations
    Distribution obligations Whether redistribution requires the license, retained notices or renaming Compliance documents missing during internal platform redistribution
    Derivative models Whether a fine-tuned model can be published under your own name Derivatives lose the compliance protection of the original model
    Competitor restriction Whether using outputs to train other models is prohibited Data teams tripping the clause when distilling from outputs

    6.3 Making compliance practical

    1. Legal first. A custom license is a legal text. Any team exposing the model externally should have legal review it and produce a written conclusion.
    2. Keep a version trail. Version the weights, configuration files, inference code and license text together, and record provenance and download date.
    3. Set output boundaries. Explicitly prohibit using model outputs to train external models unless the license clearly allows it.
    4. Consider alternatives. For scenarios with strict license requirements, choose an open-weights model with a more standard license, or use a compliant API service.

    7. Self-host or call a cloud API: trade-offs across three paths

    Once you have 1.56TB of weights, the real work starts. There are three common landing paths, and you can match them to your team profile.

    7.1 Path one: self-hosted multi-node

    • Fit: data cannot leave your environment (finance, healthcare, government), deep customization and fine-tuning are required, or you have stable sustained traffic.
    • Hardware: derive the card count with the formula in section 3.2, prefer compute nodes that support 4-bit compute and high-speed inter-card interconnect, and provision high-speed networking between nodes.
    • Software: pick an inference engine that supports MX formats, MoE expert parallelism and paged KV cache management, and pin versions.
    • Operations: memory monitoring, failed-card replacement and expert load balancing all need a dedicated owner.

    7.2 Path two: dedicated cloud instances

    • Fit: traffic has pronounced peaks and troughs, you do not want a one-off capital commitment, and you need elastic scaling.
    • Common shape: rent compute instances with high-speed interconnect, or use a model-hosting service to load your own weights, usually combined with these general cloud capabilities:
      • object storage for model weights and version archives;
      • container services to orchestrate inference instances;
      • logging and monitoring with alerting to observe latency, throughput and memory headroom.
    • Upside: pay per usage, fast to scale. Watch out for: whether the instance type actually satisfies per-card memory and inter-card interconnect requirements.

    7.3 Path three: call a hosted service

    • Fit: no GPU operations capability, needs concentrated in coding and agent tasks, and you want to validate value quickly.
    • Upside: weight size and parallel strategy are not your problem, billing is per token, and elasticity is the best of the three.
    • Watch out for: data leaves your own environment, so compliance requirements need assessment, and outputs need quality spot-checks plus prompt-injection defenses.

    7.4 Decision reference

    Team profile Recommended path Key consideration
    Strict compliance, data must stay in-house Self-hosted multi-node Large one-off investment, dedicated operations
    Pronounced peaks and troughs, cost-sensitive Dedicated cloud instances Pay per usage, elastic scaling
    No GPU operations, fast validation Hosted service Data compliance, output quality spot-checks
    Deep fine-tuning required Self-hosted or dedicated cloud instances Trainability and cost on a 4-bit base

    8. Conclusion: what open weights really means

    The significance of this release is not “another large model.” It is that the complete weights of an ultra-large MoE model have been handed to developers, and three questions that APIs previously hid are now on the table:

    1. Cheap compute, expensive memory. The 104B activated parameters determine inference compute; the 2.8T total parameters determine the memory bar. “8 GPUs to start” is only true for a specific configuration, and the real card count has to be derived from per-card memory.
    2. Quantization is the finished product. With only MXFP4, the accuracy ceiling is fixed, so accuracy validation belongs alongside performance when you make a selection.
    3. Open weights are not open source. The license body is permissive but carries obligations, and commercial use, distribution and derivative models all need legal confirmation.

    For most teams the pragmatic sequence is to validate business value with a hosted API first, then decide whether to build your own based on data compliance, traffic scale and customization needs. The larger the model, the more you need to think through everything beyond “can it be downloaded.”