2.8T-Parameter Open-Weights MoE Model: Architecture, Quantization, and the Real Cost of Deploying a 1M-Token Context

What Open Weights Actually Mean: From API Calls to Full Control

There are typically three paths to bringing a large model into production: calling a hosted API, self-hosting from open weights, and fine-tuning or continued training on top of open weights. When a model reaches the trillion-parameter scale, most teams default to the first option, because the resource threshold and engineering complexity of self-hosting are extremely high.

This model, however, ships with its complete weights open. According to public information, the weight repository occupies roughly 1.56TB of storage, with about 2.78 trillion parameters counted in safetensors format. On the surface that is just “a very large set of files,” but it sends a clear signal to developers: the model architecture, quantization parameters, layer configuration, and expert distribution are all auditable, reproducible, and open to secondary development.

For engineering teams, the value of open weights shows up on at least three levels:

Controllable inference pipeline. Hosted APIs are convenient, but request latency, concurrency limits, and data-handling policies are all dictated by the provider. Once self-hosted, a team can tailor batching strategies to its workloads, adjust caching layers, and even embed the model into an internal data loop without any data leaving the premises.

Fine-tuning and adaptation become possible. Closed APIs don’t let you touch model parameters, but open weights allow continued pretraining, supervised fine-tuning, or preference alignment for specific verticals. For industries like legal, healthcare, and finance — where terminology and output style are tightly constrained — this capability is a hard requirement.

Auditing and compliance. A growing number of enterprises need to evaluate a model’s behavioral boundaries, safety mechanisms, and potential biases. Without weights, such evaluation stops at the black-box level; with weights, security teams can dig into expert routing behavior, attention distributions, and the characteristics of output distributions.

That said, it pays to be clear-eyed: downloading 1.56TB of files is only the starting point. The real challenges are making 2.8 trillion parameters run efficiently on finite hardware and understanding the inference cost structure behind the figure “104B parameters activated per token.”

Architecture Breakdown: How MoE, Hybrid Attention, and the Vision Module Fit Together

The model follows the now-familiar sparse mixture-of-experts (MoE) design: 2.8T total parameters, but only about 104B activated in each forward pass. This “large capacity, small activation” pattern has been validated across the industry as an effective way to balance capability against cost.

The architecture can be unpacked on four levels:

Depth of the backbone. The model has 93 layers — considerably deeper than a typical large language model. Extra depth brings stronger nonlinear expressiveness but raises the bar for gradient flow and training stability. Notably, these 93 layers are not a homogeneous stack: 69 use a KDA structure and 24 use a gated sparse attention structure. This hybrid design suggests a functional division of labor between shallow and deep layers — shallow layers may handle finer-grained local information extraction, while deeper layers focus on long-range dependencies and global semantic integration.

Evolution of the attention mechanism. Full attention is optimal in modeling accuracy, but its compute cost grows quadratically with sequence length. Once the context window extends to 1M tokens, full attention becomes computationally untenable. The model therefore introduces sparse attention in part of its layers: instead of interacting with the entire history, each token selectively attends to a subset of key positions according to some policy. A gating mechanism further lets the model learn “which positions are worth attending to” rather than relying on a fixed sparsity pattern. This design matters most in long-document understanding and long-conversation memory.

Expert layer configuration. The model contains 896 experts; each token activates 16 of them, plus 2 shared experts that participate in every computation. An 896-choose-16 sparsity ratio means each token draws on only about 1.8% of expert capacity, which dramatically cuts effective compute. But sparse routing has an inherent problem: if some experts are selected constantly while others sit idle, you get load imbalance and wasted capacity. Routing strategy, load-balancing loss design, and regularization techniques such as expert dropout are therefore core engineering challenges when training models of this type.

The vision module. The model accepts image-and-text input, with visual encoding handled by a Vision Transformer variant of roughly 401M parameters. Relative to the 2.8T backbone, 401M is tiny, but in the multimodal pipeline it plays the role of a “feature adapter”: it converts image signals into a representation space aligned with text tokens, which are then fed into the backbone for joint inference. This lets the model extend its multimodal capabilities without materially increasing backbone pretraining cost.

Quantization Strategy: What MXFP4 Weights and MXFP8 Activations Mean in Practice

Weights are stored in MXFP4 format, and activation computation uses MXFP8. Neither choice is mere “compression” — both are integral parts of the training pipeline.

Conventional post-training quantization compresses precision after training is complete and usually costs some performance. Quantization-aware training (QAT), by contrast, simulates quantization error during training, so parameters learn representations that are insensitive to low precision. The result is a model that holds up well at FP4 — rather than an FP16-trained model being “squeezed down” after the fact.

To put the bit-width in concrete terms: FP4 stores each weight in 4 bits, one quarter of the FP16 footprint. That directly explains why a 2.8T-parameter model repository is “only” about 1.56TB — at FP16, weights alone would require roughly 5.6TB. Using FP8 for activations is a compromise between compute efficiency and numerical stability: FP8’s floating-point range handles the dynamic value ranges inside large models better than INT8, especially in deep networks where activation distributions tend to be long-tailed.

For teams planning to self-host, these quantization parameters directly shape hardware selection. FP8 activation compute requires GPUs that support the corresponding matrix instruction sets; FP4 weights save memory bandwidth but introduce extra overhead when dequantized to compute precision. Fortunately, mainstream high-performance inference engines already support this hybrid-precision mode, so deployers generally do not need to write compute kernels from scratch.

In practical terms, the most direct benefit of QAT is: within the same VRAM budget you can run a larger model or a longer context, and at the same model scale you can process more batches in parallel. The trade-off is that training itself becomes more complicated — the strategy for injecting quantization error, the design of straight-through estimators in backpropagation, and the differing quantization sensitivity of individual layers all need fine-grained control during training.

Deployment Threshold and Inference Optimization: From Hardware Selection to Serving

The first hurdle in local deployment is VRAM. A widely cited third-party estimate puts it this way: a server built from 8 next-generation high-bandwidth-memory accelerator cards, with roughly 2.3TB of total VRAM, is needed to run the model comfortably. The estimated cost of that hardware falls in the $550,000–$650,000 range.

The significance of that number is not just “expensive” — it delineates the boundary of where self-hosting makes sense. For large organizations with enormous annual request volumes, highly sensitive data, or deep fine-tuning needs, keeping the model resident on owned hardware may be far cheaper in long-run total cost than paying per token. For smaller teams or exploratory projects, hosted endpoints compatible with OpenAI- and Anthropic-style APIs are the more rational choice.

At the inference engine layer, the model is already supported by several mainstream high-performance frameworks. These typically provide: continuous batching for concurrent throughput, paged KV cache management to reduce memory fragmentation in long-sequence scenarios, configurable hybrid expert-parallel and tensor-parallel strategies, and efficient scheduling of FP8 compute kernels.

For teams building their own serving stack, a reasonable deployment path looks like this:

1. Characterize the workload: average input length, output length, concurrent requests, peak QPS
2. Estimate per-GPU KV cache capacity and batch size based on active parameters (104B with quantization)
3. Pick an inference engine, follow the official deployment recipe, and configure expert and tensor parallelism
4. Load-test with representative traffic, watching TTFT and TPOT for long-context requests in particular
5. Wrap it as a standardized service API with rate limiting, fallbacks, and monitoring

One caveat deserves special attention: the relationship between context length and memory usage is nonlinear. A 1M-token context is supported in theory, but actual KV cache consumption depends on the caching strategy, precision, and batch size. In extreme long-sequence scenarios, the KV cache can consume more VRAM than the model weights themselves. When deploying long-context workloads, prioritize inference framework versions with efficient KV cache compression and eviction policies.

Real-World Agent Capability: From Benchmark Scores to Net Task Improvement

Traditional LLM evaluation relies on static datasets, measuring accuracy at producing standard answers for given inputs. Agent tasks are fundamentally different: an agent must make tool calls across multiple turns, observe environment feedback, correct its own behavior, and ultimately complete an open-ended goal. That kind of capability is hard to measure with multiple-choice questions or fixed answers.

The evaluation discussed here focuses on agent performance in “real user sessions.” Signals come from multiple dimensions: whether the user explicitly confirms task success, the user’s sentiment toward the model’s output, how well the model adjusts after being corrected, and the actual execution success rate of shell commands. These signals combine into a “net improvement” metric that reflects how much users actually benefited from the model’s agent behavior.

Under this evaluation, the model scores +5.1% net improvement — first among open-weight models and 13th on the overall leaderboard that includes closed models. Compared with the previous generation’s +0.91% and 6th place, the jump is substantial. On the Bash reliability sub-metric, performance leapt from -4.5% to +7.9% — a meaningful signal: the previous generation apparently stumbled frequently on shell command execution, while the new model not only fixed the negative showing but moved clearly into positive territory.

Three technical factors help explain how capability gains translated into these numbers:

First, sparse attention helps the model maintain steadier context tracking over long-horizon agent sessions, reducing the “forgotten task goal” and “repeated mistaken operations” caused by overlong contexts.

Second, larger total capacity combined with efficient sparse activation gives the model a stronger basis for judgment in tool selection, command construction, and error diagnosis — the steps that demand broad knowledge reserves.

Third, quantization-aware training may act as a form of implicit regularization, helping the model learn more robust representations and maintain steadier output quality amid the noise and uncertainty of real environments.

Engineering Guidance: Reasoning Budgets and Multi-Turn Tool Calls

The model’s hosted API ships with two designs that are highly practical for developers: a reasoning_effort parameter that controls reasoning depth (low, high, max), and a requirement that multi-turn tool calls pass the previous turn’s reasoning_content back along with tool_calls. These map directly onto two of the biggest pain points in agent development: controllable inference cost and multi-turn state consistency.

On reasoning budget control, reasoning_effort essentially tunes the trade-off between inference compute and output quality. Lightweight tasks — simple queries, format conversion, information extraction — work fine at low, noticeably cutting first-token latency and per-call cost. Heavy tasks — complex planning, multi-step reasoning, code generation — warrant high or max, letting the model reason internally more thoroughly before answering. In practice, an effective strategy is dynamic routing by input complexity: run a lightweight classifier to gauge task difficulty, then choose the reasoning depth accordingly.

On multi-turn tool calls, the key point is maintaining state integrity. In an agent session, the model may loop through “plan → call tool → observe result → adjust plan → call again.” If each turn returns only the final text and discards the intermediate reasoning state, the model cannot continue its previous chain of thought, and behavior becomes fragmented and inconsistent. Passing the previous turn’s reasoning content and tool call information back in full is therefore the prerequisite for coherent multi-turn agent behavior.

At a higher architectural level, a robust agent service should include: an inference layer (generation and reasoning), a tool execution layer (sandboxed execution and result return), a state management layer (session context and reasoning chain persistence), and a monitoring layer (success rate, latency, and cost tracking). With the weights open, the inference layer can be built on self-hosted infrastructure, while the other layers can reuse a team’s existing technology stack.

Summary and Recommendations

The open release of a large-scale sparse model means more to the developer community than “one more downloadable model.” It demonstrates the feasibility of a technical direction: use trillions of parameters for broad knowledge coverage, keep per-token activation in the hundreds of billions to control actual compute cost, use quantization-aware training to lower storage and deployment barriers, use sparse attention to support ultra-long contexts — and ultimately deliver a measurable capability jump on real agent tasks.

For teams at different stages, the advice in this article boils down to three tracks:

Exploratory teams: Do not invest in self-hosting yet. Validate the model’s real-world fit through compatible APIs first, focusing on long-context understanding, multi-turn tool calling, and how the reasoning depth setting affects output quality. Let the data decide.

Growth-stage teams: Once monthly spend crosses a meaningful threshold, or data sensitivity mandates private deployment, start a small pilot. Verify inference engine stability and throughput on one or two high-VRAM servers, and build a cost model for long-term comparison against hosted options.

Mature organizations: Teams with ample GPU resources and platform engineering capability can build domain fine-tuning, inference kernel optimization, and standardized service wrapping on top of the open weights. Larger enterprises’ platform teams can even make the model one of the default options in an internal model gateway, exposing a unified agent capability interface.

Technology will keep evolving, but the decision logic stays clear: define the business problem first, then match model capability; validate results before committing resources; get the pipeline working before chasing peak optimization. In the trillion-parameter era, rationality is scarcer than compute.