Demystifying Trillion-Parameter Open Models: Engineering Long-Horizon AI Agents

1. Background: Open Models Enter the “Trillion-Parameter, Agent-Native” Era

The gap between open-weight models and proprietary flagships continues to narrow, with competition shifting from isolated benchmark scores to real-world, closed-loop task execution. A long-horizon agent refers to a system capable of autonomously executing chained tasks—such as terminal operations, bug fixing, information retrieval, and document processing—under multi-step execution, multi-tool invocation, and ultra-long context conditions. This imposes four overlapping requirements on the model:

1. Sufficient parameter capacity: To encode world knowledge and domain-specific expertise.
2. Controllable per-token inference cost: To sustain the high-frequency, multi-turn invocation patterns typical of agents.
3. Precisely retrievable million-token context: To maintain long-horizon memory across dozens of interaction rounds.
4. Planning and error-correction capabilities via task-level reinforcement learning: To enable rollback, retry, and path switching upon failure.

Recently, a new generation of open-source models officially released its weights, with public technical specifications perfectly matching these four pillars: a Mixture of Experts (MoE) architecture with approximately 2.8T total parameters and 104B activated parameters per token, support for 1 million token context and native image-text input, and large-scale joint reinforcement learning post-training built upon its previous iteration. Its capabilities span long-horizon code agents, software engineering, 3D design, retrieval research, and office knowledge work. Using this public information as a guide, we will dissect the universal engineering logic behind these design choices.

2. Overall Architecture: How MoE Balances Capacity and Compute

A 2.8T total parameter count with 104B activated parameters means that each token’s forward pass utilizes only about 3.7% of the model. This is the core value of the MoE architecture: trading dense activation for a combination of “high capacity, low compute.”

The routing process can be conceptualized as follows:

def moe_forward(x):
    gate = softmax(router(x))            # The router scores each expert
    topk_idx = topk(gate, k)             # Only activate the top-scoring experts
    y = sum(gate[i] * expert_i(x) for i in topk_idx)
    return y

This design carries three key engineering implications:

• Decoupling capacity and compute: Expanding total parameters (adding more experts) primarily increases VRAM usage rather than per-token computation, allowing the model to continuously stack knowledge capacity at an acceptable cost.
• Load balancing is the lifeline: If routing consistently favors a few experts, it leads to compute waste and training instability. Therefore, training typically requires auxiliary load-balancing objectives.
• Increased complexity in parallelism strategies: Distributing experts across multiple GPUs requires expert parallelism and cross-card communication, making both throughput optimization and potential bottlenecks highly dependent on this setup.

For practitioners, the activated parameter scale serves as the anchor for estimating per-token compute costs, while the total parameter scale dictates the VRAM baseline required for deployment—this is a universal framework for evaluating any MoE model.

3. Attention Overhaul: A Hybrid Stack of Gated Sparse and Gated Incremental Attention

Million-token contexts face two classic hurdles: the quadratic compute overhead of full attention, and the KV cache expanding linearly with context length. The model’s public approach overhauls the attention hierarchy: replacing 24 layers of gated latent attention (a gated attention mechanism that compresses keys and values into a low-rank latent space to save cache) from its previous version with gated sparse attention, combined with 69 layers of gated incremental attention (a linear attention variant with a gated recurrent state).

The key configuration of gated sparse attention is that each attention computation step selects only about 2,048 of the most relevant tokens from the entire history. This can be illustrated as:

def gated_sparse_attention(q, K, V):
    score = selector(q, K)               # Perform lightweight relevance retrieval over the full history
    top_ids = topk(score, 2048)          # Keep only about 2048 historical tokens
    attn = softmax(q @ K[top_ids].T / sqrt(d) + gate)
    return attn @ V[top_ids]

The rationale for this layered hybrid approach lies in the complementary nature of the two mechanisms:

• Linear attention layers “summarize” the global context using a fixed-size compressed state with constant overhead, making them ideal for handling the foundational load of language modeling.
• Sparse but globally retrievable attention layers retain the ability to “precisely extract key fragments from a million tokens.” This is the capability long-horizon agents rely on most—for instance, locating the initial requirement description or a specific failed execution result after hundreds of tool calls.

The architectural takeaway is that long-context capability is not simply about “expanding the window,” but rather a systematic trade-off between computation, memory access, and retrieval precision.

4. Post-Training: Joint RL Across Hundreds of Domains and Thinking Budgets

Beyond base capabilities, the model’s differentiation stems from its post-training paradigm: joint reinforcement learning (RL) covering hundreds of downstream domains, building upon the previous generation. Compared to single-task RL, joint training must manage positive and negative transfer between tasks, domain sampling ratios, and reward signal design. Verifiable tasks (e.g., code validated via test execution, tool calls validated via real environment feedback) naturally serve as excellent reward sources, aligning with the current industry standard for RL post-training.

Another noteworthy engineering design is the tiered thinking budget: providing low, high, and max thinking budgets during inference, configurable based on task complexity. Its essence is turning “inference-time compute” into a programmable resource—simple Q&A uses a low budget to minimize latency and cost, while complex multi-step agent tasks use the maximum budget to maximize success rates. For platform providers, this offers a coarse-grained but effective cost control knob.

5. Capability Matrix and “Task-Centric” Evaluation Methodology

Based on public information, its capability assessment focuses on three real-world workflows:

1. Code Development and 3D Creation: Covers terminal operations, code repair, and structured CAD-style file generation, supporting multi-step tasks combined with tools. Repository-level repair tasks require the model to retrieve, locate, modify, and verify across multiple files—making it one of the best scenarios for testing long-horizon capabilities.
2. Knowledge and Office Work: Covers web retrieval, financial research, office document processing, and general tool invocation, testing the model’s ability to filter and integrate information amidst massive noise.
3. Evaluation Methodology Itself: Public benchmarks default to the highest thinking tier, with some evaluations conducted via command-line programming agent frameworks. This point is more critical to practitioners than the scores themselves—evaluating long-horizon models should be task-closure-centric, paired with real toolchains, rather than relying on static Q&A. When reproducing results, aligning the thinking budget and scaffolding configurations is necessary; otherwise, results may deviate significantly.

6. Engineering Deployment: MXFP4 Quantization, Inference Ecosystem, and Context Management

Low-bit Quantization: The open weights utilize the Microscaling FP4 (MXFP4) format: weights are divided into blocks that share a scaling factor within the block, storing weight values in 4 bits. Compared to traditional full-model post-quantization, block-level scaling significantly reduces precision loss while drastically compressing VRAM and bandwidth pressure. A rough estimate shows that 2.8T parameters stored in 16 bits require about 5.6TB of VRAM, whereas 4-bit storage reduces this to the 1.4TB level (with some high-precision modules added in practice)—a prerequisite for trillion-parameter models to be hosted on mid-sized clusters.

Inference Ecosystem: The weights can be loaded by mainstream community inference engines and acceleration libraries, paired with an API style compatible with industry standards, allowing existing applications to transition smoothly.

Context Management is Key in Practice: The model explicitly requires that multi-turn tool calls must retain the complete history of thought processes and tool messages. This is because the trajectories during RL training are conditioned on the full inference history. If the application side truncates or summarizes historical thoughts, model behavior may noticeably degrade. This leads to several best practices:

• Session state must be persistently stored rather than relying on temporary in-memory objects.
• The context budget must leave ample room for “thought text + tool messages,” not just user inputs.
• Utilize prefix caching to reduce the repeated computation costs of multi-turn calls.
• Design an “archive and rebuild context” strategy for ultra-long sessions, rather than appending indefinitely.

7. License Compliance: Open Source Does Not Mean Unrestricted Commercial Use

It is crucial to note that the model weights use a custom license, rather than the permissive open-source protocols common in the industry. Before adopting such models, enterprises should establish a compliance checklist: verify commercial scope and restrictive clauses, the authorization boundaries for secondary distribution and fine-tuned derivatives, whether attribution or additional authorization is required, and whether servicing deployments (providing inference services externally) are covered. Historically, multiple open-weight projects have caused migration costs due to license changes. “Review the license before entering production” should become a standard team protocol.

8. Selection Recommendations and Summary

Overall, this class of trillion-parameter open-source MoE models is suited for three scenarios: multi-turn agent applications requiring precise long-context retrieval, automated engineering flows centered on code and tool invocation, and knowledge work systems with strict data privatization requirements. Its cost structure can be quickly estimated using three public parameters: activated parameters determine per-token compute cost, context length determines VRAM and cache costs, and the thinking budget tier determines latency and overall overhead.

The technical significance of this open-weight release is the validation of a complete pathway: using sparse MoE to control compute, hybrid attention architectures to tame million-token contexts, large-scale joint RL to internalize agent behaviors, and low-bit quantization to bring deployment thresholds within reach. For engineering teams, understanding the trade-off logic behind these designs and mapping it to their own task profiles, budget constraints, and compliance requirements is far more critical than chasing benchmark rankings.