I. Background: The Convergence of Two Open-Source LLM Evolution Paths

Over the past year, the competition among open-source large language models has centered on two main trajectories. The first is architectural sparsification: utilizing Mixture of Experts (MoE) to push total parameters to the trillion scale while keeping activated parameters per token at the billion level, achieving “high capacity with low compute cost.” The second is task-paradigm Agentic capability: shifting evaluation metrics from single-turn Q&A quality to the ability to consistently complete long-horizon tasks involving dozens of rounds of tool calling, retrieval, and code execution.

Recently, an open-source model with approximately 2.8T total parameters and 104B activated parameters officially opened its weights. Supporting million-token contexts and native image-text inputs, it serves as a representative sample of these two converging paths. Rather than providing a product review, this article leverages its publicly available technical elements to outline the technical paradigm of this generation of open-source models and offer implementation recommendations for engineering teams.

II. MoE Architecture: Decoupling “Knowledge Capacity” from “Compute Cost”

2.1 Fundamentals of Sparse Activation

MoE replaces the feed-forward layers in Transformers with multiple parallel “expert” subnetworks. A router selects only a few experts per token to participate in computation. Using this sample model as an example: the 2.8T total parameters dictate the upper limit of the model’s knowledge capacity, while the 104B parameters actually activated per token determine the computational and memory access overhead for a single forward pass. Decoupling the two is the key to “affordable training of massive models and medium-cost inference.”

2.2 Benefits and Trade-offs

Benefits include:

• Larger capacity under equivalent inference compute, resulting in better coverage of long-tail knowledge and multiple domains.
• Fixed activation parameters mean inference costs do not scale linearly with total parameters.

Regarding trade-offs, engineering teams must acknowledge three key challenges:

• Memory Wall: All experts must reside in GPU memory. Even stored in 4-bit precision, 2.8T parameters require roughly 1.4TB, which cannot fit on a single machine. Expert parallelism combined with multi-node deployment is mandatory.
• Load Balancing: Routing skew can cause certain experts to overheat. Auxiliary losses or explicit balancing strategies are required to constrain this.
• Communication Overhead: Expert parallelism introduces All-to-All communication, making the system highly sensitive to inter-node bandwidth.

2.3 Implications for Model Selection

When evaluating MoE models, looking solely at “total parameters” is insufficient. It is crucial to examine activated parameters, weight footprint, and minimum deployment topology—these factors determine the true cost per request and the hardware barrier to entry.

III. Million-Token Context: Hybrid Sparse and Linear Attention

3.1 The Quadratic Bottleneck of Full Attention

Standard self-attention incurs O(n²) computational and memory overhead for a sequence of length n. If full attention were directly applied to million-token inference, both the KV cache and the attention matrix would be unmanageable. The industry generally relies on three solutions: latent compression for KV, sliding window local attention, and sparse or linear attention.

3.2 The Model’s Hybrid Approach

According to public materials, the model replaces a subset of attention layers with dynamic sparse attention: for each query token, a lightweight scoring mechanism dynamically selects approximately 2,048 of the most relevant tokens from millions of historical tokens for precise computation. This reduces attention cost from “correlating with full sequence length” to “correlating with the selected block size,” while retaining the ability to precisely trace back distant key information.

The majority of the remaining layers employ gated Delta rule-based linear attention: using a constant-sized recurrent state to iteratively process historical information with O(n) complexity. This excels at carrying long-range semantic flows at low cost and latency.

This hybrid approach—where linear attention forms the majority and sparse attention the minority—is essentially a division of labor: linear layers ensure the model “remembers,” while sparse layers ensure it “retrieves accurately.” The direct benefit for inference is a significantly reduced KV cache and improved decoding throughput, restructuring the latency and cost profile for long-context scenarios.

IV. Post-Training: Multi-Domain Joint RL and Thinking Budgets

4.1 Why Joint RL Matters

Pre-training provides capability; post-training defines behavior. Built upon its base model, this model underwent large-scale joint reinforcement learning (RL) covering hundreds of downstream domains. Reward signals are primarily derived from verifiable outcomes—whether code passes tests, tool calls succeed, or retrieved answers are correct. Compared to single-task RL, joint training offers two major advantages: first, positive capability transfer, as tool calling and code repair share a “plan-execute-feedback” structure that fosters mutual improvement; second, the suppression of catastrophic forgetting, as multi-domain mixed sampling prevents “learning one, forgetting another.”

4.2 Three-Tier Thinking Budget Design

Inference offers three thinking budget tiers: low, high, and max. This essentially turns “reasoning length” into an adjustable parameter. The low tier suits low-latency tasks like classification and extraction; high covers standard code and document processing; max targets long-horizon Agents and complex debugging, allowing for longer chains of thought and more self-checks. Engineering teams should implement difficulty estimation at the gateway layer and route tasks accordingly to balance cost and quality.

V. Three Main Battlefields for Long-Horizon Agents and Key Engineering Details

5.1 Code and Software Engineering

Evaluation dimensions cover terminal operations, code repair, and deep software engineering. Implementation essentials include: executing commands and tests in isolated sandboxes and feeding failure information back to the model; utilizing repository retrieval tools for multi-file scenarios rather than stuffing the entire repository into the context; and adopting a test-driven cycle of “reproduce-locate-fix-regress.”

5.2 3D/CAD Generation

By invoking parametric modeling tools, natural language requirements are converted into executable modeling scripts, which are then corrected using rendering or geometric validation feedback. Such tasks are typically evaluated “with tools,” and the performance gap between pure text models and tool-augmented models is most prominent here.

5.3 Knowledge Work

Web retrieval requires a complete pipeline of “generate query-browse-filter-cite.” Financial research tests multi-document cross-referencing and table reasoning. Office document processing tests the model’s multimodal parsing capabilities for complex layouts and tables.

5.4 An Easily Overlooked Detail: Preserving Complete History

The model explicitly requires retaining complete historical thinking and tool call messages during multi-turn tool interactions. The reasoning is: the chain of thought carries previous task plans and intermediate conclusions; discarding it leads to redundant reasoning or even self-contradiction. The pairing relationship between tool calls and results is the basis for the model to determine “what has been done and what failed.” Therefore, when the server trims the context, it should truncate by turn rather than by character count. If necessary, only summarize early tool results while preserving the decisive thinking and the tool-calling skeleton.

VI. MXFP4 Quantization and Deployment Options

MXFP4 is a 4-bit scheme within the open microscaling floating-point format family: a group of weights shares a single microscaling factor, encoded in 4-bit floating point. Compared to traditional INT4, its dynamic range is closer to floating point, making it more compatible with the weight distribution of massive models and eliminating the need for complex calibration processes in most scenarios. Estimating with 2.8T parameters, the weights occupy about 1.4TB. Factoring in activations, KV cache, and framework overhead, a multi-node cluster is the baseline requirement.

These weights have been adapted for mainstream open-source inference engines that support continuous batching and paged cache management. They also provide APIs compatible with industry-standard chat completion protocols, ensuring low migration costs. An example request structure is as follows:

{
  "model": "<model-id>",
  "messages": [
    {"role": "user", "content": "Fix the failing test cases in the repository"},
    {"role": "assistant", "reasoning": "……historical thoughts……"},
    {"role": "tool", "name": "terminal", "content": "……execution results……"}
  ],
  "reasoning_effort": "max",
  "tools": ["terminal", "web_search", "file_editor"]
}

Regarding deployment patterns: core scenarios with sensitive data and stable loads are suitable for self-hosted clusters, provided the team possesses operational expertise in expert parallelism and quantized inference. Teams lacking large-scale compute reserves can utilize the Model-as-a-Service (MaaS) capabilities of cloud platforms like Baidu AI Cloud for direct invocation or hosted fine-tuning. A hybrid approach is also viable, keeping sensitive data for local inference while routing general tasks to the cloud. Cost accounting should revolve around “activated parameters × concurrency × context length,” with a strong focus on the impact of KV cache strategies in long-context scenarios.

VII. Correctly Interpreting Evaluation Scores

Public evaluations cover dimensions such as terminal operations, code repair, deep software engineering, web retrieval, CAD generation, and office Q&A. Three points should be noted when reading these scores: First, results are typically achieved at the highest thinking tier alongside a mature Agent framework; changing tiers or frameworks will yield significantly different outcomes. Second, scoring protocols across different benchmarks (e.g., whether tools are provided, number of feedback rounds) vary, making direct cross-benchmark comparisons inappropriate. Third, enterprises are advised to extract real tasks from their business logs to build custom evaluation sets, conduct regression testing with fixed prompts and toolsets, and incorporate cost and latency dimensions.

VIII. Licensing and Compliance

The weights utilize a proprietary license rather than permissive ones like Apache 2.0 or MIT. Before adoption, review the following closely: restrictions on commercial scale; requirements for attribution or branding; whether models derived from secondary training can be offered as managed services externally; and whether the weight license is separated from the accompanying code license. Additionally, whether self-hosting or calling APIs, content safety, log retention, and data compliance must be integrated into the overall solution.

IX. Conclusion

The combination of trillion-parameter MoE, hybrid sparse and linear attention, multi-domain joint reinforcement learning, and open MXFP4 quantization outlines the current technological convergence direction for open-source LLMs. For enterprises, the rational path is to anchor on business scenarios and cost structures, build internal evaluations first, determine deployment patterns second, and finally treat context management and compliance review as engineering challenges equally important as the model’s inherent capabilities.