1. A New Phase for Open-Weight Models: From Chatting to Getting Things Done

Over the past two years, open-weight model development has largely centered on general conversation and code generation, with evaluation focused on single-turn Q&A and short contexts. As the agent paradigm has gained traction, task structure has changed fundamentally. A real-world task often involves dozens of “plan — call a tool — observe the result — revise the plan” cycles. Context can accumulate to hundreds of thousands or even millions of tokens, and a single hallucinated action at any intermediate step can cause the entire task to fail.

The latest generation of trillion-parameter Mixture-of-Experts (MoE) models emerging in the open-source community makes long-horizon agent capability a core design goal. These models have total parameters in the trillion range while activating only around ten billion parameters per token. They support million-token contexts and native image-text input, and they are post-trained with large-scale joint reinforcement learning spanning hundreds of downstream domains.

This article breaks down the technical essentials of such models from three perspectives — architecture, training, and deployment — and offers engineering recommendations for enterprise adoption.

2. MoE Architecture: Trading Sparse Activation for Capacity and Deployability

Decoupling total parameters from active parameters. MoE replaces feed-forward layers with a large number of parallel “experts.” A router dynamically selects a small subset of experts for each token. With trillions of total parameters and billions of active parameters — roughly 3–4% activation — the model’s knowledge capacity approaches that of a dense giant model, while per-token inference compute is closer to that of a mid-sized dense model. This is currently the primary path for open-weight models to push capability ceilings under hardware constraints.

Fit with agent workloads. Agent inputs are highly heterogeneous: sometimes a code repository with hundreds of thousands of tokens, sometimes a webpage state with screenshots, sometimes structured financial reports. Expert specialization allows different task patterns to naturally trigger different parameter sub-networks. Structurally, this provides a degree of isolation against cross-task interference, making it more conducive to coexistence of multi-domain capabilities than a dense model of comparable size.

Cost and engineering implications. MoE memory usage is determined by total parameters rather than active parameters. All weights must reside in memory at all times. This is the dominant deployment cost and directly leads to the quantization and multi-node parallelism discussion later in this article.

3. Attention Architecture Evolution for Million-Token Contexts

3.1 The Cost Problem of Full Attention

Standard multi-head attention requires retaining KV cache for every historical token during inference, causing memory usage to grow linearly with context length. Prefill computation is O(n²). At the million-token scale, without architectural changes, KV cache memory consumption would far exceed the model weights themselves, making long contexts economically infeasible.

3.2 A Hybrid Design: Dynamic Sparse Attention with Gated Linear Attention

The typical approach in this generation of models combines two mechanisms:

• Gated linear attention maintains a constant-size state that is “rolled forward” with historical information, similar to a writable memory matrix. It is well suited for carrying global summary information, and inference cost is independent of context length. In published designs, these layers account for the majority of the stack — dozens of layers.
• Dynamic sparse attention replaces some of the original gated multi-head latent attention (MLA) layers with sparse variants. At each step, the model first performs lightweight scoring over historical tokens and selects only the most relevant subset — typically 2048 tokens — for exact attention computation. The rest of the history is carried by the linear state.

# Simplified dynamic sparse attention
def dsa(q, k_cache, v_cache, top_k=2048):
    scores = light_score(q, k_cache)     # lightweight scoring, avoids full exact attention
    idx = topk(scores, top_k)            # keep only the most relevant historical tokens
    attn = softmax(q @ k_cache[idx].T / d**0.5)
    return attn @ v_cache[idx]

An intuitive analogy: linear attention layers act like a running notebook that continuously records long-range context, while sparse attention works like pulling files on demand — retrieving only the most relevant segments when precise details are needed. Alternating these two layer types achieves a practical balance between accuracy and cost for million-token contexts, and significantly reduces the marginal inference cost of multi-turn agent sessions.

4. Large-Scale Joint Reinforcement Learning: Teaching Models to Do, Not Just to Talk

Supervised fine-tuning can teach output formats but struggles to teach a model how to complete multi-step tasks. This generation of models generally adopts joint reinforcement learning spanning hundreds of downstream domains. Three technical points matter:

1. Verifiable rewards. Terminal operation tasks are scored by command execution results, code repair tasks by test pass rates, and retrieval Q&A by answer verifiability. Rewards must be automatically computable to enable training at scale across a massive task distribution.
2. Cross-domain joint training. Mixed sampling across hundreds of domains prevents the model from overfitting to one task type and degrading on others. This is also what enables a single set of weights to cover coding, retrieval, office productivity, and 3D design simultaneously.
3. Long-horizon credit assignment. Multi-step tasks have sparse rewards. Techniques such as episode-level rollouts and step-level process evaluation are needed to reasonably attribute final success or failure to each intermediate tool call.

From a user’s perspective, the model output shifts from “plausible-looking operations” to executable plans and tool-call sequences that are much closer to being actionable, with noticeably higher self-correction rates during intermediate steps.

5. Thinking Budget Tiers and Engineering Details for Multi-Round Tool Calls

5.1 Three Thinking Budget Tiers

Inference services offer three thinking budget levels — low, high, and max. In essence, this turns “thinking length” into an adjustable parameter. Simple extraction and format conversion can use low; complex debugging and long-term planning should use max. Note that officially published evaluations are usually obtained at the highest tier, so the reported scores correspond to a relatively high cost baseline. When estimating for production, re-benchmark using the tier you actually plan to use.

{
  "model": "moe-agent-model",
  "messages": [{"role": "user", "content": "Locate and fix the regression defect in the repository"}],
  "thinking_budget": "max",
  "tools": [{"type": "function", "function": {"name": "run_terminal"}}]
}

5.2 Multi-Round Tool Calls Must Preserve Full History

One easily overlooked hard requirement: during multi-round tool calls, historical thinking content and tool-call messages must be passed back in full. The reasoning chain and tool results across turns are the only basis for the model to restore its “task state.” If historical thinking is truncated, the model may repeat steps that have already been completed or lose constraints established midway. Engineering recommendations: preserve thinking and tool messages intact; only apply conservative summarization or pagination to oversized tool outputs; and design idempotency and retry mechanisms for tool calls.

6. How to Read Agent Benchmark Results Scientifically

Public benchmarks typically cover six dimensions: real terminal environment operation, repository-level code repair, long-duration software engineering, web retrieval and information verification, CAD/3D generation with tools, and long-document office Q&A. When interpreting these results, keep three points in mind:

• Cost baseline. High scores are mostly achieved at the maximum thinking tier. The same model will show significantly different scores and latency at lower tiers.
• Scaffolding effects. Some results are obtained with external agent frameworks. The gap between a bare model and a framework-wrapped model can be large. When reproducing results, align your toolchain accordingly.
• Scores are not the same as fit. Benchmarks reflect public task distributions. Enterprises should build their own evaluation sets from real historical tasks and perform layered attribution across retrieval, planning, and execution to judge real-world usability.

7. Deployment Practice: Quantization, Inference Frameworks, and Protocol Compatibility

• MXFP4 weight quantization. MXFP4 is a 4-bit floating-point format with block-level scaling factors. It handles the wide dynamic range of expert weights better than plain INT4. But be realistic: even at 4 bits per parameter, trillions of parameters still require hundreds of gigabytes to terabytes of memory for weights alone. A single node cannot hold them. Multi-node deployment is required, combining expert parallelism, tensor parallelism, and pipeline parallelism.
• Inference framework selection. Mainstream open-source inference engines already widely support MoE weights of this kind and sparse attention operators. Lightweight inference engines for local deployment are also available. Focus on three things when selecting: coverage and performance of sparse attention operators; prefix caching and state reuse capability; and scheduling efficiency of expert parallelism.
• Protocol compatibility and migration cost. These models typically provide APIs compatible with mainstream chat/completion protocols, so existing agent code can migrate with minimal changes. When integrating, pay attention to parsing conventions for the thinking content field in streaming output.

8. License Compliance: Open Weights Do Not Mean Unrestricted Commercial Use

Models in this category mostly adopt custom licenses rather than permissive open-source licenses. Permissive licenses generally allow nearly unrestricted commercial use and redistribution. Custom licenses commonly impose constraints such as commercial usage thresholds, attribution and branding requirements, naming rules for derivative models, and regional usage restrictions. Before enterprise adoption, complete a compliance checklist: confirm commercial terms and user-scale limits; determine whether outputs may be used to train competing models; clarify liability allocation; and assess the risk of license changes in future versions. Skipping this step is one of the most common hidden pitfalls in enterprise deployment of open-weight models.

9. Conclusion: Three Practical Recommendations

1. Evaluate first, integrate later. Build your own evaluation set from real business tasks. Validate capability fit before paying attention to scores.
2. Manage context and cost. Assign thinking budgets by task tier; use state reuse and prefix caching to reduce long-session costs; strictly follow the convention of preserving full thinking and tool history.
3. Treat compliance as an engineering problem. License review is just as important as capability evaluation.

The competitive focus of open-weight models has shifted from “how human-like the conversation feels” to “whether it can finish long-horizon tasks.” Sparse-activation MoE, hybrid attention architectures, and large-scale joint reinforcement learning are the three main levers behind this capability leap. For engineering teams, mastering agent context management and evaluation methodology has more long-term value than chasing isolated benchmark scores.