How Ultra-Large MoE Models Support Long-Horizon Agent Capabilities

1. A Technical Inflection Point for Long-Horizon Agents

Recent open-source ultra-large foundation models illustrate a representative technical direction: a Mixture-of-Experts (MoE) architecture with roughly 2.8T total parameters but only about 104B activated per forward pass, supporting 1M-token contexts and native image-text input, followed by large-scale reinforcement learning post-training across long-horizon coding agents, software engineering, and knowledge work.

This article does not focus on any specific product. Instead, it breaks down the general technical problems behind this approach:

1. How to decouple model capacity from inference cost — Sparse activation via MoE architectures.
2. How to make million-token contexts affordable to compute and store — Hybrid sparse attention designs.
3. How to move models from “answering questions” to “getting work done” — Joint reinforcement learning across hundreds of downstream domains.

These three core problems, combined with three engineering topics — quantization and deployment, evaluation interpretation, and license compliance — form a complete map for selecting and deploying long-horizon Agent models today.

2. What Long-Horizon Agents Demand from Models

Unlike single-turn Q&A, long-horizon Agent tasks have three distinctive characteristics, each imposing a specific capability constraint on the model:

• Multi-step, long time horizons: A real task often involves dozens or hundreds of “think — call a tool — observe results — correct” loops. The model must maintain goal consistency across hundreds of thousands of tokens without forgetting early constraints.
• Continuously expanding context: Code snippets, web content, and execution logs returned by tools keep being written into the context. Context management pressure grows linearly, or even explosively, with the number of steps.
• The capability is execution, not recitation: Knowing how to write code is not the same as fixing a defect in a real repository. The latter requires a closed loop of locating the issue, making changes, running tests, and iterating based on failure feedback.

This leads to three hard requirements for the model: sufficiently large knowledge capacity (covering programming languages, framework protocols, and domain knowledge), sufficiently long effective context (not just a large window, but genuinely usable distant information), and sufficiently strong multi-step execution ability (planning, error correction, and tool orchestration). The architecture and training designs below are organized around these three points.

3. MoE: Leveraging Trillion-Scale Knowledge Capacity with About 3.7% Active Parameters

3.1 Core Principle

The core idea of MoE (Mixture of Experts) is to replace the feed-forward network layers in a Transformer with multiple parallel “expert networks,” while a router dynamically selects a small number of experts for each token. A simplified illustration:

# Simplified MoE forward pass (pseudocode)
def moe_layer(x, router, experts, top_k=2):
    gate = softmax(router(x))            # [seq_len, num_experts] routing scores
    w, idx = topk(gate, k=top_k)         # only top_k experts are active per token
    y = weighted_sum(w, idx, experts)    # only selected experts participate in compute
    return y

3.2 What “2.8T total / 104B active” Actually Means

The ratio of active parameters to total parameters is roughly 3.7%. This ratio reveals the essential value of MoE: knowledge capacity is decoupled from per-inference cost. Total parameters determine how much the model “remembers” — critical for code, multi-domain knowledge, and multimodal understanding. Active parameters determine the computational load and latency of each inference — an activation scale around 104B is comparable to mainstream flagship dense models. In other words, the model can “know a great deal” while “only recalling a small relevant subset each time.”

3.3 Engineering Costs Must Be Taken Seriously

MoE is not a free lunch. Three points deserve attention during deployment:

• Memory residency: All experts must reside in GPU memory. At MXFP4 (approximately 0.5 bytes per parameter), 2.8T parameters require about 1.4TB of memory, which means expert parallelism across multiple machines is mandatory; a single node cannot handle it.
• Load balancing: If routing concentrates on a few popular experts while others sit idle, both compute and quality suffer. Training requires auxiliary balance losses, and inference requires redundancy and reordering strategies.
• Communication overhead: Expert parallelism introduces cross-card All-to-All communication, which is sensitive to cluster interconnect bandwidth and is a key bottleneck for inference throughput.

4. Hybrid Sparse Attention: A Viable Path to Million-Token Contexts

4.1 Two Obstacles for Full Attention

Standard self-attention scales quadratically with sequence length, while the KV cache grows linearly. The KV cache size in bytes is approximately 2 × number_of_layers × per-token_key-value_dimension × sequence_length × bytes_per_element. As a rough estimate: with 100 layers, a per-token key-value dimension of 1024, and 8-bit storage, each token requires about 200KB, so 1M tokens would need about 200GB — and that is for a single request. Full attention over million-token contexts is neither computationally affordable nor storage-friendly.

4.2 A Layer-Wise Hybrid Design

The typical solution in recent models is not to force every layer to attend to the full history, but to mix three types of attention across layers:

• Latent compressed attention: Projects keys and values into a low-dimensional latent space, substantially reducing KV cache volume while retaining relatively high-fidelity local and global modeling.
• Gated sparse attention: Each query no longer scans the entire history. Instead, a learned selection mechanism picks only about 2048 of the most relevant tokens from the past. For a 1M-token context, attention compute in a single layer drops to roughly 1/500 of full attention and becomes nearly decoupled from total length.
• Gated incremental (linear) attention: Compresses the entire history into a fixed-size recurrent state, with constant computational cost, and is particularly good at capturing long-range sequential patterns.

Take, for example, a recent model version that replaces some original attention layers with gated sparse attention and combines them with gated incremental attention layers. This mixing reflects a pragmatic engineering philosophy: shallow layers handle local patterns, a few key layers are responsible for global retrieval, and the remaining layers use linear states to cheaply “remember” history. For long-horizon Agents, this matches real needs — the trajectory is long, but at each step only a few critical pieces of evidence need to be precisely revisited: an earlier error message, a specific constraint, an important tool return.

5. Joint Reinforcement Learning: From “Answering Questions” to “Getting Work Done”

5.1 Why Supervised Fine-Tuning Is Not Enough

Supervised fine-tuning is essentially imitation of static demonstrations, while Agent tasks require exploration and error correction. How to adjust after a failed tool call, or how to localize a defect after a failing test — these “procedural capabilities” are hard to cover with static annotations. Reinforcement learning lets the model learn by trial and error in real or simulated environments, using verifiable signals (whether tests pass, whether commands execute successfully, whether retrieved evidence supports a conclusion) as rewards. Only then can stable execution strategies emerge.

5.2 The Value of Joint Training

Single-task RL often leads to capability trade-offs, where gains in one area come at the expense of another. Joint reinforcement learning over a task pool spanning hundreds of downstream domains has two layers of value: first, tasks regularize each other, mitigating catastrophic forgetting; second, general capabilities such as multi-turn tool orchestration and long-context retrieval can transfer across domains. In practice, this requires careful control of task mixture ratios and a difficulty curriculum that keeps the model in a “slightly challenging but solvable” zone.

5.3 Thinking Budget Tiers: An Engineering Trade-Off Between Cost and Quality

Providing low, high, and max thinking budgets is essentially about turning “reasoning depth” into a schedulable resource:

Tier	Suitable scenarios	Cost / latency	Recommendation
low	Format rewriting, simple queries, batch processing	Low	Default tier for high-frequency, low-risk tasks
high	Routine code fixes, data analysis	Medium	Balanced tier for most business tasks
max	Long-horizon engineering, complex diagnosis, critical paths	High	Use for difficult tasks and acceptance testing

An engineering best practice is to route automatically by task risk: use high for the main code review pipeline, and switch to max for deep root-cause analysis after test failures. This keeps the overall cost under control while preserving quality on critical paths.

5.4 The Agent Value of Native Multimodal Input

Native image-text input means visual tokens and text are modeled in a unified context without an external OCR pipeline. For Agent scenarios, this directly expands the boundary of executable tasks: reading application screenshots before operating the interface, interpreting design drawings before generating CAD code, parsing report images before completing data verification.

6. Quantization and Deployment: MXFP4 Weights and Inference Engineering

6.1 Why MXFP4 Suits Ultra-Large Models

MXFP4 is a microscaling block floating-point format: a small group of parameters shares a scaling factor, and each parameter within the group is represented in 4-bit floating point. Compared with traditional integer quantization, the floating-point representation has a more favorable dynamic range and more controllable precision loss for ultra-large models. Compared with BF16, weight size and memory bandwidth pressure drop to roughly one quarter — a significant benefit for memory-sensitive MoE deployment and bandwidth-constrained long-context decoding.

6.2 Key Points for Self-Hosting and API Integration

• Inference frameworks: Mainstream open-source inference engines have broadly adapted to such large models. When choosing one, focus on expert parallelism support, continuous batching, and KV cache management.
• API protocols: Open weights are typically accompanied by API services compatible with common conversational protocols, keeping migration costs low for existing Agent frameworks.
• Critical engineering detail — preserve full history: In multi-turn tool calling, you must retain complete historical reasoning and tool-call messages. The reason is that the model was trained during reinforcement learning on the distribution of complete trajectories. Truncating history at inference time artificially creates a mismatch between training and inference distributions, which typically manifests as repeated calls to the same tool, forgotten early constraints, and clearly degraded behavior. A layered memory strategy — full recent context plus summarized or retrieved distant context — can be used, along with compression of bulky tool returns.

# Multi-turn tool calling loop (generic protocol illustration)
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
for step in range(MAX_STEPS):
    resp = post(f"{BASE_URL}/v1/chat/completions", json={
        "model": MODEL_ID,
        "messages": messages,        # preserve full reasoning and tool-call history
        "tools": TOOLS,
        "thinking_budget": "max",    # thinking budget tier (illustrative field)
    })
    msg = resp["choices"][0]["message"]
    messages.append(msg)
    if msg.get("tool_calls"):
        for call in msg["tool_calls"]:
            result = dispatch(call)  # execute tool locally
            messages.append({"role": "tool",
                             "tool_call_id": call["id"],
                             "content": to_text(result)})
    else:
        return msg["content"]

7. Interpreting Evaluations: From Benchmark Scores to Business Validation

Published evaluations typically cover dimensions such as terminal operations, long-horizon software engineering, code repair, deep web retrieval, CAD generation, and office document Q&A. When interpreting these results, three principles are worth following:

• Align the thinking tier: Most public scores are obtained at the max thinking tier by default. Comparing them directly against self-test results at the low tier is meaningless.
• Watch for framework encapsulation: Some scores are achieved with the help of Agent programming frameworks. Prompt design, tool implementations, and retry strategies in the framework can all significantly affect scores. Reproduction requires aligning the environment.
• Beware of distribution differences: Qualitatively, retrieval and tool-assisted generation tasks tend to score higher, while multi-step real software engineering remains the hardest area — which overlaps heavily with actual enterprise pain points. Therefore, building an in-house evaluation set close to your business distribution, followed by canary validation, is essential.

8. License Compliance and Deployment Paths

8.1 Open Weights Do Not Always Mean Unrestricted Commercial Use

It is worth noting that weights for such models often use custom licenses rather than the permissive licenses common in the industry. Before adoption, enterprises should complete a compliance review: whether commercial use is allowed and whether there is a scale threshold, redistribution and modification terms, whether source attribution is required, whether training competing models is restricted, responsibility for output content, and whether terms can be changed or revoked. Legal involvement throughout the process and retaining review records is recommended.

8.2 Three Deployment Paths

• Cloud API: Best for rapid validation. If the team is focused on the business rather than infrastructure, managed inference services, private deployment options, and Agent orchestration — combined with logging, monitoring, and alerting — can significantly shorten the path from validation to production.
• Dedicated deployment: Suitable for scenarios where data must remain on-premises or deep customization is required, but it demands in-house GPU clusters and dedicated operations staff.
• Hybrid architecture: Sensitive data pipelines are processed locally, while general capabilities run in the cloud, balancing compliance and elasticity.

Conclusion

The essence of ultra-large MoE models supporting long-horizon Agents is a systems engineering triad of architecture, attention, and post-training: MoE resolves the tension between capacity and cost, hybrid sparse attention resolves the tension of long contexts, and joint reinforcement learning resolves the tension of execution capability. What ultimately determines the business ceiling is often the engineering work outside the model itself — deployment details, evaluation methodology, and compliance review. For technical teams, understanding the general principles behind this direction has more long-term value than memorizing any specific benchmark score.