A Massive Open-Weight MoE Model: Long-Horizon Agents and Multi-Step Tool Calling in Practice

1. Background: From Single-Turn Inference to Long-Horizon Agents

As software engineering, terminal automation, 3D design, and knowledge processing tasks become increasingly complex, single-turn question-answering style inference can no longer meet real production requirements. Developers are more concerned with whether a model can maintain goal consistency over an extended time span, continuously invoke external tools, preserve critical context across multi-turn interactions, and backtrack, correct, or retry when necessary.

In this context, large-scale sparse expert models are emerging as the foundational capability layer for building long-horizon agents. Their core value is not simply the expansion of parameter count. Rather, it lies in allocating a limited compute budget to the most relevant tokens and historical states, striking a balance among long context, multi-tool calling, and complex reasoning.

This article focuses on a typical open-weight, large-scale sparse model. It discusses the architectural design, inference budget tiers, evaluation results, and key constraints when integrating the model into agent systems. The content is written from a general technical perspective and is not tied to any specific cloud vendor or commercial platform. Generic terms such as “cloud services,” “container platforms,” and “object storage” are used when referring to common capabilities like cloud resources, model hosting, and containerized deployment.

2. Model Architecture: Sparse Experts and Adaptive Attention

The model discussed here uses a Mixture-of-Experts architecture with approximately 2.8T total parameters and around 104B activated parameters. Compared with dense models, this design activates only a subset of experts during inference, allowing for greater model capacity while keeping computational cost under control.

2.1 Long Context and Native Multimodal Input

The model supports a context window of approximately 1 million tokens and natively handles image-text inputs. This means developers can place large code repositories, multiple design documents, terminal output logs, or consecutive screenshots within the same context without frequently truncating history.

From an engineering perspective, native long context reduces the complexity of external memory systems. For many medium-length development tasks, developers can first place core materials into the context and let the model perform continuous reasoning over them. When tasks span hours or longer, external storage, summarization, and indexing mechanisms are still required to prevent context bloat and information decay from earlier segments.

2.2 From MLA to DSA: Adjustments in the Attention Mechanism

Compared with the previous generation, this model replaces gated multi-head latent attention in some layers with gated dynamic sparse attention, used in combination with the original attention layers. At each step, dynamic sparse attention selects roughly 2048 tokens from historical context to participate in attention computation, rather than performing full computation over all historical tokens.

This approach offers two direct benefits in long-context scenarios:

1. More controllable computation: As context grows, attention cost does not scale linearly to unacceptable levels.
2. Better focus on key information: The model can learn to select tokens from long history that are more relevant to the current step, improving information utilization.

However, sparse attention also introduces additional challenges. The model only explicitly attends to a subset of historical states. If external systems write critical constraints into early history and subsequent sparse selection fails to cover them, a form of “forgetting” may occur. Therefore, in practical agent systems, task state compression, repeated emphasis of key information, or external retrieval compensation are still needed to mitigate this risk.

2.3 Training Approach and Quantization

The model was trained across hundreds of downstream domains, with joint reinforcement learning applied during post-training to strengthen tool calling, code modification, and long-horizon task completion. The weights are released in MXFP4 format, reducing storage and loading costs while offering more flexible deployment options across different hardware environments.

MXFP4 follows a block-scale floating-point quantization approach, compressing model size while preserving inference accuracy. For large-scale sparse models, weight quantization not only reduces memory usage but also improves weight loading efficiency. Developers should evaluate whether to use quantized deployment based on actual GPU memory, request throughput, and latency requirements, and whether additional KV cache space needs to be reserved.

3. Task Capabilities: Code, 3D Design, and Knowledge Work

3.1 Long-Horizon Code and Software Engineering Tasks

This model is primarily targeted at long-horizon coding agents, software engineering, and 3D design scenarios. Unlike short-form Q&A, these tasks typically require the model to:

• Read repository structure and locate key files;
• Propose modification plans based on requirements;
• Invoke editors or command-line tools to execute changes;
• Read test results and error messages;
• Perform multiple rounds of fixes and regression verification.

A typical workflow can be represented as:

Read task description
-> Browse directory structure
-> Search relevant code
-> Generate modification plan
-> Call tools to modify files
-> Run tests
-> Continue fixing based on failure information
-> Summarize changes and verification results

Throughout this process, the model must maintain task state across turns. Missing any step may produce results that appear correct but fail tests. Therefore, evaluation and deployment should focus on multi-step completion rates in real code repositories rather than single-shot code completion quality.

3.2 Terminal Operations and GUI Agents

Terminal automation tasks require the model to execute a series of operations in a command-line environment based on natural language instructions. At each step, the model must assess the current state, select an appropriate command, observe the output, and decide the next action. Such tasks demand strong state understanding, error recovery, and operational safety.

A common approach is to place terminal output, recent command history, and task goals into the context, allowing the model to generate the next command. Meanwhile, the system side can implement safeguards such as whitelisted directories, dangerous command interception, and timeout controls to prevent irreversible operations.

GUI agents follow a similar logic, but the input becomes screenshots, widget trees, or accessibility descriptions. The model must convert visual information into operational intent, then invoke actions such as clicking, typing, or scrolling. Native image-text input reduces reliance on external OCR or vision modules in such cases, but multi-step operational stability still requires external validation and fallback strategies.

3.3 3D Design Assistance

3D design tasks rely more heavily on spatial understanding than text or code tasks. The model can generate CAD scripts or design steps from parametric descriptions, then use external tools to complete geometric modeling and rendering. Evaluations typically include sketch understanding, dimension constraints, feature selection, and error correction.

In engineering practice, 3D design assistance is better suited as part of a human-in-the-loop workflow. For example, the model generates initial designs, explains constraint relationships, and suggests modification directions, while the user confirms and fine-tunes through visualization tools. Fully automated CAD generation remains limited by task description precision, tool interface stability, and the model’s spatial reasoning capabilities.

4. Inference Budget Tiers and Tool Calling

4.1 Three Inference Budget Tiers

The model offers low, medium, and high thinking budget tiers, suitable for tasks of varying complexity. The low tier reduces thinking steps and works well for simple Q&A, information extraction, and lightweight editing. The medium tier suits general tool calling that requires some planning. The high tier targets long-horizon code repair, complex terminal tasks, and difficult knowledge reasoning.

Higher thinking budgets are not always better. A high budget typically means longer time-to-first-token and higher compute costs. In real systems, tiered routing can be applied based on task complexity, acceptable user wait time, and cost budget. For example:

if task in ["simple Q&A", "format conversion", "short text summarization"]:
    use low budget
elif task in ["localized code modification", "document Q&A"]:
    use medium budget
else:
    use high budget

This tiered strategy achieves a better balance between experience and cost.

4.2 Multi-Turn Tool-Calling Protocol

In agent frameworks, each interaction round typically includes model messages, tool calls, tool responses, and further model reasoning. Compared with regular conversations, tool calling demands stricter preservation of message structure. Particularly for long-horizon tasks, you should not retain only the latest tool result. Instead, preserve the thinking content, tool requests, and tool responses across the full history.

A minimal message log can be described as:

[
  {"role": "user", "content": "Fix the failing test in the login module"},
  {"role": "assistant", "content": "First check the test output, then locate the relevant code."},
  {"role": "tool", "name": "run_test", "result": "3 failed, 2 passed"},
  {"role": "assistant", "content": "Failures are concentrated in token refresh logic. Check auth.py."},
  {"role": "tool", "name": "read_file", "result": "file content..."}
]

If the system truncates historical messages, it may break the correspondence between tool calls and responses, causing the model to generate invalid actions or repeat executions. Therefore, tool-calling history should be stored in a structured format, and when the context window approaches its limit, essential information should be preserved through summarization.

5. Evaluation Metrics and Technical Boundaries

Published evaluations cover terminal operations, code repair, deep software engineering, browser Q&A, 3D design, and office document processing. Different tasks use different agent frameworks and tool environments, with the highest thinking budget enabled by default.

In terms of metrics, the model performs strongly in browser Q&A and 3D assistance tasks, while there remains room for improvement in terminal automation and office document processing. This indicates that model capability is highly correlated with task format. Terminal tasks place stricter demands on operation accuracy, environment adaptation, and error recovery. Office document processing depends more on strong table understanding, multi-page information aggregation, and format restoration.

When selecting a model, developers should not rely on a single leaderboard score. Instead, they should build a small-scale evaluation set based on real business scenarios. The evaluation items should cover typical success paths and high-frequency failure paths, and record metrics such as end-to-end latency, tool-calling counts, cost, and manual correction rates.

6. Deployment, Inference, and Operational Constraints

After the weights are released, the model can run on mainstream inference engines and provide interfaces compatible with common conversational protocols. Deployment approaches typically include single-node multi-GPU inference, distributed inference, or managed inference services.

6.1 Resource Planning

For a model with 2.8T total parameters, the deployment barrier remains high even with quantization. Memory planning must account for weights, activations, KV cache, and communication overhead simultaneously. In long-context scenarios especially, KV cache requirements vary greatly across requests, which can lead to memory fragmentation and queueing delays.

Capacity testing is recommended before deployment, simulating different input lengths and concurrent requests to observe peak memory usage, throughput degradation, and queueing time. For production environments, set maximum input length, per-request tool-calling round limits, and timeout values to prevent individual long tasks from consuming excessive resources.

6.2 Context Management and External Storage

Although the model natively supports long context, unlimited history accumulation in multi-user concurrent scenarios leads to rapidly rising costs. Common optimization strategies include:

• Applying sliding-window compression to early history;
• Summarizing completed steps into brief states;
• Offloading large files and long logs to external retrieval systems;
• Writing checkpoints at key decision points to support rollback on failure.

These strategies do not replace long-context capability. Rather, they make long-context capability more efficient to use. The better the agent system is designed, the more the model can focus its attention on current decisions and subsequent planning.

6.3 Tool Safety and Permission Control

When a model has command execution, file read/write, network access, and code modification capabilities, safety must be designed in from the start. A unified tool gateway should be added to the agent execution layer. The gateway is responsible for permission validation, parameter whitelisting, sensitive operation approval, and audit logging.

For high-risk operations, manual confirmation can be introduced. For read-only and low-risk operations, automatic execution can be allowed with complete logging preserved. This balances task efficiency with safety and control.

7. Error Recovery and State Management in Multi-Step Tasks

The difficulty of long-horizon agents lies not only in whether they can generate correct actions, but also in whether they can recover when actions fail. In real environments, incorrect file paths, missing dependencies, insufficient permissions, unstable tests, and changing web page structures can all cause tool-calling failures.

A robust system should have the following capabilities:

1. Error classification: Distinguish among environment errors, parameter errors, strategy errors, and external unavailability errors.
2. Bounded retries: Retry recoverable errors with a maximum attempt limit.
3. Context rollback: Save key states to restore to a stable node after an error.
4. State summary updates: Replace early raw records with structured summaries when context becomes too long.
5. Clear termination conditions: Prevent the model from falling into an infinite repair loop.

An example flow:

Execute modification
-> Run tests
-> Tests fail
-> Determine error type
-> If dependency missing, install dependency and retry
-> If logic error, locate code and fix
-> Run tests again
-> If maximum retry count reached and still failing, output failure reason and changes made

This design significantly improves the completion rate and maintainability of long-horizon tasks.

8. Licensing and Compliance Considerations

This release uses a custom license rather than a common permissive open-source license. Developers should carefully read the terms before use, confirming whether commercial use, redistribution, modification, or offering external services based on the model is allowed.

For enterprise users, legal and compliance teams should evaluate the scope of model weight usage, especially when customer data, internal code, or commercial products are involved. Even with strong model capabilities, data compliance, model output safety, and intellectual property risks should not be overlooked.

9. Practical Recommendations

Overall, this type of model is well suited to serve as the reasoning and planning core for complex long-horizon tasks, but it requires external tools, state management, and safety controls to perform at its best. Developers are advised to adopt a progressive integration path:

1. First validate the model’s planning and tool-calling capabilities in read-only tasks;
2. Then introduce low-risk write operations with auditing and rollback configured;
3. Continuously optimize prompts, tool descriptions, and state management based on real task failure cases;
4. Control costs through tiered inference budgets and set resource limits for long tasks.

When model capabilities, tool ecosystems, and engineering governance form a closed loop, long-horizon agents can move from experimental demos to stable, production-ready systems.