Ultra-Large Sparse Models for Long-Horizon Agents: Architecture and Engineering Practices

1. From Dense to Ultra-Large Sparse Models: Why Architecture Defines Agent Limits

In scenarios involving long-horizon agents, software development, and complex knowledge work, models must do more than generate single answers. They need to maintain goal consistency over extended periods, invoke external tools, read long documents, and complete multi-step tasks. When scaling parameters in traditional dense models, computational costs and memory usage grow linearly or even super-linearly, leading to increased inference latency and deployment costs. Consequently, sparse Mixture-of-Experts (MoE) architectures have become a critical choice for balancing model capacity against computational overhead.

The ultra-large sparse model discussed in this article features a design with approximately 2.8T total parameters and roughly 104B activated parameters per inference. “Total parameters” refers to the entire scale of stored weights, while “activated parameters” indicates the actual parameter volume involved in the forward pass for a single token. The gap between the two reflects the core philosophy of sparse models: through a gating mechanism, only a subset of expert networks is activated per token, rather than the entire parameter set. Compared to a dense model with a similar total parameter count, this sparse architecture significantly reduces the inference cost per token while maintaining high representational capacity.

This architecture is particularly vital for long-horizon agents. Agent tasks typically require repeated model invocations, such as re-encoding historical messages, tool outputs, and current states after every tool call. If all parameters were activated at each step, the total computation would accumulate linearly with the number of task steps. By reducing the computation required per step, sparse models enable longer task trajectories to be completed within manageable costs.

2. Million-Token Context and Native Multimodal Inputs: Foundational Capabilities

One of the most common bottlenecks in long-horizon agents is the inability to retain sufficient information within the context. Many real-world tasks require the model to simultaneously process code repository structures, lengthy documents, terminal outputs, historical action logs, and user instructions. If the context window is too small, the model is forced to truncate early information, leading to goal drift or the redundant execution of completed sub-tasks.

Supporting a 1 million token context means the model can accommodate thousands of lines of code, complete technical documentation, or the results of multiple tool calls within a single inference session. Coupled with native text-image input capabilities, the model can also process architecture diagrams, UI screenshots, and design drafts, thereby broadening the range of tasks it can handle. For instance, in a code debugging task, the model can simultaneously review the exception stack trace, relevant source code, test logs, and terminal outputs, rather than guessing the issue based on a single error message.

It is important to emphasize that long context does not equate to long-term memory. In practical engineering, historical information still needs to be managed through message structuring, summarization, indexing, and retrieval. Multi-turn tool calling scenarios typically require retaining the complete history of reasoning and tool invocation messages, because a tool’s returned result may relate to an intent from many turns prior. If the system truncates or compresses this history mid-stream, the model may fail to correctly associate the current result with the original objective.

3. Sparse Attention and Joint RL: From Local Perception to Global Decision-Making

As context windows expand, directly computing full attention introduces quadratic complexity growth. Therefore, long-context models generally require more efficient attention structures. The model discussed here has adjusted its attention mechanism from prior versions: replacing gated multi-head latent attention with gated dynamic sparse attention in certain layers, combined with another inter-layer attention mechanism.

The key behavior of dynamic sparse attention is selecting a fixed number of historical tokens to participate in the attention computation at each step—for example, selecting 2,048 historical tokens at a time. Compared to full attention, this approach significantly reduces the computational and memory overhead for long sequences. More importantly, the selection process is not a simple positional truncation but is dynamically determined based on learned relevance. This allows the model to prioritize historical fragments most relevant to the current decision when processing a token, without traversing the entire history.

At the training level, joint reinforcement learning (RL) covers hundreds of downstream domains. Unlike fine-tuning on a narrow set of tasks, this strategy aims to equip the model with stable reasoning and decision-making capabilities across diverse tasks. Code generation, tool calling, terminal operations, and document Q&A feature different feedback signals and state spaces. Through large-scale joint training, the model learns more generalizable strategies for planning, retrieval, error correction, and verification, rather than overfitting to a single evaluation set.

To control resource consumption during large-scale training and deployment, weights are stored using MXFP4 quantization. While quantization sacrifices some precision, it significantly compresses the model footprint and bandwidth requirements. For ultra-large sparse models, low-bit quantization is practically a prerequisite for actual deployment. When evaluating quantized models, users should focus on the actual deviation on target tasks before and after quantization, rather than relying solely on ideal precision scores.

4. Tiered Thinking Budgets: Balancing Quality and Latency with Controllable Costs

Not all tasks require maximum intensity reasoning. Simple queries might need only a direct answer, while complex multi-step tasks demand repeated deduction, intermediate result verification, and alternative comparison. To address this, the model offers three thinking budget tiers: Low, High, and Max. The thinking budget acts as a resource constraint on the model’s reasoning depth: the Low tier is suited for lightweight queries, while High or Max tiers are intended for tasks requiring long chains of reasoning.

This tiered design has two direct implications for engineering practices. First, developers can set different reasoning intensities for different requests on the same model, allowing fine-grained control between cost and performance. For example, user-facing simple Q&A can use the Low thinking budget to ensure rapid response times, while batch code generation tasks running overnight can use the Max tier to maximize accuracy. Second, evaluation results typically correlate with the thinking budget. Official benchmark scores default to the Max thinking tier, meaning actual deployment performance may be lower if the Low tier is used. Therefore, capacity planning and technology selection should clearly define the target tier rather than directly quoting Max tier results.

5. Evaluation Systems for Coding, Agents, and Knowledge Work

Single-turn Q&A accuracy is insufficient to fully measure agentic capabilities. For long-horizon tasks, evaluation must cover multi-tool collaboration, environmental state changes, long-term planning, and fault tolerance. Common dimensions include terminal operations, repository-level bug fixing, deep software engineering, browsing-based information retrieval, tool-assisted 3D design, and office document Q&A.

Terminal operation evaluations focus on whether the model can generate the next command based on the current terminal state and complete operations such as installing dependencies, modifying files, running tests, and locating errors across multi-turn interactions. Code repair evaluations typically provide real or near-real-world software engineering tasks, requiring the model to understand repository state, modify multiple files, and pass tests. Browsing-based retrieval assesses the model’s ability to search, filter, and compare information across multiple web sources, making it particularly suitable for evaluating information integration within long contexts.

Tool-assisted design tasks simultaneously test command invocation, geometric reasoning, and structured output capabilities. Office document Q&A closely mirrors enterprise knowledge work scenarios, requiring the processing of tables, long documents, and cross-paragraph logical relationships. While scores in these tasks reflect the model’s capabilities, actual application performance remains influenced by prompt design, tool interface quality, and context management strategies. Users are advised to build small-scale regression sets within their actual business environments to evaluate model stability on their specific tasks.

6. Inference Deployment and API Compatibility: From Weights to Usable Services

Once weights are released, the model can be loaded and run in environments compatible with mainstream inference frameworks. A common industry practice is to use inference engines that support sparse models and long contexts, exposing services via OpenAI or Anthropic compatible APIs. This compatibility lowers the migration cost for existing upper-layer frameworks, allowing developers to reuse current agent frameworks or request encapsulation logic.

Several key points require attention during deployment. First, sparse expert models require the inference engine to correctly implement expert routing and quantized weight loading; otherwise, computational efficiency may drop or the model may fail to load. Second, multi-turn tool calls must retain the complete history of reasoning and tool invocation messages. If the middleware excessively trims messages or only retains tool output text while discarding the reasoning process, it may break the model’s decision-making chain. Third, concurrency, batching, and dynamic batching strategies for the inference service must be re-tuned based on sparse model characteristics. Since different requests may activate different combinations of experts, estimating resource consumption simply by sequence length or token count will deviate from reality.

At the service level, model deployment on general clouds typically involves foundational capabilities like container platforms, object storage, message queues, logging, and monitoring/alerting. For example, weight files can be stored in object storage, inference services launched via container platforms, and request logs and performance metrics integrated into a monitoring system. For enterprise users, this general architecture is well-suited for partitioning independent services based on task types, such as separating low-latency services from high-throughput offline task services.

7. Licensing and Production Adaptation: Open Weights Do Not Mean Unrestricted Use

The weights are released under a specialized license rather than a standard permissive open-source license. This means users must carefully review the terms of use to clarify the boundaries of responsibility regarding production deployment, commercial use, redistribution, and model outputs. Specialized licenses often impose additional restrictions on users, use cases, or distribution methods. Particularly when the model is used for external services, embedded in products, or provided via API, the constraints may differ significantly from standard open-source licenses.

From an engineering governance perspective, a license compliance assessment should be completed before integrating external weights. This includes verifying whether commercial deployment is allowed, whether modifications and redistribution are permitted, whether attribution is required, if there are regional or industry restrictions, and whether derivative content must be open-sourced. If the team cannot confirm these terms, they should seek legal counsel or avoid direct usage.

8. Building Practical Long-Horizon Agents: From Model Capabilities to System Capabilities

A long-horizon agent system can typically be decomposed into the following layers: the model layer handles understanding and decision-making; the memory layer preserves historical states, retrieved information, and structured knowledge; the tool layer defines available actions and their input/output contracts; the execution layer operates in code, document, or browser environments; and the evaluation layer continuously monitors task completion quality and resource consumption. The model is just one component, albeit the most core one.

A common long-horizon task flow looks like this:

User instruction -> Model planning
                  -> Model invokes tool
                  -> Environment returns result
                  -> Model updates state and plans next step
                  -> ... repeat for several rounds ...
                  -> Final output or result submission

In this multi-turn loop, situations like timeouts, command errors, missing search results, or abnormal tool return formats frequently occur. The model must adjust its strategy based on error information rather than mechanically executing the original plan. This capability stems from joint RL learning across both successful and failed trajectories, making it more stable than relying solely on prompt-based examples.

For developers, deploying long-horizon agents requires looking beyond public benchmark scores and establishing process-oriented metrics tailored to their business, such as task completion rate, average tool calls, rework count, context utilization, and cost per task. Integrating these metrics into a monitoring system helps promptly identify issues like prompt degradation, tool interface changes, or improper model tier selection.

Looking ahead, as context lengths increase and tool ecosystems mature, long-horizon agents will become more prevalent in software development, financial research, office automation, and engineering design. However, model capability is only the starting point; system effectiveness ultimately depends on engineering capabilities like task decomposition, tool design, state management, and cost control. Understanding model architecture, inference mechanisms, and deployment constraints will empower teams to make more robust decisions during technology selection and system design.