Open-Source LLMs Move Toward Long-Horizon Agents: Ultra-Large Sparse Architectures and Multi-Step Task Practices

Background: Why Long-Horizon Agents Require New Model Architectures

Large language models have demonstrated strong capabilities in single-turn or short-context tasks such as chat, summarization, and Q&A. However, real-world business scenarios rarely rely on a single inference pass. Developers are increasingly concerned with whether a model can sustain execution over extended time windows, for example:

• Reading a code repository, locating defects, modifying multiple files, and running tests;
• Generating 3D models or engineering drawings from natural language requirements;
• Switching repeatedly between office documents, spreadsheets, and external retrieval results to complete information synthesis and decision drafts;
• Calling external tools to fetch data, then adjusting subsequent actions based on returned results.

These tasks share common characteristics: long contexts, many steps, intermediate results that must be referenced continuously, and the model must know when to call tools, when to stop, and how to integrate tool results into the next reasoning step. As a result, context capacity, attention mechanism efficiency, reasoning budget control, and deployment patterns for open-weight models directly impact agent application effectiveness.

Recently, the open-source community has introduced a class of ultra-large sparse models: total parameter counts reach the trillion scale, but only a fraction of parameters are activated during each inference pass. This architecture reduces per-inference computation while preserving strong expressive power, making long-context agent tasks more feasible from an engineering perspective.

Model Architecture: Sparse Experts and Long-Context Attention

The newly open-weighted model adopts a Mixture-of-Experts (MoE) architecture with a total parameter count of 2.8T and approximately 104B activated parameters per inference. Compared with dense models of similar scale, the sparse expert structure routes each token to a subset of expert networks, significantly reducing computational cost.

On the context processing side, the model supports 1 million input tokens and handles both text and image inputs. What deserves closer attention is the change in attention mechanisms. Previous versions used gated Multi-head Latent Attention (MLA) in some layers, while the new version replaces 24 layers with gated DSA (Dense-Sparse Attention), working in coordination with 69 layers of Delta Attention.

The core idea of DSA is not to compute attention between the current token and all historical tokens, but to select a subset of historical tokens for computation. Specifically, 2048 tokens are selected from history for each attention computation. The value of this approach is threefold:

1. During long-sequence inference, attention computation does not scale linearly with context length;
2. The model tends to focus on historical information highly relevant to the current step in long-horizon tasks;
3. At the engineering level, caching strategies can be combined to reduce redundant computation.

From an agent perspective, this attention mechanism is better suited to scenarios characterized by long tasks, multiple steps, and high local relevance. For example, in code repair tasks, the current editing position typically relates to nearby functions, classes, and call relationships, rather than requiring the model to re-interpret the entire repository history in every round.

Reinforcement Learning Post-Training: From Single-Turn Capability to Multi-Step Decisions

The model underwent large-scale joint reinforcement learning post-training on top of the base version, covering hundreds of downstream domains. Compared with supervised fine-tuning, reinforcement learning places more emphasis on the model’s performance in the “select action, receive feedback, adjust strategy” loop.

In agent tasks, this training approach yields several behavioral changes:

• The model no longer pursues only correct single-step answers, but prioritizes whether the full multi-step action chain ultimately completes the task;
• When a tool returns errors or insufficient information, the model is more likely to retry queries, adjust parameters, or change paths;
• Faced with long documents, large codebases, or multi-turn dialogues, the model is more likely to maintain goal consistency.

It is important to note that reinforcement learning post-training does not guarantee stable performance across all environments. In actual deployment, developers still need to design clear tool interfaces, termination conditions, and exception handling mechanisms.

Reasoning Budget Tiers: Low, High, Max

Long-horizon agent tasks are computationally expensive, so the model provides three reasoning budget tiers: low, high, and max. Developers can select different tiers based on task complexity:

Tier	Suitable Scenarios	Expected Characteristics
low	Simple Q&A, short text processing, rapid prototyping	Faster responses, lower compute cost
high	Moderately complex coding tasks, document analysis	Balanced between speed and quality
max	Deep reasoning, long-chain tool calls, complex debugging	More thorough reasoning, highest latency and cost

When building agent applications, it is not advisable to use the max tier for all requests. A better approach is routing based on task type. For example:

def select_reasoning_budget(task):
    if task.type in ("simple_extraction", "basic_summary"):
        return "low"
    if task.type in ("code_review", "document_qa", "data_analysis"):
        return "high"
    if task.type in ("multi_step_agent", "complex_debug", "long_horizon_planning"):
        return "max"
    return "high"

This tiered design allows a single model to serve business scenarios with heterogeneous latency and cost requirements.

Typical Application Scenarios and Technical Considerations

Code Development and Long-Horizon Coding Agents

In terminal operations, code repair, and software engineering tasks, the model must understand command-line environments, file system states, and code context. A typical flow is as follows:

1. Read the task description and repository structure;
2. Search for relevant files and functions;
3. Generate a modification plan;
4. Invoke terminal or code execution tools for verification;
5. Decide whether to continue modifying based on test results.

These tasks impose high demands on context management. The model must not only “remember” which files have been modified, but also understand the causal relationship between test failures and code changes. Using a sparse attention model that supports long contexts preserves more useful historical information across multiple tool invocations.

Knowledge Work and Tool Calling

In web search, financial research, and office document processing scenarios, the model typically needs to collaborate with external tools. For example:

• Invoking retrieval tools to fetch the latest information;
• Reading spreadsheets, PDFs, or presentation decks;
• Summarizing, comparing, and judging based on tool-returned content;
• Generating structured reports or decision recommendations.

These tasks do not always require the longest context, but they demand strong capabilities in parsing tool call formats and return results. Developers should clearly specify tool descriptions, parameter constraints, and return formats in system prompts, while retaining complete tool-call message history to prevent the model from losing critical context in later steps.

3D Creation and CAD Generation

3D creation is an often overlooked but highly valuable long-horizon agent scenario. The model needs to generate geometric structures from natural language descriptions, modify parametric designs, or combine tools to complete modeling operations.

Unlike text or code tasks, 3D tasks typically involve spatial relationships, geometric constraints, and visual feedback. The model may first generate a structured description, then invoke modeling tools to produce results, and iterate based on rendering or validation feedback. In this context, multimodal input capabilities and long-context support can significantly improve interaction efficiency.

Deployment and Inference Optimization

The open weights use MXFP4 quantization format, run on mainstream inference frameworks, and are compatible with OpenAI-style or Anthropic-style API protocols. Key deployment considerations include:

Weights and Licensing

The model uses a dedicated license and is not under a permissive open-source license. Enterprises must carefully review the license terms before commercial use, checking whether external service provision is permitted, whether derivative models must be open-sourced, and whether there are geographical or user scale restrictions.

Multi-GPU Deployment and Memory Planning

Even after quantization, a 2.8T total parameter model requires substantial GPU memory and system RAM. Common practices include:

• Using multi-GPU tensor parallelism or pipeline parallelism;
• Offloading some layers to CPU memory;
• Using PagedAttention-style mechanisms to manage long-sequence caches.

Before actual deployment, capacity planning should be performed based on concurrency, maximum context length, and inference latency requirements.

Long Context and Message History Management

In agent tasks, multi-turn tool calls continuously append messages. Without a reasonable compression and cleanup strategy, context can expand rapidly. Recommended practices include:

• Summarizing or selectively retaining historical tool outputs;
• Moving unnecessary large outputs out of context;
• Preserving full reasoning traces at critical decision points, and keeping only conclusions elsewhere;
• Using system messages to fix task goals and tool specifications.

Capability Boundaries and Engineering Recommendations

According to official benchmarks, the model performs well in terminal tasks, software engineering, deep code questions, browsing reasoning, CAD generation, and office Q&A. However, no model can guarantee consistent benchmark-level performance across all real-world scenarios. Reasons include:

• Evaluation environments differ from real business environments;
• Tool interface quality, prompt design, and error handling strategies affect final outcomes;
• Errors in long-horizon tasks may accumulate as steps increase;
• Performance under different reasoning tiers may vary noticeably.

Therefore, developers are advised to build small-scale, measurable internal evaluation sets before introducing such models into production. Key validation points include:

• Task success rate;
• Average number of completion steps;
• Invalid tool call rate;
• Maximum stable context length;
• Cost and latency under different reasoning tiers.

Conclusion

The release of open weights for ultra-large sparse models provides new foundational capabilities for long-horizon agent applications. The core value lies not in pursuing conversational quality alone, but in enabling models to maintain stable performance over longer contexts, more tool invocations, and more complex multi-step tasks.

For developers and enterprise users, the focus should be on three levels:

1. Understanding the model’s architectural characteristics and inference costs, and selecting appropriate reasoning tiers;
2. Designing clear tool interfaces and message history management strategies;
3. Building continuous evaluation and iteration mechanisms in real business scenarios.

Only when model capability, engineering frameworks, and business scenarios are well aligned can “long-horizon agents” move from proof of concept to stable production systems.