Trillion-Scale Sparse LLMs Open Weights: Evaluating Long-Horizon Agent Execution Capabilities and Technical Deep Dive

1. Background: Open Weights Are Not Equal to Direct Deployment, But the Significance Remains Substantial

Recently, a noteworthy phenomenon has emerged in the AI industry: sparse large language models with parameter scales reaching the trillion level have begun to open their weights. Unlike the commonly seen dense models in the tens or hundreds of billions, these models exhibit distinct differences in architectural design, training methodologies, inference costs, and deployment thresholds. Open weights mean that researchers and engineering teams can access model parameter files for secondary training, distillation, quantization, pruning, or interpretability research. However, this does not imply that average developers can easily run them locally.

From a technical perspective, this shift brings impacts on at least three levels.

First, model capability evaluation is migrating from static Q&A to long-horizon agent tasks. Traditional evaluations typically focus on single-turn Q&A, code completion, or standardized test scores. In contrast, real-world agent scenarios require the model to continuously maintain state across multi-turn interactions, execute tool calls, adjust strategies based on environmental feedback, and recover from failures. This scenario demands higher requirements for model reliability, instruction-following capability, and state consistency.

Second, the gap between model scale and engineering feasibility has become more pronounced. For a model with a total parameter count reaching the trillion level and activating hundreds of billions of parameters per inference, the weight files are measured in terabytes (TB) even with low-precision quantization. Consumer-grade hardware can barely load them, let alone provide effective inference services.

Third, open weights are driving evaluation methods from subjective voting toward causal inference based on behavioral signals. Past common pairwise blind tests relied on users manually voting between two answers. Conversely, evaluations for agent tasks now extract success, failure, and correction signals from real interaction logs, estimating the model’s net improvement rate relative to the average level through randomized inference. This approach more closely reflects real-world performance in production environments.

Therefore, this article focuses on a typical technical theme: how to understand the capability performance of trillion-scale sparse LLMs in long-horizon agent tasks, as well as the architectural choices, training strategies, and deployment constraints supporting these capabilities.

2. Why Agent Task Evaluation is Harder Than Static Q&A

In static Q&A scenarios, a model typically only needs to generate a reasonable text output based on the input. Evaluation metrics are relatively straightforward, such as whether the answer is correct, the content is relevant, and the language is fluent. Even if there are minor flaws in the output, it does not necessarily lead to overall task failure.

Long-horizon agent tasks are entirely different. An agent needs to execute a series of continuous actions, such as reading files, running commands, checking outputs, deciding on the next step based on results, and recovering when exceptions occur. Failure in any intermediate step can lead to subsequent state errors, ultimately causing the entire task to fail.

Therefore, agent evaluation must focus on the following dimensions.

2.1 Tool Calling Reliability

Tool calling is the foundation of agent execution capabilities. Taking terminal operations as an example, the model needs to generate Shell commands and ensure they execute correctly in the target environment. Command syntax errors, path errors, insufficient permissions, missing dependencies, or parameter mismatches will all lead to execution failures. Traditional Q&A evaluations rarely touch upon this level of reliability, whereas in agent tasks, the tool call failure rate directly determines the task success rate.

2.2 Feedback Response and Strategy Adjustment

Real tasks are rarely completed in a single pass. Users may propose corrections during the interaction, such as requesting a different implementation method, adjusting the output format, ignoring a specific file, or changing the execution sequence. The agent must quickly adjust its internal strategy based on feedback rather than repeating previous behaviors. This capability, known as correctability, is a crucial metric for evaluating whether a model is suitable for long-horizon tasks.

2.3 User Satisfaction and Task Confirmation

Whether a task is completed ultimately requires user confirmation. However, users might not explicitly say “task completed”; instead, they may indirectly express satisfaction through praise, silence, proposing new requirements, or directly ending the session. If the model frequently asks users to repeat themselves, generates low-quality intermediate artifacts, or fails to correctly understand confirmation signals, user satisfaction will drop significantly.

2.4 Session State Maintenance

In long-horizon tasks, the model needs to maintain accurate memory of task goals, historical actions, and current environmental states within a long context. Once context information is forgotten or incorrectly associated, the model might repeat completed steps or continue operating on an erroneous target. This places extremely high demands on the model’s attention mechanism and context modeling capabilities.

Based on the dimensions above, agent evaluation is gradually shifting from manual voting to signal-driven approaches. The evaluation system extracts explicit behavioral signals—such as task confirmations, praises, complaints, correction commands, and command execution errors—from real user sessions. It then compares the net improvement rates of different models through randomized inference. This design reduces subjective bias, bringing results closer to the actual user experience.

3. The Role of the Mixture of Experts Architecture in Long-Context Tasks

Trillion-scale parameter models typically adopt the Mixture of Experts (MoE) architecture. Unlike traditional dense models, MoE models split the feed-forward network into multiple expert subnetworks. Each input token only activates a small fraction of these experts, controlling the computational overhead per inference while maintaining an extremely high total parameter count.

In the model discussed here, the total parameter count reaches 2.8 trillion, while a single inference activates only about 104 billion parameters. This means that in the computational path of any given token, experts are highly sparsely activated. This design brings several direct benefits:

1. Expanded Model Capacity: The total parameter count is far greater than the per-inference activation count. The model can store more knowledge, patterns, and task skills without linearly increasing the floating-point operations per inference.
2. Enhanced Multi-Domain Adaptability: Different experts can implicitly divide labor, forming distinct activation patterns for different domains, languages, task types, or reasoning phases.
3. Reduced Per-Token Latency: On specialized acceleration hardware, sparse activation can reduce actual computational latency through expert parallelism and capacity scheduling, provided that the engineering implementation can effectively manage expert routing and load balancing.

However, the MoE architecture also presents distinct challenges. The routing mechanism must select the appropriate combination of experts for each token. If routing is unbalanced, certain experts will be frequently invoked while others remain idle, leading to training instability or degraded inference efficiency. Furthermore, as the number of experts increases, the communication overhead in distributed training grows, making it difficult to fully leverage hardware computing power. Therefore, the training framework must combine expert parallelism, tensor parallelism, pipeline parallelism, and targeted load-balancing loss functions.

4. Gated Sparse Attention: Reducing Long-Context Modeling Costs

In long-context tasks, the computational complexity of the attention layer typically grows quadratically with sequence length. To support a context window of approximately 1 million tokens, the model must structurally optimize the attention mechanism. A common approach is introducing gated sparse attention layers.

The core idea of this attention mechanism is that not all tokens need to attend to all other tokens. The model can selectively focus on key segments through gating networks or sparse patterns, thereby reducing the number of token pairs for which similarity must be computed. Compared to full attention, gated sparse attention significantly reduces computational volume and KV cache usage while preserving long-distance information propagation capabilities.

However, sparse attention is not a free lunch. The gating mechanism must accurately judge which token associations are important and which can be ignored. If poorly selected, the model may lose critical context during long tasks. For example, if an agent needs to recall a variable name or path set in step 3 during step 200, and the sparse attention fails to retain that information, the task will fail.

Therefore, the training and fine-tuning of such models typically require the joint optimization of multiple objectives, including downstream task loss, routing load-balancing loss, and long-context memory retention metrics. Especially during the reinforcement learning post-training phase, the training environment constructs large amounts of long-trajectory interaction data, allowing the model to repeatedly practice state maintenance and sparse attention selection strategies in real long-horizon agent tasks.

5. Joint RL Post-Training: From Static Answers to Task Execution Capabilities

If a model relies solely on pre-training and standard supervised fine-tuning, it can learn to generate seemingly reasonable answers but may not perform robustly in multi-step reasoning, tool calling, and error recovery. To address this, an increasing number of LLMs are adopting large-scale joint reinforcement learning (RL) post-training.

Joint RL no longer optimizes only a single task reward; it covers hundreds of downstream domains, including code generation, command execution, file manipulation, information retrieval, and multi-step planning. Training signals are derived from environmental feedback, such as whether a program passes tests, whether a command executes successfully, whether the user explicitly confirms task completion, or whether a correction is needed.

This training approach differs from traditional RL in three main aspects.

First, reward signals are sparser but semantically stronger. An agent task may require dozens of steps, and a clear positive reward is only obtained upon final success. The model must learn to attribute which steps positively impacted the final outcome from long trajectories.

Second, negative feedback handling is more complex. User corrections and command execution errors are often weak signals; they do not necessarily represent total failure but indicate that the current strategy needs adjustment. If these signals are directly mapped to negative rewards, the model may become overly conservative and unwilling to try exploratory solutions. Therefore, training needs to distinguish between hard failures and soft corrections, designing reward structures with varying intensities and discounts.

Third, the domain coverage is broad. Single-domain RL can easily lead to strategy overfitting. For instance, a model optimized only on code generation tasks might perform poorly in terminal operations. Joint training promotes the learning of transferable task execution capabilities—rather than memorizing domain-specific templates—by simultaneously sampling data across multiple domains.

After training, the model’s performance in long-horizon agent tasks is typically measured across three dimensions: command execution reliability, user satisfaction changes, and correctability improvements. These metrics reflect behavioral improvements rather than simple test scores.

6. Specific Performance Improvements from the Data

In a large-scale agent task evaluation based on statistics from over 17,500 real long-horizon sessions, this model achieved a net improvement rate of approximately +5.1% relative to the average level. For comparison, the previous generation base model had a net improvement rate of only +0.9%. This means that after controlling for session difficulty, user distribution, and task type, the positive experience improvement brought by the new model is significantly higher than its predecessor.

Breaking down the metrics further reveals the sources of improvement more clearly.

6.1 Command Execution Reliability

This measures whether Shell commands can be executed correctly. The previous model scored around -4.5% on this metric, meaning its execution reliability was below average; the new model reached approximately +7.9%, placing it among the top open-weight models. This improvement indicates that joint RL post-training has had a significant positive impact on tool calling capabilities. The model no longer just “looks like it can write commands”; the proportion of commands that “actually run successfully” has visibly increased.

6.2 User Satisfaction

This is measured by the difference between praise and complaint signals. The previous generation was around +1.3%, while the new model is about +7.7%. Users complained less about execution results and more proactively confirmed completion or provided positive feedback. The satisfaction improvement is highly correlated with command execution reliability, indicating that user evaluation of agents is primarily based on whether the task was genuinely completed.

6.3 Correctability

This measures the model’s ability to adjust its execution direction based on user feedback. The new model scored approximately +6.5%, also demonstrating excellent performance. This shows that the model can receive new instructions mid-task and replan its action path rather than continuing to execute outdated goals.

6.4 Task Confirmation Success Rate

The proportion of users explicitly confirming task completion was about +3.9%. Although on an upward trend, the magnitude is smaller than the other metrics. This may suggest that user confirmation behavior is influenced by multiple factors, such as task complexity, usage habits, and varying acceptance criteria. The model could further improve its ability to proactively confirm boundary conditions in ambiguous scenarios, summarize deliverables, and verify completion status.

Overall, the performance improvement does not stem from a single dimension but from the combined enhancement of tool reliability, user communication, and strategy adjustment capabilities.

7. Engineering Implementation for Inference Services

Building a usable inference service for a trillion-scale sparse model at the engineering level requires holistic design from model loading and memory management to distributed inference and scheduling strategies. Below is a reference implementation approach expressed using common development languages and pseudocode.

# Simplified illustration of model loading and inference service calls
class SparseModelServer:
    def __init__(self, checkpoint_dir, num_experts, active_experts):
        self.num_experts = num_experts
        self.active_experts = active_experts
        self.weights = load_distributed_checkpoint(checkpoint_dir)
    def route(self, hidden_states):
        # Calculate expert routing scores for each token
        router_logits = self.weights.router(hidden_states)
        topk_indices = select_top_k(router_logits, self.active_experts)
        return topk_indices
    def forward_layer(self, hidden_states):
        expert_ids = self.route(hidden_states)
        outputs = []
        for expert_id in expert_ids:
            expert_output = self.weights.experts[expert_id](hidden_states)
            outputs.append(expert_output)
        return merge_expert_outputs(outputs)
    def serve(self, prompt_tokens, max_steps, environment):
        context = LongContextManager()
        for step in range(max_steps):
            context.add_user_state(environment.observe())
            hidden = self.embed(context.to_tokens())
            hidden = self.forward_layer(hidden)
            action = self.decode_action(hidden)
            result = environment.execute(action)
            context.add_tool_result(result)
            if context.should_stop():
                break
        return context.summary()

The pseudocode above merely illustrates the typical processing flow of an MoE model in an inference service. Actual production deployment is far more complex, requiring the handling of communication in expert parallelism, dynamic batching scheduling, KV cache sharding, tiered weight loading, and fault recovery.

Cloud deployments typically require clusters composed of multiple high-performance acceleration instances, utilizing high-speed interconnect networks to handle data exchange between experts. Even with INT4 quantization, the ~1.4TB of weights must be distributed across multiple accelerator cards, and loading latency must be hidden through pipeline scheduling.

8. Position in Terminal Agent Benchmarks

In terminal agent benchmarks, this model scored approximately 65.7%, placing it in the same tier as other advanced models scoring 58.2% and 66.4%. Such tests typically require the model to complete tasks like file searching, text processing, script writing, configuration modification, and error troubleshooting in a simulated terminal environment, comprehensively evaluating command generation capability, multi-step planning, and environmental interaction stability.

It is important to emphasize that benchmark scores are merely capability snapshots and do not equate to production environment performance. Real user tasks often contain more ambiguity, noise, and incomplete information, placing higher demands on the model’s long-range memory, boundary judgment, and user communication capabilities. Therefore, when interpreting these evaluation results, one should combine behavioral metrics such as tool calling reliability, user satisfaction, and correctability for a comprehensive judgment.

9. Actionable Insights for Developers and Enterprise Users

For teams planning to introduce long-horizon agent capabilities into their business, the following points are worth noting.

9.1 Prioritize Validating the Tool Calling Chain

Before large-scale model selection, first validate the definitions, permissions, and execution boundaries of tool calls in your business environment. For example, whether the model is allowed to execute commands, access the file system, or call cloud resource APIs. The clearer the tool definitions, the easier it is for the model to generate reliable actions.

9.2 Build an Internal Evaluation Set Based on Behavioral Signals

Do not rely solely on static Q&A scores. It is recommended to collect operation trajectories from real user tasks, annotating success, failure, correction, and confirmation signals to form an internal agent evaluation set. Continuously observe model iteration effects through metrics like net improvement rate, command success rate, and task confirmation rate.

9.3 Consider Cloud Inference Over Local Deployment

Trillion-scale sparse models have extreme requirements for memory, bandwidth, and scheduling capabilities. For most enterprises, using a cloud inference service is a more realistic solution. Cloud-based agent workflows can be constructed by integrating object storage, container platforms, and logging services with model inference APIs.

9.4 Emphasize Correctability and User Experience

Even if command execution reliability is high, correctability remains key to determining the success rate of long-horizon tasks. The model must be able to understand user feedback and adjust its strategy in subsequent executions. It is recommended to design explicit feedback channels at the application layer, allowing the model to read user modification suggestions and replan.

9.5 Estimate Costs and Latency Accurately

Although the per-token inference cost of a sparse model is lower than that of a dense model with the same total parameter count, long-horizon tasks execute a large number of intermediate steps. Estimations should be based on task-level costs rather than single-request costs, and mechanisms such as caching, breakpoint recovery, and redundant call control should be integrated into the workflow.

10. Conclusion

The release of open-weight, trillion-scale sparse LLMs provides research and engineering teams with opportunities for deeper internal model exploration. It also drives the evaluation of agent capabilities from static Q&A toward long-horizon task signal analysis. The Mixture of Experts architecture vastly increases model capacity while controlling per-token computation; gated sparse attention reduces the overhead of ultra-long context modeling; and joint RL post-training significantly enhances tool calling reliability, user satisfaction, and correctability.

However, open weights do not mean out-of-the-box usability. Ultra-large parameter models still present significant engineering thresholds in storage, loading, distributed inference, and cost control. For developers, it is more important to understand the behavioral metrics of these models in real agent tasks and to establish evaluation and integration plans centered on task success rates based on their own scenarios. Only by combining model capabilities, tool chains, and user feedback loops can the true value of long-horizon agent technology be realized.