1T-Parameter Open-Source Agentic Model: Architecture, Training System, Data Synthesis, and Inference Cost
This article examines a 1T-parameter open-source Agentic model across four dimensions: model architecture, training and data system, inference cost, and evaluation approach. It explains how the model extends a proven MoE design from 671B to 1T parameters, reduces activated parameters and MLA head dimension for long-sequence efficiency, uses a quality-focused 15.5T-token training corpus and a stabilized optimizer, and builds Agent capabilities through a scalable interactive data synthesis system. It also analyzes why the sparse architecture’s cost advantage depends on deployment scale and outlines practical lessons for evaluating and adopting large sparse Agentic models.
1. Background: Agentic Models Become the New Competitive Focus
Over the past two years, the evaluation focus for large language models has shifted noticeably: from early knowledge QA and text generation to complex reasoning tasks such as code and math, and now to Agent capabilities centered on whether a model can autonomously complete multi-step tasks.
An Agentic model is a large model optimized specifically for Agent scenarios. Its core difference from a general chat model is that it must not only understand instructions and generate answers, but also plan tasks proactively, select tools, initiate API calls, interpret returned results, and correct its behavior path across multiple interaction turns. The industry generally agrees that closed-source frontier models still perform strongest in Agent scenarios, and the open-source community’s effort to catch up in this direction is the main thread of this technical discussion.
The released model has 1 trillion (1T) total parameters, making it one of the largest open-source models by parameter scale. Notably, its evaluation results do not rely on an explicit chain-of-thought reasoning paradigm. As a non-thinking model, it reaches frontier-level performance among models of the same type on tasks involving frontier knowledge, math, and code. At the same time, it leads peer open-source models significantly on Agent-specific metrics such as tool calling and Agent decision planning, approaching the strongest closed-source reference models.
This article breaks down the model across four dimensions: model architecture, training and data system, inference cost, and evaluation methodology.
2. Model Architecture: Precise Tuning on a Mature MoE Design
2.1 Overall Scale: From 671B to 1T
The model is based on the mainstream sparse Mixture-of-Experts (MoE) architecture. The previous-generation open-source model with the same architecture had 671B parameters, while the new model raises total parameters to 1T. The main driver of parameter growth is expansion of the number of experts: from 256 to 384. The dense parts and attention-related structures largely retain the original design.
This design reflects an engineering philosophy of “incremental evolution”: perform controlled scaling on a validated architecture and avoid introducing unproven radical changes, thereby reducing training risk and engineering cost.
2.2 Attention Structure: MLA Dimension Reduction for Better Long-Sequence Cost Efficiency
The model continues to use Multi-head Latent Attention (MLA), an attention variant already proven effective for compressing key-value cache during attention. The key adjustment is reducing the MLA head dimension from 128 to 64.
The direct benefit is significantly lower computation and memory overhead in long-sequence scenarios while keeping attention expressiveness largely intact. This is especially important for Agent scenarios: when an Agent executes tasks, it often needs to carry lengthy tool documentation, historical call records, and context state, so long-sequence capability is essential. Trading half the head dimension for better long-sequence cost efficiency is one of the most pragmatic architectural choices in the model.
2.3 Layer Structure: Further Sparsification
In the mixed MoE and dense strategy, the previous-generation model used dense design in its first few layers, while the new model keeps only the first layer dense and makes all remaining layers sparse. This means almost all layer computation follows the route of “activating a small number of experts per token,” further increasing overall sparsity.
Correspondingly, the number of activated parameters per token is 32B, compared with about 37B in the previous-generation model. While total parameters grew by about 50%, activated parameters fell by about 13%. This is the combined result of the sparsification design and MLA dimension reduction: trading larger parameter capacity for similar or even lower inference cost.
3. Training System: Fast, Incremental Data Expansion and a New Optimizer
3.1 Training Data: Restrained Growth from Quantity to Quality
Total training data increased slightly from 14.8T tokens to 15.5T tokens. Notably, the team did not blindly scale up data volume; instead, it focused on refining data quality, especially interactive data related to Agent tasks (see Section 4). This aligns with current industry consensus: high-quality 15T data with a reasonable task distribution can support capability ceilings far better than 30T of coarse corpus.
3.2 Optimizer: An Improved Momentum Orthogonalization Optimizer with Clipping Constraints
Training uses an improved momentum orthogonalization optimizer, a variant commonly known in the industry as a type of optimizer with gradient clipping constraints. The core idea is to improve the “directional quality” of gradient updates by orthogonalizing the update matrix, alleviating imbalanced update magnitudes across parameter dimensions during large-model training, and thereby supporting larger learning rates and more stable convergence.
For trillion-parameter-scale training, optimizer stability directly determines experimental cost—a failed training run can consume tens of thousands of GPU-hours. Choosing an optimizer reinforced with clipping mechanisms is essentially an engineering trade-off between convergence speed and training reliability.
4. Core Infrastructure for Agents: Data Synthesis System
If architectural tuning is the “technique,” then a data synthesis system that is comprehensive, scalable, realistic, and interactive is the “foundation” supporting Agentic capabilities. The team’s data synthesis system emphasizes four properties:
- Diversity: Cover as many industries, task types, and tool categories as possible to avoid overfitting to a narrow distribution.
- Scalable: Automate the data production pipeline so new tasks and trajectories can be generated at low cost and in batches.
- Realistic: Synthetic data should reproduce the complexity, parameter constraints, and failure modes of real-world APIs.
- Interactive: An Agent is not a single-turn “question and answer” system; it can execute actions in an environment, observe feedback, and revise strategies.
Through this system, high-quality task trajectories are continuously produced, allowing the model to be repeatedly strengthened across trajectory quantity, quality, and variety—the fundamental reason its Agent-specific metrics can approach the strongest closed-source models.
4.1 Data Synthesis Methodology Referenced from Benchmarks
The work references the methodology of public benchmarks for “large models + tool calling”: collecting large-scale public API interfaces and packaging them as tools for the Agent, thereby building a complete capability gradient from “static calls” to “ambiguous instructions” and then to “multi-turn collaboration.” The value of such benchmarks is that existing tool-calling evaluations are often small in scale, single-dimensional, and disconnected from real tasks, while this methodology fills gaps in scale, dimensions, and cost, providing a comparable and diagnosable evaluation basis for subsequent research.
4.2 Hierarchical API Context Tree: From Domain Trees to Interface Definitions
Its concrete implementation uses a hierarchical API context tree design: first divide into 8 major domains and 68 subdomains (for example, the legal domain is further divided into banking, and the finance domain into securities), then use a large model together with real interface documentation for self-evolving generation—producing function signatures, parameter constraints, and example calls.
The ingenuity of this method lies in combining “real documentation + self-evolving generation”: directly collecting tens of thousands of real interfaces raises copyright, stability, and security-boundary issues, while generating everything from scratch would lose fidelity. Using real documentation as seeds and letting the model derive interface definitions in batches strikes a balance between fidelity and scalability, and also lays the foundation for an automated pipeline in the data synthesis system.
5. Inference Cost: Architectural Benefits Require Matching Architecture to Realize
The cost advantage brought by sparse design is not automatic; it depends heavily on inference-system engineering:
- Large-scale deployment: If mainstream inference optimization methods (large-scale parallelism, cache reuse, etc.) are used with cluster deployment of 128 cards or more, the new model’s inference cost can be roughly comparable to the previous-generation 671B model—total parameters grew by 50%, but cost barely changed. This is the dividend of sparsity.
- Small-scale, low-latency scenarios: If deployed on 16 low-compute inference cards and targeting very low per-token latency (low TPOT), the new model’s higher overall sparsity and more complex expert routing will amplify scheduling and memory pressure in small clusters, making per-request cost higher.
The practical implication for enterprises is direct: trillion-parameter sparse models are suitable for throughput-first, large-scale online services; for budget-constrained private deployments with a fixed number of cards, routing overhead and expert parallelism strategies need careful evaluation, with expert pruning or quantization/distillation when necessary.
6. Summary and Implications
This open-source practice offers three replicable lessons for the industry:
- Architecture evolution should proceed in small, fast steps: Expanding expert count and compressing attention heads on a validated MoE architecture to scale capacity and reduce cost is safer than starting over.
- The core barrier for Agent capability lies in data: A comprehensive, scalable, realistic, and interactive data synthesis system is the infrastructure for continuous evolution of Agentic models, and its importance is no less than that of model structure itself.
- Cost must be calculated by deployment form: The cost advantage of sparse large models depends on large-scale inference architecture, and small-scale deployment requires rebalancing.
For developers and enterprises seeking to evaluate or adopt such models, the recommendation is to start from their own task distribution: if the business involves extensive tool orchestration and multi-turn task execution, priority should be given to the model’s performance on Agent-specific metrics such as tool calling and decision planning, rather than only general leaderboard scores. In the future, as the open-source community continues to improve Agentic data synthesis and evaluation methodologies, “open-source models catching up with the strongest closed-source models in Agent scenarios” may only be a matter of time.
