Scaling Sparse MoE Models to Trillions of Parameters: Near-Flag-Level Performance with a Fraction of the Inference Cost

One: The Focus Is Not “Bigness” But Smarter Sparsity

A new class of ultra-large-scale sparse models has recently emerged in the industry. These models feature a total parameter count in the trillions, but only a small fraction of parameters are activated during each inference pass. The vendor positions this release as its “most capable version” and simultaneously offers a high-speed inference variant: the intelligence level remains consistent, while output speed improves by roughly two to three times.

Two underlying changes deserve closer attention:

1. Sparse architectures are now systematically closing the gap with closed-source flagship models.
2. Inference pricing has been reduced to approximately one-fifth of comparable flagship models.

For developers, the first point determines whether the model is usable at all; the second determines whether it can be used at scale. This article sets aside marketing language and focuses solely on the technical implementation and engineering benefits of such models.

Two: Parameter Semantics — Trillions Total, 100 Billion Activated

Clarifying the parameter semantics is essential before any discussion of benchmarks or cost.

The model is built on a Mixture-of-Experts (MoE) architecture. The total parameter count is around three trillion, but not all parameters are computed during every inference. According to publicly available technical reports, the total parameter count is approximately 2.8 trillion, with roughly 104 billion activated per inference.

In short: the model is extremely wide, but inference only traverses a narrow path.

This is the fundamental difference between sparse and dense models. A dense model computes through all parameters regardless of input. A sparse model uses a routing mechanism to dispatch each input to the most relevant subset of expert sub-networks. The total capacity is large, but the compute per request is controlled.

From an engineering standpoint, this yields several direct benefits:

• Significantly reduced inference latency.
• Materially lower cost per call.
• Much higher concurrency on the same hardware budget.
• Headroom to expand model capacity for additional knowledge and skills.

Of course, sparse architectures come with trade-offs. Unstable router training, load imbalance across experts, and the possibility that some long-tail tasks only activate sub-optimal experts are all issues that require ongoing monitoring in practice. Based on current results, the engineering team has clearly invested heavily in router quality and training stability.

Three: 1M Token Context and Native Vision for Real Engineering Scenarios

The model supports a 1 million token context window and natively supports vision understanding.

What does 1M tokens mean for a developer? A medium-sized code repository, a multi-round system design discussion, dozens of API documents, or a full day of log troubleshooting can all fit into a single context. Compared to earlier 8K or 32K windows, million-scale contexts significantly reduce the engineering overhead of “chunking, stitching, and re-reading.”

Typical scenarios include:

• Cross-file code comprehension and refactoring.
• Long-chain fault diagnosis and log attribution.
• Long-document summarization, compliance review, and knowledge extraction.
• State retention in multi-turn agent tasks.

Native vision support further expands input types. Screenshots, architecture diagrams, hand-drawn sketches, and table images can all be fed directly into the model without a separate OCR or image-captioning module. For terminal use cases that require “reading an image and modifying code” or “reading an image and writing configuration,” this significantly simplifies the pipeline.

Four: Blind Testing First — A More Pragmatic Release Strategy

Notably, this model was anonymously released to third-party platforms for developer blind testing before the official announcement.

This practice is becoming an industry trend: enter the arena anonymously, face real users and evaluation pressure, and only reveal the identity once results are stable. This is the opposite of the traditional “publish a technical report first, then ship a product” flow, but it is more developer-friendly.

The value of blind testing:

• Results are less distorted by brand bias.
• The model faces a more realistic distribution of questions.
• The scorecard published at launch carries more weight.

Blind testing is not without limitations. During the anonymous period, stability, safety compliance, and billing behavior cannot be verified. These still require observation after the official release.

Five: Benchmark Breakdown — Coding Agents Near Flagship Level, Long-Horizon Tasks Show Clear Gains

The following are the official benchmark results, compared against current mainstream closed-source flagship models. To avoid focusing on specific brands, we refer to them as “Flagship Model A” and “Flagship Model B.”

1. Terminal-Based Coding Agent

On the Terminal-Bench 4.0 benchmark, this model family scores 65.7%, surpassing Flagship Model A by about 7.5 percentage points and trailing Flagship Model B by just 0.7 points.

This benchmark tests agent capabilities in a real terminal environment. It is not merely about completing functions or generating single-file code. It includes:

• Understanding the current directory structure.
• Running commands and observing output.
• Adjusting plans based on error messages.
• Modifying files and re-verifying results.

This is much closer to real development workflows and provides better insight into whether a model can independently close a task loop. A score of 65.7% implies that for a significant portion of terminal tasks, the model already demonstrates usable autonomous execution capability.

2. Long-Horizon Software Engineering Tasks

On SWE-Marathon v1.1, this model achieves 51.7%, surpassing both flagship models. This benchmark focuses on long-horizon tasks: multi-step planning, cross-file modifications, and maintaining state consistency over extended execution.

Long-horizon tasks are the dividing line for agent capability. Short tasks can be won by single-shot generation quality; long tasks test state memory, error recovery, and strategy adjustment. The results suggest that sparse models have built a real advantage in long-horizon software engineering.

3. Comprehensive Agent Evaluation

On Agents’ Last Exam, the model scores 43.2, also ahead of both flagship models. This benchmark focuses on comprehensive agent capability in open-ended environments, including multi-turn interaction, tool calling, and task decomposition.

4. Third-Party Independent Evaluation

An independent evaluation organization assigns an intelligence index of 55, ranking fourth globally, an improvement of 11 points over the previous generation. The same organization notes that once weights are opened, the gap between open and closed-source models will narrow significantly.

This suggests that the combination of “open weights + sparse architecture” is approaching or even partially exceeding closed-source flagship models in localized areas.

Six: Acknowledging the Clear Weaknesses

Benchmarks must not be read through a single lens. The official data also exposes clear deficiencies:

• Scientific computing terminal tasks: still behind flagship models.
• Certain knowledge reasoning tasks: visible gaps.
• On the BenchCAD 3D reconstruction benchmark, this model scores 92.7%, lower than Flagship Model A at 95.9% and Flagship Model B at 96.2%.

Most of these weaknesses relate to visual geometry, physical-world modeling, and complex symbolic reasoning. This indicates that capabilities beyond language and code still need strengthening. For developers whose use cases heavily depend on these abilities, a cautious approach is warranted.

Seven: Cost Analysis — Why Sparse Architectures Bring Prices Down

The official claim is that pricing is one-fifth that of flagship models. This figure is not marketing; it is a direct result of architecture.

A rough engineering estimate:

• A comparable dense model with a total parameter count in the trillions would activate nearly all parameters per inference.
• A sparse model, while similar in total parameters, activates only around a hundred billion.
• Per-inference compute is reduced by roughly an order of magnitude.
• Throughput on the same hardware increases significantly.
• Per-token cost drops accordingly.

In other words, low pricing is not a subsidy — it is the visible manifestation of an architectural dividend. For developers planning to embed large models into high-frequency business pipelines, sparse models offer a more realistic cost structure.

In actual usage, beyond the per-token price, attention should also be paid to:

• Output speed.
• Time to first token.
• Billing logic under long contexts.
• Throughput ceilings and rate-limiting policies.

The high-speed version increases output speed by two to three times, making it suitable for scenarios with stricter interactive response requirements. For batch processing, offline analysis, and other tasks where real-time latency is not critical, the standard version offers better value.

Eight: Practical Recommendations — Which Scenarios Are Worth Trying First

Given the capability and cost profiles above, the following scenarios are good candidates for initial validation:

1. Coding Agents

Terminal-level tasks, cross-file refactoring, and long-horizon debugging already show measurable benefits. It is reasonable to let the model take over repetitive engineering operations, with human developers performing final review only.

2. Long-Document Processing

A one-million-token context can directly handle contracts, technical specifications, and system logs, eliminating the need for chunking and stitching logic.

3. Multimodal Input Scenarios

Screenshot recognition, chart comprehension, and visual grounding in UI automation can simplify existing image-processing pipelines.

4. High-Concurrency Online Services

For businesses that require high-volume calls, the high-speed or standard version should be tested for throughput. The cost advantage of sparse models is further amplified under high concurrency.

One caveat applies to all new models: run a small-scale gray release before full deployment. Build an evaluation set that covers terminal tasks, long-horizon tasks, knowledge Q&A, and safety compliance. Track the model over time rather than relying on a single benchmark score.

Nine: A Broader Perspective — Sparse Models Are Reshaping the Competitive Landscape

For a long time, there was a clear gap between open models and closed-source flagships. The gap was not just in benchmark numbers; it was more evident in the stability of real engineering tasks.

However, a series of recent releases suggests that sparse architecture may be a pivotal variable. It allows models to retain massive capacity while pulling inference cost and latency back into an acceptable range.

When open models reach the intelligence level of flagships and simultaneously cost one-fifth as much, developers are forced to recalculate the cost of trust. If the failure rate on a terminal task drops from 50% to 40%, but the per-call cost drops by an order of magnitude, engineering decisions in many scenarios will change:

• From “rely on a single closed-source flagship” to “open models handle high-frequency tasks, flagship models handle complex decisions.”
• From “manual output filtering” to “parallel multi-model calls with automated verification.”
• From “limited call budgets” to “large-scale batch invocation.”

This shift will not happen overnight, but the direction is clear. For developers, the goal is not to chase the latest model version but to build an evaluation system and model-routing capability that places different models into the right slots based on capability and cost.

Ten: Conclusion — Digest Every Release with an Engineering Mindset

Model releases are becoming more frequent, and benchmark numbers keep climbing. But what truly matters for a business is: at what cost can you use a model, in which part of the pipeline, and how much manual work does it replace?

A total parameter count in the trillions, an activated parameter count in the hundreds of billions, a million-token context, terminal-level agent capability, and pricing at one-fifth of flagship models — taken together, these numbers constitute a new engineering option, not a marketing event.

Developers should focus on three things:

1. Evaluate with your own tasks, not just the official leaderboard.
2. Pay attention to activated parameters and real latency, not just the total parameter count.
3. Treat the model as a component, not as an answer.

The decisive factor in technology selection is not who scores highest on benchmarks, but who can reliably complete a closed loop in your environment at an acceptable cost.