A New Coordinate System for LLM Evaluation: From Parameter Scale to Agent Tasks and Cost Efficiency

I. Clarifying Parameter Metrics: Total Parameters vs. Activated Parameters

When discussing ultra-large-scale models, parameter size is often the most misleading metric. The industry currently favors the Mixture of Experts (MoE) architecture, which splits the model into multiple expert subnetworks, activating only a portion of them during each inference. Therefore, seeing a figure like “3 trillion total parameters” does not equate to the computational load of a traditional dense model with the same parameter count.

Taking the recently evaluated model as an example, public data indicates a total parameter count of approximately 2.8 trillion, but the actual parameters activated during each forward pass are at the hundred-billion level. This gap directly impacts inference costs. If only activated parameters are counted, the floating-point operations per inference are significantly lower than those of a traditional dense model with an equivalent total parameter scale.

A direct result of this design is the ability to implement more aggressive pricing strategies. Many developers are puzzled as to why a model with massive parameters has a low unit price. The reason is that the MoE architecture shifts inference costs from the “total parameter magnitude” back to the “activated parameter magnitude.”

However, one must be wary of the opposite extreme regarding parameter metrics: do larger total parameters always mean stronger capabilities? The answer is no. Larger total parameters provide more room for expert specialization, but training data quality, routing strategies, and load balancing design are equally crucial. Therefore, when selecting a model, it is recommended to base cost estimates on “activated parameters × inference steps × token count,” rather than being led astray by total parameters.

II. Expanding Context Windows: Which Tasks Truly Benefit?

A one-million-token context window is currently a major watershed for large models across mobile and cloud environments. Its greatest value lies not in “accommodating more conversation history,” but in its ability to directly process a medium-sized code repository.

In traditional development workflows, analyzing a code repository usually requires retrieval tools to split files, build indices, and feed the stitched fragments to the model. This approach has two clear limitations: incomplete retrieval can cause context loss, and the model’s understanding of cross-file call chains heavily depends on the quality of the stitched fragments.

With a context window reaching the million-token level, the model can read the entire repository’s directory structure, key file contents, and dependency relationships in one pass. For medium-sized projects consisting of hundreds of source files, external retrieval can essentially be eliminated. When evaluating model capability here, the focus should be on two dimensions: “the stability of long-distance dependency extraction” and “whether attention degrades over ultra-long sequences.”

It is important to note, however, that long context is not lossless. Even if a model supports a one-million-token window, it does not mean attention quality is perfectly uniform across every position. In practice, input scale preferences still need to be managed: content located towards the middle and end of the sequence is typically recalled first, while information at the beginning may be diluted. Therefore, even with abundant context, structurally organizing input remains necessary. For example, place project descriptions, core configurations, and key files at the beginning of the input, and auxiliary files and log information towards the end.

III. Agent Task Benchmarks: Terminal Environment Metrics vs. General Leaderboards

Traditional leaderboards primarily focus on knowledge Q&A, reading comprehension, and mathematical reasoning. These tasks offer limited insight into real-world engineering scenarios. Over the past two years, new benchmarks capable of evaluating agent proficiency have gained traction, particularly tests that simulate real terminal environments.

The difficulty of terminal environment testing lies in its interactivity. The model must not only generate commands but also observe execution results and adjust subsequent actions based on feedback. For instance, to locate a build failure within a given container environment, the model must read logs, diagnose the error source, modify files, rerun commands, and confirm the fix. This process is far more complex than “generating a correct code snippet.”

According to recent public data, this model achieved 65.7% on the terminal environment benchmark, outperforming a previous closed-source flagship by approximately 7.5 percentage points and falling just 0.7 percentage points short of another top-tier closed-source model. This result indicates that the gap between models in the agent execution chain is rapidly closing.

Performance in long-horizon software engineering tasks is also noteworthy. In such tasks, fulfilling a single requirement often demands dozens to hundreds of continuous steps, including requirement comprehension, code localization, multi-file modifications, test supplementation, and regression fixes. This places extremely high demands on the model’s state retention, task decomposition, and error recovery capabilities. Evaluation results show that this model surpassed two closed-source flagships in this specific category simultaneously.

For developers, these metrics offer greater value for model selection than general Q&A accuracy. If your deployment scenarios involve automated code modification, agent assistants, or CI/CD pipeline integration, you should prioritize terminal environment and long-horizon task benchmarks rather than solely looking at academic exam leaderboards.

IV. Anonymous Pre-release and Third-Party Evaluations: Signals from Open Weights

A notable trend is that an increasing number of open-weight models are opting to enter third-party arenas for blind testing anonymously before their official release. This approach essentially subjects the model to stress tests by the community and evaluation institutions before revealing its identity. Unlike the traditional flow of “publishing a technical report, then launching the product,” anonymous blind testing minimizes scoring biases caused by brand effects.

Signals from independent third-party evaluation institutions are equally critical. This model scored 55 points on the Intelligence Index, an 11-point increase over the previous generation, ranking fourth globally. The evaluating institution also projected that once the weights are open, it will become the highest-scoring open-weight model, shrinking the gap between open-weight models and closed-source flagships from 12 points to just 3 points.

The logic behind this projection is straightforward: open weights allow enterprises to self-deploy, fine-tune, distill, and perform secondary optimizations based on their business scenarios. Conversely, closed-source models only allow API access, limiting flexibility. For development teams that need to control their model’s operating environment to meet data compliance or internal security requirements, the timing of weight availability for open-source models is a critical consideration.

V. Image Understanding and Scientific Computing: A Rational View of Shortcomings

Although this model excels in coding agents and long-horizon tasks, it still lags behind two closed-source flagships in 3D reconstruction benchmarks by approximately 3 percentage points. Shortcomings also exist in scientific computing tasks. These tasks typically involve numerical solving, formula derivation, and precise unit conversions, demanding highly rigorous formal expression during the computation process.

For image understanding and 3D reconstruction scenarios, these capabilities are primarily used in auxiliary design, blueprint parsing, or visual Q&A phases. If high precision is required in these phases, it is recommended to utilize post-processing validation or combine the LLM with specialized vision models, rather than relying on a single language model to complete the entire pipeline.

VI. Selection Recommendations: “Cost Per Agent Task” as the Core Metric

In summary, the evaluation framework for LLM selection needs to shift from “parameter scale” to “cost per agent task.” This metric encompasses at least four variables:

1. Single inference cost, which is positively correlated with activated parameters, inference steps, and input/output token count;
2. Task completion rate, derived from real-world benchmarks like terminal environments or long-horizon tasks;
3. Average inference steps per task, where more steps typically imply higher costs for intermediate error recovery;
4. Error recovery capability, meaning whether the model can self-correct based on environmental feedback after a failure.

If a model improves agent task completion rates by 20% but its unit price is only one-fifth of a comparable model, its economic advantage becomes highly significant when calculated from the perspective of “total token cost per successfully completed terminal task,” far outweighing a simple unit price comparison.

For enterprise developers, it is advisable to first establish an internal capability baseline on a small-scale, real-world task set, and then conduct comparative experiments against different candidate models using the same task set. Metrics should not be limited to final completion rates; average step counts, average token consumption, and the distribution of failure scenario types should also be recorded. This approach prevents being misled by a single leaderboard and ensures model selection genuinely aligns with specific business requirements.