LLM Evaluation Shifts Toward Agentic Capabilities: Inside the New Intelligence Index and Agent Arena

1. Reading the Leaderboard: From Benchmark Scores to Real-World Performance

A leading global LLM evaluation platform has released its latest intelligence index rankings. In a comprehensive evaluation covering more than 700 mainstream models, several Chinese models performed strongly: Kimi K3.1 Pro ranked first among domestic Chinese models, with Tencent Hy4 close behind, demonstrating solid overall intelligence. Meanwhile, in the Agent Arena evaluation, which is closer to real business scenarios, Tencent Hy4 reached 13th place on the global leaderboard and ranked first among open-source models worldwide.

The value of this leaderboard lies not only in the rankings themselves but in what they reveal about a profound paradigm shift in how large models are evaluated. Traditional evaluations rely mainly on static question-answering datasets to assess a model’s knowledge base and basic reasoning ability—typically multiple-choice questions, knowledge quizzes, and logic puzzles. While such evaluations are highly standardized and reproducible, their limitations are becoming increasingly apparent: high scores on static questions do not necessarily translate into stable performance in actual operations.

The new generation of evaluation benchmarks focuses more on a model’s “agentic capability”—its performance in complex tasks that involve tool calling, multi-step planning, environment interaction, and state memory. These evaluations simulate the complete workflow an agent completes in the real world, rather than isolated knowledge Q&A. In Agent Arena, for example, a model must autonomously understand the goal, decompose it into steps, invoke external tools, handle mid-course exceptions, and ultimately deliver results. This process places far higher demands on semantic understanding, logical planning, noise resistance, and tool proficiency than traditional Q&A.

2. Technical Depth: Core Factors Driving Intelligence Index Gains

Behind the strong performance of the top-ranked models is a set of coordinated innovations in key technologies.

2.1 Fine-Grained Evolution of Mixture-of-Experts Architectures

The new generation of models generally adopts or optimizes Mixture-of-Experts (MoE) architectures. The core idea is to partition a massive parameter space into multiple “expert” subnetworks and use a gating mechanism to dynamically route input samples, activating only the few experts most relevant to the current task. This design substantially increases the model’s overall capacity and expressiveness without significantly raising inference compute costs.

Leading models on the leaderboard have made several fine-grained improvements to MoE architectures: first, optimization of expert granularity, moving from coarse-grained expert partitioning toward fine-grained, diversified expert responsibility allocation; second, smarter routing strategies, where the gating network no longer relies simply on input vector similarity but introduces reinforcement learning signals so the model can adaptively learn optimal routing policies during training; third, innovative load-balancing approaches that use auxiliary loss functions or local routing mechanisms to avoid the imbalance where popular experts are overloaded while cold experts sit idle.

2.2 Breakthroughs in Long-Context Processing

Test results for the leaderboard models show that context window length and long-text comprehension quality have become important dimensions of model capability. Leading models maintain high information retrieval accuracy and logical consistency when processing ultra-long texts exceeding 100,000 tokens. This is enabled by several technical breakthroughs:

• Improved positional encoding: Building on traditional rotary positional encoding, interpolation and extrapolation mechanisms are introduced, enabling models to handle inference contexts far exceeding training sequence lengths.
• Sparsification of attention mechanisms: By combining local window attention with global sparse attention, computational complexity is significantly reduced while ensuring sufficient information flow.
• Adapted training strategies: A progressive “short-to-long” training strategy is used—first stabilizing convergence on short sequences, then gradually extending sequence length for fine-tuning, effectively mitigating loss oscillation during long-sequence training.

2.3 Maturation of Inference-Time Compute Strategies

The performance of leading models in Agent Arena benefits greatly from a new paradigm of inference-time compute. Unlike traditional single-pass forward inference, the new generation of models introduces more complex decision mechanisms at inference time, such as:

• Chain-of-thought self-consistency checking: The model generates multiple reasoning paths and selects the most reliable answer through voting or scoring mechanisms.
• Active planning and tool orchestration: When facing complex tasks, the model not only generates text but can actively plan multiple subtasks, select appropriate tools (such as search or code execution), and invoke them in sequence.
• Adaptive reasoning depth: The model can dynamically adjust reasoning depth based on task complexity—responding quickly to simple questions and allocating more compute for deep thinking on complex ones.

These strategies move models from “passive answering” toward “active problem-solving” and form the engineering foundation for genuine agentic capability improvements.

3. The Dual-Track Evaluation System: Complementary Intelligence and Agentic Capabilities

Notably, this leaderboard release adopts a dual-track evaluation system, with the comprehensive intelligence index and the agent arena each emphasizing different aspects. This design itself represents an industry-recognized methodological advance aimed at avoiding the limitations of a single dimension.

The comprehensive intelligence index focuses on the breadth of a model’s knowledge coverage, language understanding accuracy, basic reasoning ability, and generation quality. Its evaluation items typically have clear scoring criteria, making it suitable for rapid horizontal comparison of different models’ baseline levels. This type of evaluation is very helpful for enterprises conducting initial screening during model selection, eliminating clearly unsuitable candidates at low cost.

The agent arena simulates real business scenarios and focuses on evaluating a model’s actual delivery capability in automated tasks. Its scoring dimensions include task completion rate, tool-calling success rate, path planning efficiency, error recovery capability, and the practical utility of final outputs. Such evaluations are more costly and take longer, but their results have greater reference value for production environments—especially for enterprises building intelligent customer service, automated operations, code generation assistants, and similar business scenarios.

In practice, we recommend that enterprises build a “score + scenario” two-way evaluation mechanism: first establish a screening baseline through the comprehensive intelligence index, filtering out the top few candidate models that meet basic capability thresholds; then validate agentic capability in a high-fidelity test environment closely aligned with their own business, paying particular attention to model performance in domain terminology understanding, private tool adaptation, and exception input handling.

4. Engineering Practice: Turning Evaluation Insights into Selection Decisions

For developers and enterprise architects, the most important thing when facing the massive data in leaderboards is not memorizing the ranking order but building a scientific framework for model evaluation and selection.

4.1 Build a Differentiated Evaluation Weight System

Different business scenarios place very different weights on model capabilities. For example, an intelligent application for legal document review places far higher demands on long-text comprehension, clause matching, and logical consistency than on creative writing ability; while a marketing copy generation tool places more emphasis on language fluency, style adaptation, and diversity. Therefore, we recommend that enterprises define a weight matrix based on real business scenarios, mapping the multi-dimensional metrics in leaderboards to their own needs rather than relying simply on overall rankings.

4.2 Consider Domain Adaptation and Fine-Tuning Costs

Leaderboard evaluations cover general scenarios, while actual business is often highly vertical. A model ranked 20th in general evaluations may, after sufficient domain fine-tuning, outperform a general model ranked 5th in a specific business scenario. Enterprises should include “potential fine-tuning gains” as an important variable in evaluation—considering dimensions such as the model’s degree of openness, the usability of fine-tuning tools, and domain fine-tuning experience accumulated in the community.

4.3 Balance Inference Cost and Performance

Leaderboard rankings do not include inference cost information, but cost is often a key factor in production decisions. A highly ranked model with slow inference and high per-token cost may be less suitable in high-concurrency real-time interaction scenarios than a slightly lower-ranked but more efficient model. We recommend that enterprises include inference cost, latency, and throughput in their comprehensive evaluation system and use stress testing to validate model performance under real loads.

4.4 Establish Dynamic Evaluation and Replacement Mechanisms

The technology iteration cycle in the LLM field has shortened from “years” to “months.” A model that ranks well this quarter may be surpassed by several new models next quarter. Enterprises should avoid treating model selection as a one-time decision and instead build a dynamic management mechanism that includes model monitoring, regression testing, canary switching, and emergency rollback. Regularly re-evaluating deployed models and candidate new models using the same test sets, and building automated workflows for model version replacement, can effectively reduce the risk of accumulating technical debt.

4.5 Build Scenario-Specific Evaluation Datasets

No public leaderboard can fully replace an evaluation system built by an enterprise on its own business data. We recommend that development teams extract typical interaction samples from real business logs and combine them with manual annotation to build a “golden test set.” This test set should cover diverse scenarios including normal flows, boundary conditions, and exception inputs to validate model capability on the long-tail distribution.

5. Summary and Outlook

This round of rankings shows that domestic Chinese large models have reached internationally leading levels in both comprehensive intelligence and agentic capability, and the gap between top models is narrowing—competition has entered a new stage where “details determine success.” In the future, LLM evaluation systems will become more diverse and multi-dimensional, moving from single intelligence tests toward systems engineering covering performance, cost, safety, and controllability. For developers, understanding the technical logic and business implications behind these evaluations and internalizing them into daily workflows will be key to maintaining sustained competitiveness.