Multi-Dimensional Benchmarking Reveals a New Trend of Capability Divergence in Large Language Models

1. From Leaderboard Rankings to Capability Profiles: A Shift in Evaluation Paradigm

Early LLM evaluations often relied on a single aggregate score for ranking. The limitation of this approach is clear: an aggregate score is a weighted average across multiple capabilities, yet real-world business scenarios rarely require a model to perform evenly across every dimension. A model that excels at mathematical reasoning may not be well suited for long-document summarization. A model with precise instruction following may still prove fragile in multi-step tool calling.

A recent general-purpose benchmark evaluation covering multiple domestic and international models adopted a task design spanning five categories: agentic coding, mathematical reasoning, ultra-long context, precise instruction following, and hallucination control. This task taxonomy is itself noteworthy — it maps to the five most common engineering requirements in current LLM deployment, rather than the conventional categories used in academic benchmarks. The results show that models leading in aggregate scores do not necessarily lead in individual categories, which illustrates precisely why a capability profile holds more decision-making value than a leaderboard position.

This article does not recommend or evaluate any specific product. Instead, it uses the capability divergence observed in the evaluation results as a starting point to discuss how developers can build their own evaluation and model selection methodology.

2. Structural Reasons Behind Capability Divergence

2.1 Differences in Training Objectives and Data Composition

The root cause of divergent performance across categories lies in the objectives set during pre-training and post-training. Taking ultra-long context as an example, a model that scores significantly higher in this category typically indicates heavy investment in training data related to long-document understanding, cross-paragraph retrieval, and positional generalization, along with targeted engineering efforts to optimize attention mechanism efficiency. High scores in mathematical reasoning, by contrast, depend more on high-quality reasoning chain data, process supervision signals, and sampling strategies during inference.

In other words, category scores reflect the allocation preferences of training resources, not simply how “smart” a model is. Recognizing this helps developers avoid the misconception that a high aggregate score implies strong performance across the board.

2.2 Capability Distribution Differences Between Closed-Source and Open-Source Models

In the landscape presented by this evaluation, the top aggregate positions are occupied by closed-source models, while the highest-ranking open-source model trails the leading closed-source models by a considerable margin. This is unsurprising: closed-source models generally benefit from more concentrated compute budgets and more complete post-training pipelines, whereas open-source models face constraints in parameter scale, training data availability, and alignment costs.

What deserves more attention, however, is the relative strength at the category level. In hallucination control, the leading open-source model demonstrated notably strong performance. This suggests that hallucination control does not depend entirely on model scale, but correlates more closely with data cleaning quality, factuality alignment strategies, and constraint mechanisms applied during inference. For teams with limited budgets whose business is sensitive to factual accuracy, this is a valuable signal: capability gaps can be partially addressed through engineering measures.

2.3 What Score Ties and Ranking Rules Imply

The evaluation treats models with score differences within one point as tied in ranking. This rule produces numerous ties in the middle range. From an engineering perspective, this is a reasonable approach — with a limited evaluation sample size, a difference of less than one point likely falls within statistical noise, and forced ranking would amplify random fluctuation.

The practical takeaway for developers is this: when several models cluster within the same score range, you should not use this to make fine-grained rankings. Instead, turn your attention to how category scores match your business scenarios. Models in the middle range are often indistinguishable by aggregate score, yet may show substantive differences on specific tasks.

3. Capability Requirements and Evaluation Considerations Across the Five Task Categories

3.1 Agentic Coding

This dimension assesses a model’s integrated performance in multi-step tool calling, code generation, and execution feedback loops. Key evaluation points include the rationality of tool selection, the accuracy of parameter construction, error recovery capability, and state maintenance across multiple interaction turns. In engineering practice, this capability maps directly to scenarios such as automated operations, CI pipeline assistance, and code review.

It is worth emphasizing that a high score in agentic coding does not equate to being able to replace developers. It measures the quality of a model’s planning and execution within a controlled tool set. In real production environments, the stability of tool interfaces, permission boundaries, and failure retry strategies often determine system usability more than the model itself.

3.2 Mathematical Reasoning

Mathematical reasoning evaluates a model’s derivation accuracy on formal problems, typically including math problem solving, logical reasoning, and symbolic manipulation. The evaluation challenge in this dimension lies in answer verification — the correctness of open-ended reasoning processes is difficult to judge automatically, so most evaluations rely on final-answer comparison, which can mask errors occurring in intermediate steps.

For developers, if your business involves risk calculation, quantitative analysis, or engineering estimation, you should pay attention to model performance in this category. At the same time, establish a process-level sampling review mechanism rather than relying solely on final-answer accuracy.

3.3 Ultra-Long Context

This dimension evaluates a model’s ability to retrieve information from long inputs, integrate content across paragraphs, and generalize positional information. Evaluations typically combine “needle in a haystack” retrieval tasks with long-document question answering. Score differences usually stem from two factors: the truly usable length of the effective context window, and the degree of control over long-distance attention decay.

An important engineering note: ultra-long context capability has a non-linear relationship with inference cost. In a RAG architecture, blindly expanding the context window is not always optimal. Retrieval precision and the quality of context organization matter just as much.

3.4 Precise Instruction Following

This dimension measures a model’s adherence to format constraints, output structures, and multi-condition restrictions. Typical tasks include structured output generation, format conversion, and multi-constraint satisfaction. This capability is particularly critical for API integration scenarios, where downstream systems depend on stable output structures — any format drift leads to parsing failures.

In practice, it is advisable to pair instruction-following capability with output validation mechanisms: even if a model scores well, schema validation and retry fallback should be implemented at the application layer.

3.5 Hallucination Control

Hallucination control measures a model’s tendency to refuse fabrication and actively express uncertainty when facing ambiguous information. Evaluations typically combine factuality-based question answering with adversarial prompting. Scores in this dimension correlate closely with a model’s refusal policy and the quality of factuality alignment data.

For scenarios such as knowledge-base Q&A, customer support assistance, and compliance review, hallucination control often takes priority over aggregate ranking. A model that is slightly weaker in mathematical reasoning but strong in hallucination control may be more suitable for these use cases than a model with the opposite configuration.

4. A Model Selection Methodology for Engineering Practice

4.1 Build a Scenario-Weighted Scoring Model

Rather than adopting aggregate scores from public leaderboards directly, developers should construct weighted scoring based on their own business needs. For example:

# Scenario-weighted scoring example (weights should be adjusted based on actual business needs)
weights = {
    "agent_coding": 0.30,      # Agentic coding
    "math_reasoning": 0.15,    # Mathematical reasoning
    "long_context": 0.20,      # Ultra-long context
    "instruction_following": 0.25,  # Instruction following
    "hallucination_control": 0.10   # Hallucination control
}
def weighted_score(scores: dict) -> float:
    return sum(scores[k] * w for k, w in weights.items())

Weight determination should be based on the relative magnitude of business losses: a system failure caused by format parsing errors typically costs more than a single reasoning answer deviation, so the weight of instruction following is often underestimated.

4.2 Build a Private Evaluation Set

Public leaderboard evaluation sets inevitably deviate from real business distributions. It is advisable to sample from production traffic to construct a private evaluation set covering the following types:

• High-frequency real requests (reflecting main-path quality)
• Boundary and anomalous inputs (reflecting robustness)
• Historical failure cases (reflecting regression risk)
• Adversarial prompt samples (reflecting hallucination tendencies)

A private evaluation set does not need to be large, but it should be version-controlled to ensure comparability before and after model iterations.

4.3 Compose Capabilities Instead of Relying on a Single Model

The reality of category-level divergence means that a single model can rarely be optimal across all dimensions. A viable architectural strategy is capability composition: split long-context understanding, structured output, factuality verification, and other stages across different models or processing phases, coordinated through an orchestration layer. The cost of this approach is increased system complexity, but the benefit is that each stage can use the component that performs better in its specific dimension.

4.4 Continuous Evaluation

Model versions iterate frequently, and conclusions from a one-time evaluation quickly become outdated. It is advisable to integrate evaluation into a continuous integration pipeline: every model version change triggers automated evaluation, producing category-level score comparisons and regression alerts. Evaluation results should be correlated with production metrics such as parsing failure rates, manual review rates, and user feedback distributions to form a closed loop.

5. Conclusion

The core message of the evaluation results is not which model ranks in which position, but that capability divergence has become the norm in LLM applications. A high aggregate score does not guarantee scenario fit. Category-level strengths need to be aligned with business priorities. Models in the middle range should be differentiated through private evaluation sets rather than aggregate rankings.

For developers, there are three actionable takeaways. First, replace aggregate rankings with weighted scoring, with weights derived from business loss magnitude. Second, calibrate public leaderboards with private evaluation sets, focusing on distributional gaps rather than absolute scores. Third, replace reliance on a single model with capability composition and an orchestration layer, assigning each stage to the component most suitable for that dimension. Building an evaluation system itself holds more long-term value than any single leaderboard result.