Domestic LLM Capabilities: A Full Benchmark Analysis — Capability Divergence and Engineering Selection

1. Evaluation Context: Why Developers Should Read Benchmark Results Carefully

A recent general-purpose LLM benchmark evaluation was published, covering 19 models from both domestic and international providers. The evaluation set spans five task dimensions: agentic coding, mathematical reasoning, ultra-long context, precise instruction following, and hallucination control. Closed-source models still lead on the overall leaderboard, while open-source models from domestic providers have formed clear tiers.

For developers, the real value of a leaderboard is not “who is first” but capability divergence across dimensions: no single model leads evenly across all dimensions, which means a “one model replaces all” strategy rarely holds up in engineering practice. Understanding this divergence is what enables sound model composition and architecture decisions.

2. Overall Leaderboard: The Structural Gap Between Closed-Source Leaders and Open-Source Challengers

By total score, two closed-source models occupy the top two positions with 73.62 and 71.76, respectively. The highest-ranked domestic open-source model scores 61.46, followed by 58.95 and 50.66, forming the first tier of domestic open-source models.

These numbers convey three key points:

1. The absolute gap exists but is not insurmountable. The total score gap between the top closed-source model and the strongest domestic open-source model is about 12 points, roughly “one generation” on a composite benchmark—but the gap distribution across specific tasks is uneven.
2. The open-source tier is already stratified. The three score bands of 61, 58, and 50 correspond to three different engineering positions: the top can handle primary workloads, the middle can handle auxiliary and batch tasks, and the lowest is better suited for cost-sensitive scenarios.
3. Total score is the crudest metric. It is a weighted average, and the weighting design determines the ranking. Per-dimension data is therefore more valuable for decision-making than the total score.

3. Dimension-by-Dimension Breakdown: Where Each Model’s Strengths Lie

3.1 Ultra-Long Context: A Hard Indicator of Capability Ceiling

The top closed-source model scores 95.63 on ultra-long context. In engineering terms, this means that with inputs at the hundreds-of-thousands token level, the model still maintains high information recall and localization accuracy. Long-context capability directly determines the feasibility of several scenarios:

• Whole-repository code understanding and cross-file refactoring suggestions;
• One-shot summarization and Q&A over long documents (contracts, specifications, logs);
• Stable retention of historical state across multi-turn conversations.

Note that a high long-context score does not mean “filling the window is all you need.” In practice, focus on the effective context length—the input size at which accuracy on your target task does not degrade significantly. This typically requires testing with your own data rather than trusting benchmark scores directly.

3.2 Mathematical Reasoning: A Specialized Strength for Structured Problems

One closed-source model scores 76.32 on mathematical reasoning—the highest in this dimension. Mathematical reasoning capability is strongly correlated with the following tasks:

• Formal derivation and boundary-condition checking for complex business rules;
• Correctness verification of SQL/query logic;
• Algorithm problem solving, complexity analysis, and code correctness reasoning.

A common misconception in practice is equating “strong mathematical reasoning” with “strong coding.” The two are related but not equivalent: coding tasks also stack API knowledge, engineering conventions, and instruction-following ability, which must be evaluated separately.

3.3 Instruction Following and Hallucination Control: Two Pillars of Production Reliability

The top closed-source model scores 88.97 on instruction following and 83.19 on hallucination control—a “balanced” profile. The leading domestic open-source model scores 67.78 on hallucination control, a notable advantage among domestic models. The engineering value of this dimension is often underestimated:

• Instruction following determines output format stability. In pipelines requiring structured output (JSON, specific fields, fixed templates), a low instruction-following score directly increases parse failure rates, which in turn drives up retry costs.
• Hallucination control determines the trustworthiness of factual tasks. In knowledge Q&A, document extraction, and code citation scenarios, a high hallucination rate means an additional verification layer is mandatory.

A practical rule of thumb: for format-oriented tasks, look at instruction following; for fact-oriented tasks, look at hallucination control. When neither meets the bar, do not try to force it with prompts—add an external validation layer or switch models.

4. Engineering Strategies for Capability Composition

Since capability divergence is the norm, engineering teams should abandon the “pick the single strongest model” approach and shift to a task-routing architecture:

# Abstract illustration of task routing (pseudocode, no vendor-specific APIs)
TASK_PROFILES = {
    "long_doc_qa":    {"need": ["long_context", "hallucination"], "min_score": 80},
    "code_refactor":  {"need": ["coding", "instruction"],       "min_score": 75},
    "math_verify":    {"need": ["reasoning"],                   "min_score": 70},
    "batch_extract":  {"need": ["instruction", "cost"],         "min_score": 60},
}
def route(task_type, candidates):
    profile = TASK_PROFILES[task_type]
    # Filter by required dimensions, not by total score
    valid = [m for m in candidates
             if all(m.dim_score(d) >= profile["min_score"] for d in profile["need"]
                    if d in m.dimensions)]
    # Among equally capable models, prefer the lower-cost one
    return min(valid, key=lambda m: m.cost_per_1k_tokens)

The core of this routing logic is: use dimension thresholds for filtering, use cost for ranking. Total score only serves as a secondary reference after all dimension thresholds are met.

5. How to Use Benchmark Data Correctly

Benchmark scores carry three types of systematic bias that can easily lead to poor decisions if taken at face value:

1. Distribution mismatch between evaluation sets and business data. Benchmark tasks are typically constructed generically, while business data has strong domain characteristics (proprietary terminology, internal formats, industry conventions). Build a minimal evaluation set of 200–500 of your own samples covering typical tasks and edge cases.
2. Score reproducibility issues. The same model shows significant score variation under different temperatures and prompt templates. Engineering evaluations should fix decoding parameters and prompt templates, and average across multiple samples.
3. The weighting black box of composite scores. Total score rankings depend on weight design, and those weights may not match your business distribution. If long-context tasks account for a high proportion of your workload, look at that dimension separately rather than the total.

A practical small-scale evaluation workflow:

1. Sample 300 real requests from production logs, stratified by task type
2. Define automatically verifiable success criteria for each task type (format validation / key field hit / unit test pass)
3. Run all candidates with unified parameters and prompt templates; record success rate and P95 latency
4. Compute "weighted success rate = Σ(task proportion × success rate for that type)," combine with unit price to derive cost per task
5. Reserve 10% of samples for manual spot-checking to calibrate the credibility of automated metrics

6. Selection and Governance Recommendations

Combining the leaderboard structure with engineering practice, here are the recommendations:

• Dual-track primary + fallback models. The primary model handles high-value tasks; the fallback handles batch processing and degraded traffic. Their capability dimensions should complement rather than overlap.
• Include hallucination control in SLOs. Set a spot-check pass-rate threshold for factual outputs. When below the threshold, trigger human review or model switching rather than relying solely on prompt constraints.
• Long context does not mean unlimited context. Prioritize retrieval and segmentation. Treat long context as “a means to reduce information loss,” not “an excuse to skip engineering design.”
• Watch for regression across version upgrades. Model version iterations may bring structural changes in capabilities. Run your own evaluation set before upgrading to avoid “total score up but critical task degraded.”
• Account for cost per task. Replace “price per thousand tokens” with “cost per thousand successful tasks” to reflect the true overhead of retries, validation, and human fallback.

Conclusion

The most important takeaway from this benchmark is not the ranking itself, but that capability divergence is now significant enough to support a task-routing engineering architecture. Closed-source models lead in balance and long context, while domestic open-source models have established local advantages in dimensions such as hallucination control at lower cost. The right approach for developers is to treat the leaderboard as a rough version of a capability map, draw a precise version using their own business data, and then make model composition, validation layer design, and cost accounting decisions accordingly—the leaderboard provides direction, and your own evaluation provides the answer.