Evaluating Chinese LLMs and Engineering Model Selection Strategies

1. Background: From Static Benchmarks to Real-World Experience

Over the past two years, evaluating large language model (LLM) capabilities has relied heavily on static benchmark datasets. While these benchmarks offer high comparability and low reproduction costs, they are susceptible to data contamination and prompt engineering tricks. Consequently, scores often diverge significantly from a developer’s actual experience. As models scale into the trillion-parameter range, the industry has gravitated toward two complementary evaluation approaches:

• Comprehensive Intelligence Index: Maintained by third-party evaluation organizations, this metric covers reasoning, knowledge, math, coding, and instruction-following. It outputs a normalized total score, making it suitable for quickly determining a model’s overall capability tier.
• Anonymous Double-Blind Leaderboards: The same prompt is sent to two different models simultaneously. Real users vote for the better output without knowing the models’ identities. The results are ranked by win rate or Elo-based scoring. This approach closely mirrors actual user experience and is particularly effective for evaluating subjective tasks like code generation and copywriting.

Combining these two systems prevents misjudgments such as “high scores but low actual competence” or “good subjective feel but weak foundational skills.” The new generation of ultra-large-scale Chinese models discussed in this article has demonstrated notable progress across both types of evaluations.

2. Capability Positioning: Entering the Frontier Tier

According to public comprehensive intelligence indexes, the latest flagship Chinese models have scored near the international first tier, narrowing the gap with top-tier overseas closed-source models to single-digit margins. While not completely surpassing the strongest tier, they consistently outperform the vast majority of same-generation models, firmly placing them in the “frontier level” category.

More noteworthy is their performance in specific scenarios. On anonymous double-blind front-end code generation leaderboards, one such model secured the top spot with a clear advantage, achieving a head-to-head win rate exceeding 70%—significantly higher than overseas models that had long dominated this space. Key takeaways include:

1. High credibility of the leaderboard mechanism: Because the process is anonymous, double-blind, and relies on real user votes, models cannot gain unfair advantages through targeted prompt templates or training data leakage. The win rate reflects comprehensive performance under diverse, real-world demands.
2. Front-end code generation is a high-value, focused scenario: It requires the model to simultaneously handle structured layout understanding, style semantic mapping, component splitting, and accessibility awareness. It demands robust long-context handling and precise instruction following. Leading in scenario proves that the model’s stability in long-context engineering tasks has reached a highly practical and usable level.
3. First time at the top for a Chinese model: Previously, Chinese models only managed to approach the top spot. This breakthrough demonstrates that Chinese AI teams have achieved mature engineering capabilities in ultra-large-scale training, data proportioning, and alignment strategies.

3. Scale and Cost: The Tradeoffs of 2.8T Parameters

This model features a parameter count reaching the 2.8 trillion level, utilizing a typical ultra-large-scale sparse architecture. Increasing the parameter scale leads to two direct consequences:

• Higher capability ceiling: Larger parameter counts generally mean greater knowledge capacity and deeper reasoning, leading to more stable performance in multi-step reasoning and cross-file code understanding tasks.
• Increased unit call cost: The required compute and memory for inference increase, naturally driving up API pricing.

Based on public pricing, the model’s input and output costs are a few dollars and slightly over ten dollars per million tokens, respectively. This is roughly on par with some mid-tier overseas models but significantly lower than the flagship models of the top overseas tier. Compared to the previous generation of Chinese models, the price has roughly tripled.

This raises a core engineering question: Is it worth calling the most expensive model for every task? The answer is a definitive no.

4. Engineering Selection: Tiered Routing as a Pragmatic Strategy

In actual projects, it is recommended to implement tiered routing based on task complexity rather than using a flagship model for everything. Below is an actionable strategy framework.

4.1 Task Tiering

Task Type	Typical Scenarios	Recommended Model Tier
High Complexity	Cross-file refactoring, architecture design, complex algorithm implementation, long-chain Agent orchestration	Flagship ultra-large model
Medium Complexity	Single-file code generation, unit test writing, document summarization, structured extraction	Previous generation mid-to-large model
Low Complexity	Intent classification, format conversion, simple Q&A, field completion	Lightweight model or rule engine

4.2 Routing Implementation Example

def route_model(task):
    """Select model tier based on task complexity"""
    if task.complexity == "high" or task.context_tokens > 100_000:
        return "flagship-large"
    elif task.complexity == "medium":
        return "mid-tier"
    else:
        return "lightweight"

The key to routing lies in complexity determination. You can score this based on several dimensions:

• Context length: Once a certain token threshold is exceeded, only models with strong long-context capabilities can maintain stability.
• Reasoning steps: Tasks requiring multiple tool calls or chain-of-thought should be prioritized for flagship models.
• Output constraint strictness: When strict adherence to complex schemas or formatting rules is required, instruction-following capability becomes critical.
• Cost of failure: If a generation error could cause a production incident, it is worth spending more to use the most powerful model.

4.3 Balancing Cost and Quality

Consider a coding assistant with an average of one million daily calls. Using a flagship model for all requests could cost several times more than a hybrid routing approach. In reality, tasks that genuinely require a flagship model typically account for less than 30% of total requests. Routing allows you to drastically reduce overhead without significantly compromising overall quality.

5. Scenario Validation: Why Front-End Code Generation Matters

Front-end code generation is a “touchstone” for verifying a model’s comprehensive capabilities for three reasons:

1. Multimodal inputs are common: Mixtures of design drafts, screenshots, and natural language descriptions require the model to have strong semantic alignment capabilities.
2. Complex output structure: The model must generate correct DOM structures, organize styles logically, and consider component reuse and state management.
3. High verifiability: Generated results can be directly rendered, interacted with, and compared, lowering the barrier for user voting verification and making double-blind battle results more reliable.

The high win rate of the flagship Chinese model in this scenario indicates that it has reached a practical level in the following areas:

• Fidelity in capturing design intent;
• Code readability and maintainability;
• Adaptability to mainstream front-end frameworks and styling solutions;
• Ability to maintain context consistency in long conversations.

6. Trends and Recommendations

Overall, the gap between Chinese models and the global frontier is closing rapidly, and in certain specific scenarios, they have even taken the lead. For developers and enterprises, the following recommendations are practically meaningful:

1. Do not rely on a single leaderboard. Combine comprehensive indexes with double-blind battle results to reach a judgment closer to reality.
2. Build your own evaluation set. Construct a regression test set using real business data to periodically evaluate candidate models; this is more reliable than public leaderboards.
3. Adopt a tiered routing architecture. Reserve your most expensive compute for the hardest tasks, and use mid-to-low tier models to cover long-tail requests.
4. Focus on long-context and tool-calling capabilities. These two capabilities determine whether a model can support AI Agent applications and will be key battlegrounds in the next phase of competition.
5. Maintain tech stack replaceability. Encapsulate model calls through a unified abstraction layer to avoid vendor lock-in, allowing for quick switches as capabilities and prices evolve.

The trajectory of Chinese LLMs—moving from “usable” to “good to use,” and now to “leading” in specific scenarios—is clear. For engineering teams, now is an opportune time to systematically integrate these frontier model capabilities into production environments.