Chinese LLM Benchmarking: Three Dimensions Beyond the Overall Score
This article reviews a new round of Chinese large-model benchmarks in which top models cluster within a one-point band, explaining why such scores are statistically indistinguishable and why selection should shift to completion rate, cost, subskill performance, and task-specific evaluation. It covers agentic coding, token economics, hallucination control, long-context use, precise instruction following, multi-step reasoning, private evaluation sets, layered routing, and cloud-based observability.
1. What the Latest Benchmark Reveals: Top Models Are Clustering in the Same Tier
Recently, a new round of Chinese general-capability benchmarks was released. Two participating models tied with the previous round’s leader; the two new models’ overall scores differed by less than one point. Under the benchmark rules, models whose scores differ by no more than one point are treated as tied.
This result carries several implications:
- Capabilities are becoming more homogeneous: On the general tasks covered by the benchmark, leading models now struggle to create a clear gap between one another.
- The focus of model selection is shifting: Asking “which model has the higher score?” is no longer enough. Timeout rates, per-task cost, and strengths across individual subskills have become more important decision variables.
- Optimization directions are clear: The new models made targeted improvements in areas such as complex reasoning, engineering development, inference overhead, native visual understanding, computer operation, long-text processing, agent planning, and code development. These are exactly the capabilities that matter most in real-world deployments.
For developers and enterprises, leaderboards provide useful directional signals, but they cannot replace evaluation against your own workloads. The sections below first explain why the benchmark rules are designed this way, then examine each dimension in turn, and finally provide practical guidance for adoption.
2. Benchmark Rules: Why a One-Point Gap Is Treated as a Tie
Setting the tie threshold at one point is not arbitrary. It reflects uncertainty in LLM evaluation. Model benchmark scores have uncertainty from four main sources:
- Question sampling error: A test bank is only a finite sample of the real capability space. Statistically, the standard error of the mean is approximately
s/√n, wheresis the sample standard deviation of question scores andnis the number of questions. Fewer questions and higher per-question variance increase uncertainty in the average score. - Scoring error: Subjective questions are usually evaluated by a judge model or human reviewers. Scoring granularity, interpretation of scoring criteria, and judge-model bias can all affect results.
- Sampling randomness: When decoding parameters that introduce randomness are non-zero, running the same question multiple times can produce different answers.
- Task completion: Agentic tasks may be interrupted by failed tool calls or runtime timeouts. In such cases, a zero score may not necessarily reflect a capability gap.
Therefore, models within the same tie band should be treated as statistically indistinguishable, and the decisive basis for selection should shift toward subskill capabilities and engineering metrics.
Another often-overlooked detail is that the leaderboard ranks long-context performance, hallucination control, precise instruction following, agentic coding, and other subitems separately. The overall score is only a weighted aggregate. Models with nearly identical total scores can still differ substantially in individual capabilities, so the final choice must be made in the context of the actual business scenario.
3. Agentic Coding: From “Getting It Right” to “Delivering on Time”
Agentic Coding is one of the most closely watched benchmark directions today. It no longer requires a model merely to produce a correct code snippet. Instead, it places the model in a closed loop that resembles real development work:
- Understand the task description and relevant codebase;
- Use tools such as file read/write, command execution, and test execution;
- Go through multiple rounds of reasoning and revision;
- Judge the result by whether the code runs and whether tests pass.
Benchmark data shows that this type of task has a long-standing pain point. In the previous round, the timeout rates for two participating models were 69.23% and 26.92%, meaning that a substantial proportion of questions could not be completed within the allotted time. In the new round, all participating models completed every test case, the timeout rate dropped to 0%, and the average number of output tokens per task was also noticeably reduced.
This change reflects systematic engineering optimization:
- Controlling reasoning depth: Balancing between “thinking clearly” and “thinking too much,” avoiding repeated self-reversal that leads to overly long output.
- Compressing tool output: File contents, command execution results, and error messages can be very large. They need to be truncated, summarized, or retrieved on demand so that irrelevant information does not drown the context.
- Improving error recovery: When compilation fails or tests fail, the model should use error information to locate and fix the problem precisely, rather than rewriting large sections wholesale.
- Optimizing context management: In long sessions, retain key decisions and constraints while discarding redundant history.
It is important to emphasize that completion rate is not the same as correctness. A 0% timeout rate only means that models can “finish” the tasks. Some completed outputs may still fail compilation, fail tests, or merely satisfy surface-level requirements. When evaluating coding agents, teams should track the following metrics together:
| Metric | Meaning | Engineering significance |
|---|---|---|
| Completion rate | Proportion of tasks completed within the time limit (1 − timeout rate) |
Whether the model can be reliably embedded in time-bounded pipelines |
| Test pass rate | Proportion of tasks whose outputs pass all test cases | The real “delivery quality” |
| Average steps per task | Average number of tool calls and interaction rounds | Reflects planning efficiency and the amount of redundant actions |
| Average tokens per task | Sum of input and output token consumption | Directly determines latency and call cost |
4. Token Economics: Stronger Can Also Be Cheaper
One of the most notable results from this benchmark is that capability improvement and cost reduction occurred at the same time. The new participating models had average per-task costs of RMB 1.90 and RMB 2.52, representing decreases of approximately 87.0% and 28.5% compared with the previous generation. The previous-generation model had a test timeout rate as high as 69.23%, and timeouts and retries are a major source of hidden cost.
4.1 What Drives Cost
Under a usage-based billing model, the cost of a call can be simplified as:
Call cost ≈ input tokens × input price+ output tokens × output price+ extra calls caused by failed retries
Two points are often underestimated in Chinese-language applications:
- Token count is not proportional to character count: A Chinese character may be split into one or more tokens depending on the model’s tokenizer. Budgeting should not simply convert character counts into token counts.
- Long contexts amplify cost: The longer the context, the more input tokens are required and the higher the latency. When running agentic tasks in batch, inefficiency in a single step can accumulate into significant cost across hundreds or thousands of tasks.
4.2 Where Cost Reductions Come From
- Model and inference side: Through architectural and inference-process optimization, models can produce the same quality conclusions with fewer output tokens, effectively reducing per-task cost for long-chain workloads.
- Pricing side: After inference efficiency improves and services scale, vendors often lower prices, which reduces the unit price per token.
- Calling side: Eliminate timeouts and retries, simplify prompts and context, make good use of caching, and route requests to smaller model variants where appropriate.
The following Python example shows how to estimate per-task cost and account for retries. In practice, the billing logic can be adjusted to match the pricing model of the cloud platform you use:
from dataclasses import dataclass@dataclassclass Pricing:input_per_1k: float # Input price: RMB per 1K tokensoutput_per_1k: float # Output price: RMB per 1K tokenscached_input_per_1k: float = 0.0 # Input price for cached tokensdef estimate_cost(usage: dict, price: Pricing, retries: int = 0) -> float:"""Estimate the total cost of a task, including retries, based on input and output token usage."""cached = usage.get("cached_tokens", 0)fresh_input = usage["input_tokens"] - cachedper_call = (fresh_input * price.input_per_1k / 1000+ cached * price.cached_input_per_1k / 1000+ usage["output_tokens"] * price.output_per_1k / 1000)return per_call * (retries + 1)# Example: the same task can cost much less after context optimizationprice = Pricing(input_per_1k=0.004, output_per_1k=0.012, cached_input_per_1k=0.0004)before = estimate_cost({"input_tokens": 20000, "output_tokens": 4000, "cached_tokens": 0},price, retries=1,) # Retry once because the task was interrupted or validation failedafter = estimate_cost({"input_tokens": 6000, "output_tokens": 2000, "cached_tokens": 4000},price,)print(f"Estimated cost before optimization: {before:.2f} RMB; after optimization: {after:.2f} RMB")
5. Component Capability Breakdown: Four Technical Threads
5.1 Hallucination Control: Make Outputs Grounded
Hallucination refers to model-generated content that appears plausible but lacks factual support. Common types include:
- Factual errors: People, dates, numbers, or citations do not match the facts;
- Off-topic answers: The output is fluent but does not address the user’s actual question;
- Confident mistakes: The model gives a definitive conclusion without expressing uncertainty.
In this benchmark, the best-performing model scored 73.61 on hallucination control, an improvement of 40.56 points over the previous generation. Such a large jump suggests that this capability had previously been a common weakness among participating models.
In engineering practice, a common approach is a three-layer defense: constrained generation, retrieval augmentation, and output validation.
- Constrained generation: Explicitly require the model to answer only from provided materials, and to respond “not mentioned in the materials” when the answer is unavailable.
- Retrieval-Augmented Generation (RAG): First retrieve relevant fragments from a knowledge base, then ask the model to generate an answer based on those fragments, with source citations.
- Output validation: Use rules or a second model call to verify key facts against the original materials.
The following is a typical example of a constrained-generation prompt:
Please answer strictly based on the provided materials and follow these rules:1. Use only information that appears in the materials; do not rely on the model’s own memory.2. After each key conclusion, cite the corresponding source number in brackets.3. If the materials are insufficient, answer only "Not mentioned in the materials" and do not guess.Materials:[1] ...[2] ...Question: ...
5.2 Long Context: Effective Use Matters
Participating models scored 77.80 and 88.86 on the long-context subitem, with the latter improving by 40.99 points over the previous generation. First, two concepts need to be distinguished:
- Context window length: The maximum number of tokens the model can accept in a single request;
- Effective context capability: The ability to accurately locate and use key information within a long input.
Having a sufficiently large window does not mean the model can use it well. “Needle-in-a-haystack” information retrieval tests have become a common evaluation method precisely because they directly examine the latter. In practice, tasks such as long-document question answering, contract review, and codebase analysis place higher demands on effective context capability. Common engineering practices include:
- Placing key information at the front or back: Put the question and constraints at the beginning or end;
- Structured input: Use clear headings and numbering to divide sections;
- Retrieve and compress: Retrieve relevant fragments first, then summarize or compress them before inserting them into the context;
- Progressive reading: Summarize a long document in segments first, then answer based on the summaries.
5.3 Precise Instruction Following: Format and Constraints Are Also Functional Requirements
Precise instruction following measures whether a model can strictly obey explicit constraints such as format, length, negation rules, and structural requirements. Examples include “output only JSON,” “do not exceed 100 words,” and “do not use passive voice.” The best-performing model in this round scored 50.00 on this subitem, which is still below the passing line. This shows that, for all participating models, precise instruction following remains a capability that needs improvement.
In engineering practice, precise instruction following directly determines whether model output can be consumed by downstream programs. A common approach is structured output constraints with JSON Schema: define field names, types, value ranges, and required fields in advance, require the model to follow the schema, validate the format on the calling side, and automatically retry or repair when validation fails.
{"type": "object","properties": {"intent": {"type": "string", "enum": ["query", "submit", "complaint", "other"]},"summary": {"type": "string", "maxLength": 120},"entities": {"type": "array","items": {"type": "object","properties": {"name": {"type": "string"},"type": {"type": "string", "enum": ["time", "location", "amount", "person"]}},"required": ["name", "type"]}}},"required": ["intent", "summary", "entities"]}
5.4 Complex Reasoning and Agentic Planning: From Single-Step to Multi-Step
The best overall-performing model focused on improving complex reasoning and engineering development capabilities while optimizing inference overhead. Another participating model strengthened agentic planning and code development. Together, these trends point to the same direction: mainstream models are moving from “single-turn question answering” toward “multi-step task execution.”
The quality of multi-step tasks depends on three stages:
- Task decomposition: Whether the model can break vague requirements into clearly bounded and verifiable subtasks;
- State maintenance: Whether the model can consistently remember goals and constraints during execution without being misled by intermediate information;
- Error recovery: When one step fails, whether the model can adjust its strategy based on the error rather than mechanically retrying.
When debugging such agents, one of the most effective troubleshooting methods is to record a trace log for every tool call and its return.
6. Practical Implementation: Build Your Own Evaluation Set
When top models are tied, the most reliable approach is to build an evaluation set that closely matches your own business. A recommended five-step process is:
- Collect tasks: Sample 100–300 real business requests. Stratify them by task type, cover both high-frequency and long-tail scenarios, and redact personal and sensitive data.
- Define pass criteria: For each task type, provide measurable pass conditions. For example, a code task could require “compilation succeeds and all unit tests pass,” while an extraction task could require “the output conforms to the JSON Schema and field values match the annotations.”
- Set resource limits: Apply the same time limit, token budget, and concurrency conditions to all candidate models to ensure a fair comparison.
- Record multiple dimensions: For each result, record quality, latency, token usage, cost, and timeout status, and retain the full call trace.
- Use weighted scoring: Assign weights to quality, cost, and stability based on business priorities.
import time# Example: a unified evaluation runner; all candidate models use the same standarddef run_eval_task(task, call_model, timeout: float = 120.0, token_budget: int = 32000):start = time.monotonic()try:result = call_model(task["prompt"], timeout=timeout, max_tokens=token_budget)except TimeoutError:return {"status": "timeout", "elapsed": time.monotonic() - start}except Exception as exc:return {"status": "error", "elapsed": time.monotonic() - start, "error": str(exc)}passed = evaluate(result, task["expected"]) # Business-specific pass/fail logicreturn {"status": "done","passed": passed,"elapsed": time.monotonic() - start,"usage": result.usage, # Input, output, and cached token counts}def weighted_score(metrics: dict, weights: dict) -> float:"""Apply business weights; metrics are expected to be normalized to 0–1 beforehand."""normalized = {k: v for k, v in metrics.items() if k in weights}return sum(normalized[k] * w for k, w in weights.items()) / sum(weights.values())# Example weights: prioritize quality while accounting for stability and costscore = weighted_score({"quality": 0.92, "stability": 0.98, "cost_efficiency": 0.85},{"quality": 0.5, "stability": 0.3, "cost_efficiency": 0.2},)print(f"Weighted score: {score:.3f}")
Evaluation is not a one-time task. The evaluation set should be incorporated into a continuous integration pipeline so that every model version upgrade or prompt change triggers automated regression testing. This helps detect capability regressions early, such as worse precise instruction following leading to structured-output parsing failures.
7. Engineering Guardrails: Layered Routing and Budget Control
When top models are tied in capability, a more pragmatic architecture is layered dispatch: simple tasks go to lower-cost, smaller models, while complex tasks go to models with stronger overall capabilities. Routing can be handled by rules or a lightweight classification model.
# Example: route tasks to different model tiers based on task characteristicsdef route(task: dict) -> str:# High-complexity tasks: multi-step reasoning, code generation, and debuggingif task["need_reasoning"] or task["need_code"]:return "strong_model"# Very long inputs: use vector retrieval to narrow the scope, then use a strong long-context modelif task["input_tokens"] > 30000:return "long_context_model"# Simple classification or extraction tasks: prefer lightweight modelsreturn "light_model"
The following measures can help maintain quality while controlling cost:
- Cache reuse: Use prefix caching for system prompts, fixed instructions, and repeated Q&A;
- Timeouts and retries: Set a timeout and maximum retry count for each call to prevent long-tail requests from dragging down the overall system;
- Token budgets: Set an upper limit for each task, and truncate or degrade gracefully when the limit is reached;
- Monitoring and alerting: Track call volume, token consumption, timeout rates, and failure rates by model, business line, and scenario, and trigger alerts when abnormal fluctuations occur;
- Data protection: Before calling external model services, redact sensitive information and follow relevant data security and personal information protection requirements.
8. Cloud Implementation: Building Observable Evaluation and Routing Pipelines
On cloud platforms, the above approach can be implemented by combining general cloud capabilities:
| General capability | Role in evaluation and routing |
|---|---|
| Model inference services | Provide unified API access to models of different sizes and versions, with switching handled through configuration on the business side |
| Logging services | Centrally collect prompts, outputs, token usage, and latency for each call to support evaluation review and auditing |
| Monitoring and alerting | Set threshold-based alerts for call success rate, timeout rate, token consumption, and per-task cost |
| Object storage | Store redacted evaluation sets, call-trace snapshots, and regression test reports |
| Message queue | Smooth traffic spikes and support asynchronous workloads such as batch regression evaluation |
| Container platform | Deploy evaluation runners and agent tool environments, with elastic scaling based on task load |
Example architecture:
Business request│▼Lightweight classification ── simple tasks ──▶ Lightweight model│├── complex tasks ───────────────────────▶ Strong model│└── very long input ──▶ vector retrieval/summary compression ──▶ Long-context model│Model inference service (unified API)│┌──────────────┬─────────────┴─────────────┬──────────────┐▼ ▼ ▼ ▼Logging Monitoring/alerts Object storage Message queue(call traces) (timeouts/cost) (eval sets/reports) (batch regression)
Before launch, it is recommended to use shadow traffic so that candidate models receive real requests in parallel without returning their results to users. After comparing pass rates, cost, and data security performance, roll out the models gradually by business line.
9. Conclusion
The latest benchmark shows that top Chinese general-purpose models have entered the same overall capability tier, and model selection is shifting from “score-only evaluation” to multi-dimensional assessment:
- Look at ties: When overall scores differ by no more than one point, models should be treated as statistically indistinguishable;
- Look at completion: Timeout rates determine whether agentic tasks can be embedded in real-time or near-real-time workflows;
- Look at cost: Capability improvement and lower per-task cost can occur at the same time;
- Look at subskills: Hallucination control, long-context use, precise instruction following, and agentic coding each have different strengths and should be matched to business scenarios;
- Look at yourself: Your own evaluation set and continuous regression testing are the ultimate basis for model selection.
For developers and enterprises, rather than chasing minor changes in leaderboard rankings, it is more valuable to invest in evaluation systems, layered routing, and cost observability. This is what enables teams to obtain stable, controllable, and cost-efficient technical capabilities in an environment where models evolve rapidly.
