Evaluating Coding and Reasoning in Chinese General-Purpose LLMs: A Practical Benchmarking Guide
A practical, two-layer framework for evaluating how general-purpose large language models evolve across generations, focused on coding and reasoning. The article covers a four-layer, six-dimension evaluation model spanning language coverage, context hallucination, long-chain reasoning, math accuracy, instruction following, and token efficiency; a generation-by-mode comparison method for isolating real gains; common evaluation pitfalls; and how to build a reproducible evaluation pipeline. Readers will learn how to measure improvement between versions, decide which gains translate into engineering value, and identify shortcomings to compensate for at the system-design level.
1. Introduction: Why Coding Became the Main Battleground for LLM Iteration
Over the past two years, the iteration pace of general-purpose LLMs in China has gone through roughly three phases:
- Broad rollout: Full-line coverage across general dialogue, multimodality, long context, and code. The product lines were complete, but no single capability stood out.
- Focused convergence: Consolidating resources on the general model, with agents and developer tooling as the main focus.
- Targeted breakthrough: Establishing a relative lead on coding—a high-value, strongly verifiable track—while improving inference efficiency at the same time.
This progression from “broad” to “specialized” and then toward “balanced” is fairly typical in the industry. The reason is that differences in general dialogue capabilities (everyday Q&A, text rewriting) are increasingly hard for ordinary users to perceive, whereas coding tasks have clear correctness standards—whether code compiles, whether tests pass, whether the logic holds—which makes them easier to turn into quantifiable reputation. Coding capability is also the foundation of agent capability, for three reasons:
- When an agent executes a task, it must generate code, call tools, read execution results, and iterate on fixes—the entire loop is essentially coding capability.
- The structured nature of code makes it easy to evaluate automatically, so model iteration can get fast feedback.
- Enterprise R&D scenarios show strong willingness to pay and clear demand, so model capability translates directly into business value.
For this reason, every version upgrade is worth evaluating systematically. This article uses the capability evolution of recent generations of Chinese general-purpose LLMs as an observation sample, and builds a two-layer evaluation method covering capability performance and inference efficiency, to help readers answer three questions:
- How much did the new model improve over the previous generation?
- Which improvements actually translate into engineering gains?
- Which shortcomings need to be compensated for at the system-design level?
2. The Iteration Trajectory: From “Multiple Parallel Versions” to “Single-Point Breakthrough”
Looking back at the evolution of recent generations of Chinese general-purpose LLMs, the shift in capability focus is clear:
| Version Stage | Capability Focus | Typical Characteristics |
|---|---|---|
| Early versions | Filling general-purpose gaps | Covering basics such as dialogue, knowledge Q&A, and text generation, still behind the leaders |
| Capability transition | General capability + agent positioning | Beginning to define agents as the core direction; tool calling and task planning start to matter |
| Coding deepening | Code and engineering capability | Forming a leading domestic advantage on tasks such as code generation and repair; benchmark scores reach the first tier |
| Capability leap | Performance and efficiency together | A clear jump over the previous generation, while substantially reducing token consumption |
The most notable change is the shift from “coding deepening” to “capability leap.” The former mostly involved local improvements—supplementing domain-specific data, adjusting dialogue strategy—with a relatively flat overall capability curve. By the capability-leap stage, a defining feature emerged: with deep thinking disabled, the model’s overall performance could approach the previous generation’s performance with deep thinking enabled. In other words, capabilities that the previous generation could only reach with “extra thinking” had become default capabilities in the new generation.
This shift matters a great deal for engineering practice. Deep thinking brings longer reasoning chains and higher token costs. If the new model already meets most business needs without thinking, then:
- Regular requests can disable deep thinking by default for lower latency and cost;
- Only complex tasks enable deep thinking on demand, forming a “tiered reasoning” calling strategy.
Note that the above is only an industry-level trend observation. Different models differ in capability focus and iteration pace, and the method described below is a general framework. Readers should verify against the actual model versions they use, using the methods in this article.
3. Methodology: Set the Baseline Before Talking About Improvement
The most common mistake when evaluating model version iterations is to focus only on “how many points the new version scored higher than the old one.” A more rigorous approach answers three preliminary questions first:
1. Are the comparison targets comparable?
The same model typically offers two inference modes:
- Non-reasoning mode: Outputs results directly, with low latency and low token consumption.
- Reasoning mode: Performs an internal reasoning step before giving an answer—well suited to complex problems, but with significantly higher token consumption and latency.
When comparing versions, the mode must be kept consistent. Going further, build a “generation × mode” four-quadrant comparison to observe how capability changes across modes:
| Comparison | Observation Focus | Engineering Implication |
|---|---|---|
| New (non-reasoning) vs old (non-reasoning) | Capability gains in the default mode | Affects the experience of the vast majority of online requests |
| New (reasoning) vs old (reasoning) | Capability ceiling on complex tasks | Affects hard-problem solving and agent tasks |
| New (non-reasoning) vs old (reasoning) | The starting point of the new generation’s “default capability” | Judging whether a low-cost mode can replace the old generation’s high-cost calls |
| New (reasoning) vs old (non-reasoning) | The incremental value of deep thinking | Judging when it’s worth enabling reasoning mode |
2. Are the evaluation conditions consistent?
If any of the following differ, the conclusions cannot be directly compared:
- Prompts: system prompts, wording of instructions, output format requirements;
- Decoding parameters: temperature, maximum generation length, thinking budget, and so on;
- Tool configuration: available tool set, tool descriptions, return result format;
- Execution environment: compiler version, dependency libraries, test framework, timeout limits.
Version your evaluation configuration and keep a record of every change.
3. Does it cover the real business distribution?
Public benchmark questions are well-structured with clear prompts, and differ substantially from the “dirty inputs” of real business (incomplete logs, mixed formats, implicit constraints). Use public benchmarks for horizontal comparison and a self-built business question set for vertical judgment; the two complement each other.
4. Evaluation Framework: Four Layers, Six Dimensions
A sound evaluation framework should include four layers:
- Capability layer: stratified by language, task type, and code size;
- Mode layer: distinguishing reasoning from non-reasoning;
- Metric layer: covering correctness, stability, controllability, and efficiency;
- Baseline layer: comparison against the previous generation, reference models in the same tier, and the model’s own historical versions.
| Evaluation Dimension | Core Focus | Key Metrics | Typical Question Types |
|---|---|---|---|
| Language coverage | Multi-language generation and repair | Per-language pass rate, syntax/compile error rate | Generation, completion, and repair across different programming languages |
| Context hallucination | Factual faithfulness in long context | Extraction accuracy, hallucination rate, missing-info detection rate | Log parsing, long-document extraction, cross-file information location |
| Long-chain reasoning | Capability ceiling on multi-step, high-dependency tasks | Extreme-question pass rate, process correctness, consistency across samples | Multi-file refactoring, complex algorithms, long-chain logical deduction |
| Math and calculation | Numerical reasoning and precision | Exact-match rate, relative error, share of calculation errors | Exact calculation, unit conversion, formula-based code implementation |
| Instruction following | Understanding and executing constraints | Format compliance rate, constraint satisfaction rate, out-of-scope edit rate | Conditional instructions, format limits, edit-scope constraints |
| Token efficiency | Output quality per unit cost | Average token consumption, share of effective output, score per unit cost | Full token accounting across all dimension questions |
Among these, language coverage answers “can the model handle everyday development,” context hallucination and long-chain reasoning answer “can the model handle complex scenarios in real engineering,” math and instruction following answer “is the output accurate and controllable,” and token efficiency answers “can the capability be deployed at scale at an affordable cost.” Together the six dimensions form a complete judgment of a version upgrade.
5. Dimension-by-Dimension Analysis: From “Answering Correctly” to “Being Dependable”
5.1 Language Coverage: Strengthening “Multilingual Fundamentals”
Language coverage evaluation focuses on the model’s generation and repair ability across different programming languages, including:
- Dynamic scripting languages: flexible syntax; the focus is on edge-case handling, exception catching, and readability.
- Statically typed languages: strong compile-time constraints; the focus is on type correctness, interface design, and compile pass rate.
- Functional languages: emphasizing immutability and composition; the focus is on higher-order function use and side-effect isolation.
- Declarative query and configuration languages: the focus is on syntactic accuracy and semantic soundness.
- Infrastructure-as-code languages: the focus is on resource dependencies and idempotency.
Two common approaches in practice:
- Balanced distribution: roughly equal question counts per language, making it easy to compare models’ language preferences horizontally.
- Business-aligned distribution: allocating question counts by the team’s real tech-stack proportions, closer to actual returns.
A preferable approach combines both: use a balanced set to spot language weak spots, and a business set to guide model selection and prompt optimization.
From the performance of recent generations of Chinese general-purpose LLMs, the improvement in coding capability shows a clear “holistic” character. On one key version, pass rates across the main language categories improved significantly over the previous generation, with no obvious language weak spots remaining, which reduces the value of per-language comparison with the previous generation. The evaluation focus should therefore shift to the capability-ceiling and stability issues discussed in the next section.
5.2 Context Hallucination: “Staying Faithful to the Source” in Long Context
Definition and harm
Context hallucination means the model introduces information that does not exist in the context, or misinterprets the context. It commonly appears in scenarios such as:
- Extracting error codes, request IDs, and timestamps from logs;
- Reading a long configuration file and then modifying a specified field;
- Locating a function or variable definition across multiple files;
- Extracting parameter types and default values from interface documentation.
Such errors often come wrapped in “correct formatting and a confident tone.” Without validation, wrong information flows directly into downstream systems, causing misjudgments or even production incidents.
Evaluation design
When designing context-hallucination questions, adopt the principle of “long input, precise answer, machine-decidable”:
- Long input: the context contains a large amount of irrelevant information (e.g., thousands of lines of logs), with target information scattered across the beginning, middle, and end, to test the “lost in the middle” problem;
- Distractors: add entries with similar format but wrong content (e.g., multiple error codes, several near-identical requests) to test whether the model can locate precisely;
- Missing items: deliberately omit a target piece of information to test whether the model recognizes “information missing” rather than fabricating an answer;
- Structured output: require results in a structured format for automatic grading.
An example question:
[Task] Extract the specified fields from the application logs below and output them as JSON.[Rules]1. Use only information present in the logs; fill "NOT_FOUND" for fields that do not appear.2. Copy error codes and request IDs character by character; do not rewrite them.3. The logs contain multiple request records; extract only the record whose REQUEST_ID is req-8842.4. Before outputting, re-check each value against the original log.[Logs](Several hundred lines of sample logs here, including the req-8842 record and severalnear-identical distractor records, none of which contain an upstream_timeout_ms field.)[Fields to extract] http_status, error_code, latency_ms, upstream_timeout_ms, REQUEST_ID
Scoring points
- Exact match: values must be identical to the source (case and units matter);
- Missing-info detection: missing fields must be marked as missing, not filled with an approximate value;
- Hallucination magnitude: if a wrong value is given, record its deviation from the true value to measure the “severity” of the error;
- Traceable citations: if the model is asked to also provide the source snippet or line number, this further checks whether its basis is genuine.
Improvement approaches
Drawing on the iteration experience of recent generations of Chinese models, hallucination control in long-context scenarios usually comes from three areas:
- Training side: strengthening long-context data and “faithful to the source” tasks, teaching the model to clearly express “cannot determine” when information is insufficient;
- Inference side: enabling reasoning mode so the model first locates and then re-checks, reducing the chance of answering from impression;
- Engineering side: combining model output with structured validation (regex checks, field whitelists, comparison against the raw log), forming a double safeguard of “model extraction + rule validation.”
It should be noted objectively that even models in the industry’s first tier are not necessarily the ones with the lowest hallucination rate. For example, in questions that require identifying a “target number,” some leading models still occasionally misread the question and use nonexistent values. This shows that context hallucination remains a widely acknowledged common challenge for large models today, and engineering must retain a validation step.
5.3 Long-Chain Reasoning: Determining “Peak Capability” and Stability
Problem definition
Long-chain reasoning tasks require the model to complete multiple consecutive reasoning steps before reaching the correct answer. Typical forms include:
- Multi-step code transformation: modifying interfaces across multiple files, adjusting call chains, and keeping behavior equivalent;
- Complex algorithm implementation: involving combinations of data structures, boundary conditions, and performance constraints;
- Multi-round debugging: tracing the root cause layer by layer from error messages and providing a fix;
- Long-chain logical deduction: puzzles, planning, or rule-matching problems that require step-by-step interlocking deduction.
What such tasks demand is not just “can it do it” but “is every step correct.” Any error along the reasoning chain can lead to a wrong final answer.
Evaluation design
- Stratify by chain length: divide questions into short, medium, long, and extra-long chains, and plot a “pass rate vs. chain length” curve;
- Verifiable process: for multi-step tasks, require the model to output intermediate steps, check them one by one, and locate the first faulty step;
- Multiple samples: sample the same question multiple times, tally pass rate and variance to measure stability;
- Difficulty gradient: use a reference model widely regarded as strong as a yardstick, and observe where the target model sits on the “pass line,” “average line,” and “limit line.”
Evaluation conclusion
From the iteration of recent generations of Chinese models, long-chain reasoning is the part of the coding-capability upgrade that most clearly separates models. The reasoning mode of some leading North American models has only a limited advantage on this dimension and can be seen as a “gatekeeper of the first tier”; recent Chinese models have “one foot in the door”—better performance on extreme questions, but insufficient stability, often losing slight points on occasional small issues and struggling to reliably earn a perfect score.
For this “straddling the threshold” level of capability, three engineering measures are recommended:
- Task decomposition: split extra-long tasks into several verifiable subtasks, shortening each reasoning chain and reducing the probability of error accumulation;
- Step-wise validation: after each step, verify with tests or static checks; if it fails, roll back and retry;
- Fault-tolerant retries: set a limited number of retries for key steps, and instruct the model in the prompt to first list an execution plan and then execute step by step.
An example is shown below:
[Task] Add a user credit-score feature to user-service.[Steps]1. Read the existing user entity and repository interfaces, and output the current call relationships.2. Provide a change list (files, functions, change type), and wait for confirmation before modifying.3. After finishing each file, run the relevant unit tests, and proceed only after they pass.4. Once everything is done, summarize the test results; if any fail, locate the first failing case and explain the cause.
5.4 Math and Calculation: Precision Determines “Reliability”
Math and calculation in coding tasks mainly appear in three kinds of scenarios:
- Numerical requirements: such as amount splitting, percentage calculation, and statistical aggregation;
- Algorithm complexity analysis: such as deriving time/space complexity;
- Unit and boundary handling: such as timestamp conversion, base conversion, and floating-point precision.
Many models “know how to do it but calculate the specific values wrong,” with typical symptoms such as:
- Carry errors in multi-digit multiplication;
- Missing operators in intermediate steps of long expressions;
- Precision loss when mixing decimals and fractions.
Evaluation design
- Exact match: when the answer is an integer or a finite decimal, require an exact match with the standard answer;
- Tolerance match: when the answer involves irrational numbers or complex decimals, set a reasonable relative-error threshold;
- Process scoring: for multi-step calculations, score by step, distinguishing “wrong approach” from “calculation error”;
- Edge cases: design questions that approach the limits of floating-point precision, or use extremely large/small values.
Improvement path and limits
Math and calculation capability is a direction that recent generations of Chinese models have focused on strengthening. After one key version made a large improvement in math capability, the precision problem on direct-calculation questions was basically resolved, and simple arithmetic could be done with high precision and no obvious error.
But it should be recognized that this capability depends heavily on targeted training data and varies widely across models—there are even cases where leading models have not specifically optimized this area. Therefore, in engineering practice, one should:
- Hand key calculations to programs or calculation tools, with the model handling modeling, expression, and logic organization;
- For sensitive values such as amounts and inventory, have business code perform the final calculation and validation;
- Treat “weak math” as an expected capability boundary rather than a defect in evaluation, and avoid the risk through architecture design.
5.5 Instruction Following: Delivering “As Required”
Instruction following determines whether the model can “deliver as agreed.” In coding scenarios, instructions fall into two categories:
- Direct instructions: clear, single requirements, such as “output only JSON,” “do not add comments,” “use camelCase naming”;
- Conditional instructions: requirements with conditional branches or priorities, such as “return a default value if the input is empty, otherwise throw a parameter exception,” or “prefer plan A; if its dependencies are unavailable, fall back to plan B.”
Evaluation design
- Machine-verifiable: turn instructions into machine-checkable rules, such as JSON parsing validation, regex matching, static checks, and unit tests;
- Constraint checklist: list all constraints for each question and judge each one, rather than looking only at the final result;
- Distractor design: place information in the context that conflicts with the instructions, to test whether the model can correctly identify priorities;
- Out-of-scope detection: check whether the model modified content outside the instruction’s scope.
An example question:
[Task] Read utils/format.js and fix the bug in it.[Constraints]1. Only utils/format.js may be modified; no other files may be changed.2. Keep the function signatures and export style unchanged.3. Do not add any third-party dependencies.4. After the fix, explain the root cause in one sentence; do not paste the full code.
Evaluation conclusion and engineering advice
The instruction-following capability of recent generations of Chinese models is no weaker than the benchmark models in the coding field, and is broadly at the same level:
- Conditional instructions: a slight edge in understanding conditional branches and priorities;
- Direct instructions: output-format stability is slightly behind, but this can be fully compensated with engineering measures such as structured output and fixed templates.
This means that in agent scenarios, Chinese models are fully capable of handling instruction-intensive steps such as tool calling, parameter construction, and result formatting. Engineering-wise, it is recommended to hard-code key constraints into the system prompt and use programs to “close the loop” with validation on the output.
6. Efficiency: Token Efficiency Is the “Last Mile” of Capability Deployment
6.1 Why Efficiency Matters as Much as Capability
No matter how strong a model’s capability is, if token consumption is too high, it is hard to deploy in large-scale scenarios. Reasoning mode trades longer internal reasoning for higher accuracy, but token consumption grows several-fold. Therefore, evaluating a version upgrade cannot look only at “scores”; it must also look at “cost per point.”
From the evolution of recent generations of Chinese models, inference efficiency has also made clear breakthroughs: in the capability-leap version, non-reasoning-mode token consumption dropped to about 40% of the previous generation.
The significance of this change lies in the fact that:
- The same budget can handle more requests, with a significant drop in unit cost;
- For teams or vendors with limited compute resources, token efficiency directly determines the ceiling of service capacity;
- Combined with the previous section’s conclusion, most tasks can be done in non-reasoning mode at lower cost, with reasoning mode enabled on demand only for complex tasks.
6.2 Building a “Quality-Cost” Dual-Axis Evaluation
It is recommended to plot a quality-cost curve for each evaluation:
- Horizontal axis: average total tokens per question (including input, reasoning, and output);
- Vertical axis: pass rate or composite score;
- Points: different versions and modes.
The option in the top-left corner (low consumption, high quality) is the current best cost-performance solution. In addition, record the following metrics:
| Metric | Definition | Purpose |
|---|---|---|
| Average total tokens | Sum of input + reasoning + output tokens | Measures resource cost per task |
| Share of effective output | Output tokens / total tokens | Measures whether tokens are used effectively |
| First-pass tokens | Average tokens of requests that pass on the first attempt | Balances first-attempt success rate and cost |
| Score per unit cost | Score / average total tokens | Compares cost-performance across versions and modes |
6.3 Engineering Optimization Strategies
Based on the composition of token consumption, you can optimize at all three ends—input, reasoning, and output—at the same time:
- Context trimming: inject only task-relevant code snippets and logs, use retrieval to locate relevant files, and avoid dumping the whole repository;
- Tool-result compression: truncate and structurally summarize the return results of tools such as search and log queries before handing them to the model;
- Tiered reasoning: use non-reasoning mode for simple tasks, and enable reasoning mode with a thinking-budget cap for complex tasks;
- Cache reuse: enable caching for stable, unchanging system prompts and code context to reduce the cost of repeated input;
- Output constraints: specify output format and length limits to reduce redundant explanation.
Example code:
def route_and_call(task, complexity):# Predict complexity from task features to decide the reasoning mode and resource budgetif complexity == "simple":return call_model(task, mode="no_think", max_tokens=2048)if complexity == "medium":return call_model(task, mode="think", reasoning_budget=2048, max_tokens=4096)return call_model(task, mode="think", reasoning_budget=8192, max_tokens=8192)def run_eval(cases):# Run evaluation cases in batch and aggregate quality and cost metricstotal_tokens = 0passed = 0for case in cases:result = route_and_call(case.prompt, case.complexity)total_tokens += result.usage.total_tokensif case.verifier(result.output):passed += 1quality = passed / len(cases)cost = total_tokens / len(cases)return {"pass_rate": quality, "avg_total_tokens": cost, "quality_cost_ratio": quality / cost}
7. Summary of Evaluation Dimensions
Putting the six dimensions together gives a fairly complete judgment of a version upgrade:
| Dimension | Typical Performance in the Key Version | Engineering Recommendation |
|---|---|---|
| Language coverage | Broad improvement across languages vs. the previous generation; no obvious weak spots | Can adopt directly per the business tech stack |
| Context hallucination | Reasoning mode suppresses it better; log parsing is stable; but non-reasoning mode still trails individual leading models | Keep structured validation; enable reasoning mode when necessary |
| Long-chain reasoning | Extreme performance approaches the first tier; stability slightly lacking | Task decomposition + step-wise validation + limited retries |
| Math and calculation | Precision problem basically resolved; high precision and no error on simple calculations | Have programs compute and validate key values |
| Instruction following | At the same level as coding benchmark models; slightly better on conditional instructions | Hard-code constraints in the system prompt; validate output at the exit |
| Token efficiency | Non-reasoning-mode token consumption dropped significantly | Tiered reasoning, context trimming, cache reuse |
Overall, this round of upgrades shows the feature of “capability improvement and cost reduction happening at the same time,” which is the fundamental reason its engineering value is higher than that of earlier minor-version iterations.
8. Common Evaluation Pitfalls
- Looking only at the total score: the total score masks differences across languages and task types; insist on stratified evaluation;
- Mixing old and new modes: directly comparing new non-reasoning with old reasoning easily distorts conclusions;
- Stale questions: public questions may be covered by training data; use a self-built business question set as primary and public benchmarks as a supplement;
- Looking only at the final answer: ignoring intermediate steps and output format makes it impossible to tell whether the problem lies in capability or instruction following;
- One-off testing: models keep iterating and prompts also change; establish a periodic regression mechanism;
- Ignoring cost: pursuing only the highest accuracy while ignoring token consumption and latency makes large-scale deployment difficult.
9. Implementation: Build a Reproducible Evaluation Pipeline
It is recommended to consolidate the above methods into an automated evaluation pipeline, with an overall flow that includes:
- Question-set management: tag by language, task type, difficulty, chain length, and complexity; version-manage them;
- Sandbox execution: run generated code in isolated containers, execute unit tests and static checks, set timeouts and resource limits, and prevent malicious or infinite-loop code from affecting the host;
- Metric collection: uniformly record pass rate, hallucination rate, constraint satisfaction rate, token consumption, and latency;
- Report output: generate stratified reports and quality-cost curves by dimension, supporting vertical comparison across versions;
- Regression alerts: automatically alert when a key metric falls below the baseline threshold, triggering manual review.
Example code:
def evaluate(model, cases):report = {"by_language": {}, "by_mode": {}}for case in cases:result = model.run(case.prompt, mode=case.mode)metrics = {"passed": case.verifier(result.output),"constraint_satisfaction": case.check_constraints(result.output),"total_tokens": result.usage.total_tokens,"hallucination": case.check_facts(result.output, case.context),}for key in ("by_language", "by_mode"):bucket = report[key].setdefault(getattr(case, key), [])bucket.append(metrics)return summarize(report)
10. Conclusion
Chinese general-purpose LLMs are undergoing a clear evolution from “broad rollout” to “general focus” and then to “targeted coding breakthrough.” Recent capability-leap versions have reached the previous generation’s reasoning-mode level in non-reasoning mode, and are in the same tier as the leading benchmarks in the coding field on long-context hallucination suppression, long-chain reasoning extreme performance, math precision, and instruction following, while token consumption has dropped substantially. For developers, this means they can apply large models to scenarios such as code generation, log analysis, and agent refactoring at lower cost.
At the same time, common shortcomings—insufficient long-chain reasoning stability and hallucinations that still exist in certain scenarios—have not been fully eliminated. In system integration, one should hold to a dual-track approach of “capability evaluation + engineering safeguards”:
- On one hand, use a layered, mode-segmented, reproducible evaluation system to continuously track changes in model capability;
- On the other hand, use task decomposition, step-wise validation, structured output, and rule validation to reliably convert model capability into business value.
Only by combining “evaluating capability” with “designing systems” can one consistently make sound selection decisions amid the fast pace of model iteration, turning each version upgrade into genuine certainty in engineering gains.
