Evaluating Chinese Reasoning Capabilities in LLMs: Methodology and Practice

    This article provides a practical methodology for evaluating the Chinese reasoning and text comprehension capabilities of large language models (LLMs). It defines a four-dimensional evaluation framework—accuracy, latency, token usage, and cost—and explains how to interpret version-to-version improvements and conduct fair multi-model comparisons. The content includes a decision matrix for model selection across different business scenarios, along with practical recommendations for building a repeatable evaluation pipeline.

    Why a Rigorous Chinese Evaluation Framework Is Needed

    Over the past year, major LLM vendors have released new versions at a rapid pace, with marketing focusing on terms like “strongest reasoning” and “natively multimodal.” For developers and enterprise users, the real question is not which model has the loudest campaign, but which one performs most consistently and cost-effectively in their specific business scenarios.

    This leads to an unavoidable topic: model capability evaluation. Scores on general-purpose leaderboards are often based on English corpora and standardized multiple-choice questions, which do not translate well to Chinese business contexts. The complex nesting of Chinese instructions, the precision required for industry-specific terminology, and the logical consistency demanded over long contexts are all dimensions that English-based benchmarks fail to cover. Therefore, building a rigorous evaluation set centered on Chinese—one that tests the underlying logic and linguistic capabilities of models with intentionally demanding questions—is a necessary prerequisite for model selection.

    It should be noted that leading flagship models today generally come with native multimodal capabilities, able to process large-scale video, audio, images, and code repositories simultaneously. However, on an evaluation set focused on Chinese text and logical reasoning, these cross-modal abilities only show a fraction of their potential. This article focuses precisely on this “lowest common denominator”—pure text and logic ability—because it serves as the foundation for all higher-level capabilities.

    Four Evaluation Dimensions: None Can Be Omitted

    A complete model evaluation should cover at least four key metrics, corresponding to accuracy, speed, efficiency, and cost.

    1. Accuracy

    Accuracy is the core of any evaluation, but overall accuracy must be broken down by domain to be meaningful. A reasonable set of categories typically includes:

    • Language understanding and instruction following: Tests the model’s ability to handle complex, verbose, and multi-constraint Chinese instructions. This dimension is often overlooked yet has the greatest impact on real-world user experience—the more colloquial the prompt and the more constraints it contains, the more visible the differences in instruction-following ability become.
    • Reasoning and mathematical computation: Tests multi-step logical deduction, numerical calculation, and formal reasoning. This is the most direct reflection of a model’s intellectual ceiling.
    • Vertical industry domains: Questions from education, finance, law and public administration, healthcare, and mental health test a model’s performance within the boundaries of specialized knowledge. Among these, healthcare questions demand extremely high factual accuracy and serve as a litmus test for a model’s hallucination control.

    2. Average Response Latency

    Measured as the average time per API call. For interactive applications, the difference between 53 seconds and 64 seconds means entirely different user experiences; for batch offline tasks, latency directly determines throughput. Notably, latency must be observed together with accuracy—a model that is fast only on easy questions has no practical value.

    3. Token Usage

    Measured as the average number of tokens consumed per call. Token usage reflects the model’s “solution path length”: significantly higher token consumption may indicate redundant self-verification, while abnormally low consumption may suggest the model is cutting corners. In one real-world test, a new model version averaged approximately 3,157 tokens per call, a marginal increase of about 1.2% compared to the previous version’s 3,119.

    4. Cost

    Typically expressed as the RMB cost per thousand calls. The calculation formula is:

    1. Single call cost = input tokens × input unit price + output tokens × output unit price
    2. Cost per thousand calls = single call cost × 1000

    Using real test data as an example: if each call consumes approximately 3,157 tokens (weighted across input and output), and the cost per thousand calls is approximately ¥250.5, then the single-call cost is about ¥0.25. For a business with millions of daily calls, this number multiplied by call volume represents the real monthly bill. Therefore, cost must be compared side by side with accuracy in the same table, rather than discussed in isolation.

    Interpreting Version Comparisons: Speed and Depth Improvements

    Comparing new and old versions of the same model on an identical evaluation set is the most direct way to verify vendor claims. In a typical version comparison, several key signals emerge:

    1. Overall Accuracy Gains

    The new version improved overall accuracy from 72.5% to 74.8%, a gain of 2.3 percentage points, moving its ranking from 6th to 2nd on the test leaderboard. At the high end of the score range, each percentage point of improvement is far harder to achieve than at the lower end, so a 2.3-point gain carries real significance.

    2. Structural Reading of Dimensional Gains

    What matters more than the aggregate gain is its structure:

    • Instruction following saw the largest improvement (+4.9%, reaching 72.4%), indicating that the new version’s main optimization focus was understanding complex instructions—typically the most direct output of engineering-driven fine-tuning investment.
    • Reasoning and mathematics improved by 3.3% to 85.1%, and this dimension was already at a top-tier level, representing a case of the strong getting stronger.
    • Education improved from 64.8% to 68.6% (+3.8%), the most significant gain among vertical domains, reflecting an expanded knowledge base and solution patterns.
    • Finance, law, and public administration saw modest gains, while healthcare and mental health remained at a high level of 88.7%—zero improvement on a high baseline is not necessarily negative; it may indicate that the dimension was already saturated or that the model’s conservative strategy has not changed.

    3. The “Anomalous” Efficiency Signal

    The most valuable finding in this comparison: despite significant gains in overall accuracy, average latency dropped from 64 seconds to 53 seconds, a reduction of about 17%, while token consumption increased by only 1.2%.

    This is an important architectural signal. The speed at which a model generates content is constrained by the efficiency of the underlying inference framework. Achieving nearly 20% faster generation with roughly the same output volume typically indicates substantive optimization at the level of the inference engine, parallel scheduling, or model sparsification—not simply more parameters. For self-hosted deployments or cost-sensitive businesses, this metric deserves no less weight than accuracy itself.

    Multi-Model Comparison: Three Methodological Essentials

    Running multiple leading models against the same evaluation set—often called a “showdown” evaluation—is the final step in model selection. To ensure trustworthy conclusions, three ground rules must be upheld:

    1. Unify Evaluation Conditions to Prevent Setup Differences

    All models under test must use identical questions, identical prompt templates, and identical decoding parameters (temperature, maximum output length, etc.). Any difference in these settings renders the comparison meaningless. It is recommended to version-control all prompt templates, system prompts, and sampling parameters to ensure reproducibility.

    2. Grade Objective and Subjective Questions Separately

    Reasoning and mathematics questions have unique correct answers and can be scored automatically. However, subjective questions in language understanding and education require a scoring rubric. A recommended approach is a hybrid method: “reference answer key-point matching + manual spot-check review,” with pre-established consistency checks (e.g., double-blind scoring with arbitration for disagreements) to prevent grading noise from masking genuine model differences.

    3. Beware the Average Score Trap: Look at Percentiles, Not Just Means

    An average latency of 53 seconds may conceal a long tail—some complex questions take several times the mean. It is recommended to record P50, P95, and P99 percentiles as well. Token consumption likewise merits percentile analysis. For production systems with service-level agreements (SLAs), P95 latency is often more decision-relevant than the mean.

    From Evaluation to Deployment: A Model Selection Framework

    The ultimate purpose of evaluation data is to support decisions. Based on the four dimensions above, a simple decision matrix can be constructed:

    Business Scenario Accuracy Weight Latency Weight Cost Weight Selection Tendency
    Real-time dialogue / customer service Medium Very High High Sacrifice marginal accuracy for lower latency
    Batch document analysis High Low High Choose best accuracy-to-cost ratio
    Education / healthcare assistance Very High Medium Medium Prioritize vertical-domain accuracy
    Code and logical reasoning Very High Medium Low Choose top-tier reasoning capability

    Two engineering recommendations follow from this framework. First, evaluation sets should include a proportion of actual in-house business questions—public benchmarks reflect general capabilities, while private questions reveal business fit. Second, establish a recurring re-evaluation cycle. Model iterations happen on a monthly basis; a selection decision made six months ago may already be completely outdated today.

    Conclusion

    LLM evaluation is not a one-time activity but an ongoing engineering discipline requiring methodological support. A well-designed Chinese evaluation set, four quantitative dimensions, and a reproducible comparison process together form the bridge from vendor claims to verifiable data. As model capabilities evolve rapidly, teams that master this evaluation methodology will gain a tangible advantage in technology selection and cost governance. For teams looking to lower the barrier to building their own evaluation pipelines, ready-made evaluation and model service offerings on cloud platforms can serve as a practical starting point for customizing an assessment workflow around their own business needs.