Evaluating LLM Reasoning in Practice: From Metrics to Result Interpretation
This article walks through a rigorous LLM evaluation of roughly 15,000 Chinese-language test questions covering language understanding, instruction following, math reasoning, and vertical domains such as education, finance, law, and healthcare. It explains the four core metrics—accuracy, latency, token consumption, and cost—compares a preview release against its predecessor, and offers practical guidance on how developers should build their own evaluation baselines and interpret benchmark results for production model selection.
Why Systematic Model Evaluation Matters
Over the past year, flagship models from major vendors have iterated at an accelerating pace, and official capability claims have grown increasingly similar. For developers who need to select and deploy a model in production, launch-event numbers and demo videos are no longer sufficient to make a reliable decision. What actually matters is evaluation data produced under a unified test set, a unified invocation method, and a unified measurement methodology.
This article uses a rigorous evaluation focused on Chinese text and logical reasoning as an example, breaking down how a professional evaluation should be designed, executed, and interpreted. The subject of this evaluation was the preview version of a vendor’s flagship reasoning model, tested on roughly 15,000 questions spanning language understanding, instruction following, mathematical reasoning, and vertical domains including education, finance, law, and healthcare.
The Core Metric System
A rigorous LLM evaluation needs to observe four categories of metrics simultaneously—none of them is optional:
Accuracy (overall score): The most intuitive capability metric, reflecting the proportion of correct answers on the test set. But accuracy must be examined with dimensional breakdowns. A model scoring 74.8% overall might reach 88.7% on medical questions while only achieving 68.6% on education questions—vertical-domain variance is what actually drives selection decisions.
Average latency: The end-to-end response time per call. It directly affects user experience and system throughput ceilings, especially for complex reasoning tasks where longer chains of thought mean noticeably longer wait times.
Average token consumption: The number of tokens consumed per call, including input and output. Reasoning-oriented models tend to “think more to answer correctly,” so rising token consumption is a common price of improved capability.
Normalized cost: Token consumption combined with pricing, converted into the cost per thousand calls. This is the metric most easily overlooked during engineering deployment—and the one most likely to blow up later.
These four metrics form a “capability–efficiency–cost” triangle. A model that improves accuracy by 2 percentage points while doubling latency and tripling cost is not necessarily an upgrade in a real business scenario.
Comparing Iterative Versions
Placing the new preview version and its predecessor on the same test set reveals several signals worth noting:
A jump in overall ranking: Accuracy rose from 72.5% to 74.8%, a gain of 2.3 percentage points, moving the model from 6th to 2nd on the overall leaderboard. Note that score gaps among top-tier models are often only 1–3 percentage points, which means ranking changes are sensitive to statistical error and should be judged together with per-dimension performance.
Largest gains in language and instruction following (+4.9%, reaching 72.4%): This is the dimension most worth watching. The complexity of Chinese instructions lies in multi-condition constraints, implicit semantics, and formatting requirements. A substantial improvement here implies better stability in downstream scenarios such as agent tool calling and structured output.
Reasoning and mathematics reaching 85.1% (+3.3%): Mathematical reasoning is the litmus test of logical capability. A high score in this dimension usually predicts solid performance on code generation and multi-step planning tasks as well.
Diverging vertical-domain performance: Education rose from 64.8% to 68.6%, finance and law improved slightly, and healthcare held at a high 88.7%. Even for flagship models, capability ceilings differ markedly across domains—vertical-scenario selection cannot rely on the overall score alone.
Latency shortened against the trend: This is the most engineering-valuable signal in the comparison. Average latency dropped from 64 seconds to 53 seconds, while token consumption rose only marginally from 3,119 to 3,157. Lower latency with essentially flat token usage indicates a substantive optimization in underlying inference efficiency, rather than score gains bought by generating more content. For latency-sensitive applications, this change matters more in practice than the 2.3-point accuracy improvement.
Three Dimensions for Horizontal Comparison
When evaluating a model within the competitive landscape, three perspectives are recommended:
Capability: Compare overall and per-dimension leaderboards to identify the model’s strength zones (e.g., reasoning, healthcare) and weak zones (e.g., long-tail knowledge in education).
Efficiency: Compare latency and token consumption within the same accuracy tier. In practice, it is common to see models with near-identical scores but dramatically different efficiency—and efficiency advantages convert into significant cost advantages at scale.
Cost: Convert per-task cost using the output price (per million tokens). In this evaluation, the normalized cost was roughly 250 RMB per thousand calls; whether that is acceptable must be assessed against the business value per transaction.
Reflections on Evaluation Methodology
Multimodal capability is hard to cover with text-only evaluation: The value of natively multimodal models lies in processing video, audio, images, and large codebases simultaneously. A Chinese text evaluation can only probe the language and logic foundation; cross-modal capability requires purpose-built test sets.
Preview-version data has a shelf life: Between the preview and the official release, the model may continue to be tuned. Evaluation conclusions should always be labeled with the version number and test date, and should not be applied directly to the official release.
Beware of “teaching to the test” scores: If the evaluation set overlaps with the training data distribution, scores will be inflated. Cross-validating across multiple independent benchmarks and adding private business samples is a more reliable verification approach.
Selection Advice for Developers
- Run a POC first: Use real business data for small-scale comparative testing before consulting public leaderboards.
- Build your own evaluation baseline: Bring accuracy, latency, token consumption, and cost into continuous monitoring, and run regression verification with every version upgrade.
- Focus on effective intelligence per unit cost: Score divided by cost per call is closer to real value than the absolute score.
Model iteration will only accelerate. Building a reusable, comparable evaluation framework matters far more than chasing the score of any single release.
