LLM Evaluation Guide: How to Scientifically Measure Mathematical Reasoning
Head-to-head comparisons of large language models based on a few ad hoc prompts and subjective impressions are unreliable and risky to use for production decisions. This guide presents a reproducible, statistically sound, and iterative workflow for judging a model's ability on a specific task, using math-heavy reasoning as the main example while also covering writing quality. It explains why informal comparisons fail (sample bias, randomness, version drift, prompt effects, and evaluator bias), how to decompose reasoning into measurable sub-skills, and the four pillars of rigorous math evaluation: layered and contamination-resistant problem sets, robust answer matching beyond exact string matching, correct statistics (pass@1/pass@k with confidence intervals), and controlled variables for fair, budget-matched comparison. It then covers evaluating writing and explanation quality, turning results into model-selection decisions, and integrating evaluation into the development lifecycle.
“Model A’s reasoning flagship is worse than Model B’s flagship,” “A new model has already caught up with the previous generation’s top configuration,” “The long chain-of-thought version of a model is actually weaker”… Arguments like these appear with almost every model release. Such conclusions may not be wrong, but the vast majority come from a handful of trial answers on scattered problems plus subjective impressions. The moment you adopt a model as a formal capability in a product, relying on this kind of comparison for selection carries real risk: differences on hard problems may come from sampling variance, version changes can invalidate earlier conclusions, and the evaluator’s focus can sway the results. So this article does not try to answer “which model is stronger”; it answers “how to design a reproducible, statistically sound, and iterative evaluation process to scientifically judge a model’s capability on a specific task.” Using reasoning-intensive math problems as the main thread and touching on writing capability, it offers an engineering-ready approach.
1. Why informal head-to-head comparisons fail
LLM evaluation has several systematic biases that either amplify or mask the real differences between models.
- Sample bias: Evaluators tend to pick a few “representative” problems from memory, which may fall within the capabilities they happen to be familiar with. Without controlling the distribution of difficulty and problem type, the conclusions cannot be generalized.
- Randomness: Sampling temperature, context length, and whether chain-of-thought mode is enabled all affect the same model’s performance; the same model can also produce different answers across runs. A single answer has no statistical significance.
- Version drift: After a model is updated, its capability shifts overall or changes structurally. The strength ranking of the previous generation may not hold for the new one, so evaluation conclusions must be recorded together with the evaluation date and the model version.
- Prompt effects: For the same problem, asking for “the final answer” versus asking to “derive step by step and self-check” can yield completely different results. When comparing across models, if prompts and output budgets are inconsistent, you are comparing the evaluation method rather than the model’s capability.
- Evaluator bias: The person judging “who is stronger” tends to focus on the dimensions they value, such as derivation rigor or expression style, so the same set of results can lead to different conclusions.
The first step in any engineering decision is to fix the evaluation goal, the problem set, the judgment rules, and the statistical methodology, so that conclusions are repeatable and auditable.
2. Decompose the capability: what is actually tested on math problems
Mathematical reasoning is not a single metric. It can be broken down into at least the following sub-capabilities:
| Capability dimension | Typical behavior | Main evaluation method |
|---|---|---|
| Problem understanding | Accurately extract conditions and constraints, identify implicit assumptions | Manual checks against the canonical interpretation |
| Mathematical modeling | Translate a natural-language problem into equations, inequalities, or a probabilistic model | Check whether the model’s formulation is equivalent and whether any constraint is missing |
| Reasoning planning | Choose a solution path, handle case analysis and boundary conditions | Step-by-step verification of the derivation |
| Computation | Symbolic and numerical computation, avoiding large-number and precision errors | Symbolic equivalence checking, numerical comparison |
| Self-verification | Substitute back to verify, check degenerate cases, catch one’s own errors | Stepwise error localization |
| Expression organization | Clear steps, explicit conclusion, proper formatting | Human scoring or score-model assistance |
The accuracy of mathematical reasoning depends largely on how well each layer is evaluated. Once decomposed, you can see that different models’ weaknesses sit in different layers: some models are stably correct on easy problems but have a high error rate on hard ones, which is a weak execution layer; others give correct answers but skip derivation steps or rely on unstated assumptions, which is a weak planning layer. Locating the weakness is far more useful for decisions than handing out a blanket ranking.
3. Four key aspects of math reasoning evaluation
3.1 Problem set design: layered, contamination-resistant, traceable
The problem set determines the credibility range of the conclusions. Follow three principles:
Layered sampling. Group problems by difficulty, for example basic word problems, standard competition problems, high-difficulty problems that require constructive techniques, and open-ended research problems (those with no public canonical answer). Compute the pass rate for each layer separately, so you can see whether a model’s ability improves uniformly or only leads in certain tiers.
Contamination resistance. If problems and answers already appear widely in public corpora, the model may be “recalling” rather than “reasoning.” You can reduce this influence with:
- Rewriting problems into equivalent variants (swap numbers, reorder the narrative, change the context);
- Using self-built private problem sets with the same difficulty distribution as public ones;
- Testing multiple variants of the same problem and checking whether the pass rate drops significantly.
Traceable metadata. Annotate each problem with its source, difficulty tier, canonical answer and equivalent answer forms, and the applicable scoring rule. Keep the problem set under version control, so any conclusion can be traced back to a specific set of problems.
3.2 Judgment rules: exact matching is almost unusable
Math answers exist in many equivalent forms: 1/2, 0.5, and 50% are equivalent; x=2 and the solution set {2} are equivalent; a simplified trigonometric expression may look completely different. Judging with exact string matching systematically underestimates a model’s capability. Engineering practice usually combines “rule-based normalization + symbolic verification + numerical tolerance comparison,” as illustrated below:
import refrom sympy import simplify, sympify, sqrtfrom sympy.parsing.sympy_parser import (parse_expr, standard_transformations,implicit_multiplication_application,)def normalize(ans: str) -> str:"""Extract the final answer and apply basic normalization"""m = re.search(r"\\boxed\{(.+)\}", ans, flags=re.S)# Extract the explicitly marked final answertext = m.group(1) if m else anstext = text.strip().rstrip(".")text = re.sub(r"\\(?:left|right|,|!| )", "", text) # Remove LaTeX layout commandstext = text.replace("\\dfrac", "\\frac").replace("^{\\circ}", "")text = re.sub(r"\s+", "", text)text = text.replace("%", "/100") # Convert percentage to decimalreturn textdef to_expr(text: str):"""Parse text into a computable symbolic expression"""transformations = standard_transformations + (implicit_multiplication_application, # Support notations like 2x, 3(x+1))return parse_expr(text.replace("^", "**"), transformations=transformations)def is_correct(pred: str, gold: str, tol: float = 1e-6) -> bool:"""Numerical + symbolic double check"""a, b = normalize(pred), normalize(gold)if a == b:return True # Fast pathtry:ea, eb = to_expr(a), to_expr(b)if simplify(ea - eb) == 0: # Symbolic equivalence: 1/2 and 0.5return Truediff = abs(complex(ea) - complex(eb))scale = max(1.0, abs(complex(eb)))return diff <= tol * scale # Numerical approximation toleranceexcept Exception:return False
Two practical notes. First, sets, intervals, and multi-valued answers need dedicated judges (element-wise matching, interval endpoint comparison) rather than a single-expression approach. Second, the numerical tolerance should be set per problem type: integer answers require exact equality, irrational or trigonometric values allow relative error, and geometry or statistics problems must account for measurement precision. The strictness of the judgment rules must align with the business need: educational problem-solving scenarios are stricter about the process, while numerical-query scenarios can focus more on the final value.
3.3 Statistical methodology: distinguish single-attempt from at-least-one-of-many
LLM output is stochastic, so evaluation is inherently a statistical problem. Two common metrics:
- pass@1: the expected probability that a single sample is correct;
- pass@k: the probability that at least one of k samples is correct.
They answer different questions. The former measures the reliability of “getting it right on the first try,” which directly matches the user experience of chat products. The latter measures the model’s “potential capability ceiling,” which is closer to search-style generation, sampling ensembles, or multi-round self-correction. Naively estimating pass@k across multiple samples is biased; the paper “Evaluating Large Language Models Trained on Code” gives an unbiased estimator: if a problem is sampled n times with c correct answers, then
pass@k = 1 - C(n-c, k) / C(n, k)
Example implementation:
from math import combdef pass_at_k(n: int, c: int, k: int) -> float:"""Unbiased estimate: with c correct out of n samples, the probability of at least one correct"""if n - c < k:return 1.0return 1.0 - comb(n - c, k) / comb(n, k)
When the number of problems is small, the per-problem pass rate fluctuates heavily. In practice, report the overall mean together with a confidence interval; example:
import mathfrom statistics import NormalDistdef wilson_interval(successes: int, total: int, z: float = 1.96):"""Wilson confidence interval, more robust than the normal approximation"""if total == 0:return (0.0, 0.0)p = successes / totaldenom = 1 + z ** 2 / totalcenter = (p + z ** 2 / (2 * total)) / denomhalf = z * math.sqrt(p * (1 - p) / total + z ** 2 / (4 * total ** 2)) / denomreturn (max(0.0, center - half), min(1.0, center + half))
Note that the confidence interval narrows as the sample size grows, but math problem types vary widely; even a narrow interval only means the conclusion holds on the chosen problem set. When the sample size is insufficient, a gap of a few percentage points between two models may fall entirely within noise.
3.4 Control variables: fair comparison under the same budget
Different models have different default output lengths, different defaults for whether chain-of-thought is on, and different thinking-budget policies. To compare “math reasoning capability,” you must explicitly fix or report:
- Whether chain-of-thought is enabled, and its maximum length or thinking budget;
- Sampling temperature and number of samples;
- Prompt template and answer-format requirements (for example, whether a “final answer” label is required);
- Whether external tools (such as symbolic math tools) are allowed.
A more realistic view is that capability comparison belongs on the axis of “accuracy per unit of reasoning budget.” A model that outputs more or thinks more elaborately is not necessarily more capable; it may just be using longer derivations to mask uncertainty. So it is worth also computing “accuracy / average output length” or “accuracy / average latency,” putting “capability” and “cost” side by side in the same table.
4. Evaluating writing and expression
Math evaluation answers “right or wrong,” but in real business a model also handles writing, summarizing, and explaining, tasks that need to be “clear, readable, and human-like.” These are hard to judge by exact matching, so use:
- Blind review: Hide the model’s identity, randomly order the candidate outputs, and have reviewers score them, avoiding brand impressions and order bias;
- Structured rubric: Break the total score into observable dimensions, score each independently, and attach brief comments. Example:
Scoring dimensions (1-5 each):1. Accuracy: any factual errors or task-irrelevant content?2. Completeness: does it cover all required points of the task?3. Structural clarity: clear hierarchy; can readers grasp the key points quickly?4. Language naturalness: awkward wording, or template-like output?5. Redundancy: repetition, filler, or content unrelated to the conclusion?
- Make metrics concrete: “A simpler chain-of-thought, more human-like expression” is essentially an observation of output style. In evaluation, convert it into operable metrics such as “the output length needed to complete the same task,” “the redundancy-dimension score in the rubric,” or “the share of reviewers preferring naturalness,” rather than staying at the level of impressions.
- Combine score-model assistance with human spot checks: A score model for large-scale screening can significantly cut cost, but score models have their own biases (for example, favoring longer, more formatted answers). Before going live, calibrate the score model with human-labeled data and periodically re-check by sampling.
5. From results to selection decisions: map capability to scenarios
Conclusions should not stop at leaderboard-style rankings; they must answer “can I use this model in my business?” You can turn evaluation into a selection decision in these steps:
- Define the business-critical metrics: for example, first-attempt accuracy, average response latency, cost per request, and format-compliance rate;
- Weight by task distribution: when simple tasks dominate the business, the simple-problem pass rate naturally carries more weight; if the business heavily involves high-difficulty reasoning, focus on the high-difficulty tiers;
- Define capability ranges: Map the results to a “reliable usage range for this class of tasks.” For example, if a candidate’s pass rate on basic and standard difficulty is significantly higher than the alternatives, position it as “the preferred candidate for routine reasoning and problem-solving scenarios”; if it clearly lags on open-ended hard problems, it is not suitable for such scenarios;
- Continuous regression: After a model update, rerun the same problem set to observe regressions or structural changes, avoiding “decisions based on last time’s impression.”
This method also applies to evaluating in-house models: first establish a baseline problem set, then validate improvements with the same statistical methodology.
6. Integrating evaluation into the R&D process
Evaluation is not a one-time activity; it should be a fixed part of the R&D process. Recommended practices:
- Fix the baseline: Put the problem set, judgment rules, statistical methodology, and prompt templates under version control, and log every change;
- Layered execution: Use a lightweight set for daily smoke tests, and run the full set before a release;
- Archive results: Save each evaluation’s model version, parameter configuration, raw outputs, and judgment results, so disputed samples can be traced back;
- Review anomaly samples: Manually review samples judged incorrect or borderline, which both validates the judgment rules and surfaces prompt and output-format issues;
- Sustainable integration: Package evaluation as a callable service or script; model inference can be done via model services on cloud platforms such as Baidu AI Cloud. Decoupling the evaluation service from specific models makes it easy to switch and compare across different models and versions.
7. Conclusion
Back to the original argument: judgments about model strength never converge if they rely on impressions and scattered trials. The engineering-reliable approach is to break “which is stronger” into “under a given problem distribution, given judgment rules, and given budget constraints, which model has higher first-attempt accuracy, and at what cost.” The methods here can be summed up in four sentences: layered problems, strict judgment, statistical multi-sampling, and controlled-variable comparison. Once these steps are fixed, model selection turns from subjective impression into a reproducible engineering experiment, and capability weaknesses and applicable boundaries become clear.
