A Practical Guide to Evaluating LLM Math Reasoning: Task Design, Scoring, and Cost Measurement
Choosing a large language model usually comes down to how good it is at math—yet many evaluations rely on subjective impressions: run a few problems, see one impressive answer, and declare the model "strong" or "weak." This guide presents a practical, reproducible method for evaluating LLM math reasoning. It covers building a tiered task set that separates genuine reasoning from memorization, a five-dimension scoring system (correctness, completeness, rigor, readability, traceability), how to measure cost, latency, and stability—including the relationship between chain-of-thought length and accuracy—and an automated pipeline that turns subjective impressions into observable data. It closes with five practical principles for model selection.
Choosing a large language model (LLM) almost always comes down to one question: “How good is it at math?” Yet many evaluations stop at subjective impressions—run a few problems, see one answer that looks impressive, and conclude the model is “strong” or “weak.” Such conclusions rarely hold up. A single good result may come from luck, or from the model having memorized a similar solution in its training data. Meanwhile, the factors that actually shape day-to-day experience—cost, latency, writing quality, and rigor of reasoning—usually go unmeasured.
This article starts from a real experience of a subscriber. He picked a proposition from a public math reference repository that had no existing proof, gave it a one-line prompt, and received a complete English-language lecture note. The task consumed about 2% of his monthly quota. This shows that, without a reference proof, the model can produce math writing of nontrivial difficulty. But it also raises a series of questions that deserve careful answers: How should difficulty be defined? How should quality be judged? How should cost and consistency factor in? This article offers a practical, actionable evaluation method built around these questions.
1. Why We Need to Redesign Math Capability Evaluation
1.1 Common problem sets can’t separate reasoning from memory
Most math problem sets used in schools, competitions, and textbooks have long been public and come with standard solutions. They are very likely to appear in a model’s training data. A high score on these problems doesn’t necessarily prove real reasoning ability—the model may simply be reproducing a solution it has seen before. For that reason, an evaluation set should favor problems with scarce public solutions, or no ready-made solution at all.
1.2 Real needs are closer to “research-level” reasoning
Returning to the starting point: that one-line prompt has three parts. It targets a proposition in a public math reference repository whose entry states the claim but provides no proof; it asks the model to produce the proof independently and output it as an English lecture note. This closely matches many real work needs—understanding an argument glossed over in a paper, filling in a missing derivation in a technical design, or turning scattered conclusions into a clean document. These tasks test not memory but the ability to combine definitions, theorems, and reasoning chains.
1.3 “Solving hard problems” is only one dimension
Math capability is a composite concept with at least four layers:
- Symbolic manipulation and algebraic transformation: expanding, simplifying, and solving correctly;
- Logical reasoning: recognizing implicit assumptions, handling edge cases, avoiding circular arguments;
- Mathematical exposition: presenting the derivation clearly with proper language and formatting;
- Cost efficiency: how much compute and latency a usable result requires.
Testing only the first layer overestimates capability; testing only the fourth ignores the ceiling. A complete evaluation must cover all of these dimensions.
2. Step 1: Build a Tiered Evaluation Task Set
2.1 Difficulty tiering
Organize the evaluation set into three tiers, each serving a different diagnostic purpose:
| Tier | Typical characteristics | Primary diagnostic goal |
|---|---|---|
| Basic | Direct application of a definition or formula, clear steps | Symbolic accuracy, format compliance |
| Competition | Requires multi-step reasoning and technique, has a standard answer | Reasoning-chain length, strategy selection |
| Research-level | No existing proof in the reference repository; must construct a full argument | True reasoning ceiling, writing completeness, cost |
There is also a research-level format well suited to automated grading: give several conclusions, exactly one of which is false, and ask the model to identify the false one and provide a rigorous proof. These problems have a clear standard answer and can partly replace manual scoring. Still, the model’s answer should be spot-checked by humans, to guard against cases where it guesses the right conclusion without a valid proof.
2.2 Source selection and contamination control
Prefer two kinds of material:
- Propositions in a public math reference repository that lack a proof but already have an accepted solution—this gives a trustworthy reference answer while lowering the chance of a direct “recital”;
- Unpublished internal derivation problems, which have no data leakage by nature.
Note that a proposition lacking an existing proof in a public reference repository does not make it an open problem recognized by the math community. It only means that repository hasn’t collected a proof; the problem may well have been solved long ago. Conflating the two severely misjudges task difficulty.
If research-level problems are limited in number, expand the set through variable perturbation—changing parameter ranges, adjusting assumptions, or rephrasing the problem. But be careful: some propositions depend on specific parameter values, and careless changes may make the conclusion false or even turn the problem into a false statement. After perturbation, have someone with the relevant background review the result.
2.3 Output format and prompt design
During evaluation, the output format must match real requirements. Taking this article’s starting point as an example, generating an English lecture note requires checking:
- Whether terminology is translated accurately;
- Whether math typesetting is standard, including equation numbering, theorem environments, and citation format;
- Whether the structure is complete—definitions, lemmas, theorems, and proofs organized sensibly.
The prompt should describe “what is wanted,” not prescribe “how to do it” step by step. Overly granular prompts steer the model onto a path preset by the task author, so what you measure is “instruction-following ability” rather than independent reasoning. A reference prompt template:
Task: Prove Proposition [XXX] from the task card.Requirements:1. Complete the proof independently; do not cite or restate any published proof;2. Provide a complete derivation, with justification for each key step;3. Write a lecture note in English, including definitions, lemmas, theorems, and proofs;4. Output mathematical formulas using standard typesetting syntax;5. Self-check when done: any unproven lemmas, implicit assumptions, or unhandled edge cases?
2.4 Structured task cards
For batch execution and result traceability, structure each problem as a task card:
{"task_id": "math-l3-001","difficulty": "level3_research","language": "zh","prompt": "Prove Proposition [XXX] from the task card; provide a complete derivation...","requirements": ["independent", "English lecture note", "standard math typesetting", "traceable steps"],"forbidden": ["citing published proofs by others"],"rubric": ["correctness", "completeness", "rigor", "readability", "traceability"],"reference_answer": "Expert-curated standard proof or evaluation points","auto_check": ["final_answer_match", "formal_verify"],"budget_hint": "level3","provenance": "public_math_reference#entry-xxxx"}
The provenance field records where the problem came from, making it easier to later track whether there is a data contamination risk. The reference_answer and auto_check fields make explicit whether a problem can be graded automatically and by what method.
3. Step 2: Build a Multi-Dimensional Scoring System
The most common mistake in math reasoning evaluation is judging only whether the conclusion is right. A proof may reach a correct conclusion while hiding a circular argument; it may reach a wrong conclusion while showing solid local reasoning. Score on at least the following five dimensions, each on a 1–5 scale:
- Correctness: whether the final conclusion is correct; whether every step in the proof chain holds;
- Completeness: whether definitions, lemmas, and the main proof are all present; whether special cases and edge cases are covered;
- Rigor: whether there are implicit assumptions, skipped steps, or circular arguments; whether the conditions of the theorems used are met;
- Readability: whether the writing is clear, the structure sensible, the typesetting standard, and the terminology accurate;
- Traceability: whether each reasoning step can be traced back to a clear basis—a definition, a proved lemma, or a given condition.
The composite score can be computed as follows, with weights adjusted for the business context. For example, if the goal is mainly generating teaching material, raise the weight of readability:
Composite score = 0.40 × Correctness + 0.25 × Completeness + 0.20 × Rigor+ 0.10 × Readability + 0.05 × Traceability
For scoring execution, combine “automated checks + multiple reviewers”:
- Automated checks: For parts that can be verified mechanically—final numerical results, conclusions under specific conditions—write scripts to check in batch; where feasible, bring in formal-proof-checking tools to verify steps that can be formalized. It bears emphasis that natural-language proofs are hard to fully formalize, so automated checks can only serve as an aid.
- Cross review: Have at least two people with the relevant professional background independently blind-score each lecture note—neither can see the other’s scores. When scores diverge widely, a third person arbitrates.
- Consistent standards: Before formal evaluation, run a calibration round to align understanding of subjective dimensions like “rigor” and reduce inter-rater variance.
In addition, the rigor dimension should be tallied separately and not masked by the composite score. A high correctness score paired with chronically low rigor often signals a systematic flaw—for example, a habit of skipping key steps. Such flaws easily mislead in real use.
4. Step 3: Measure Cost and Output Efficiency
The user case above offers a useful perspective: the task consumed about 2% of the monthly quota. This suggests evaluation shouldn’t record only “whether it got it right,” but also “how much it cost.”
4.1 Cost composition of a single task
The compute for a math reasoning task comes mainly from three parts:
- The input prompt;
- The internal reasoning process (the chain of thought);
- The final output.
Many services bill input and output separately, and the reasoning process usually counts toward the output side. So the actual cost of a complex task may be far higher than intuition suggests. Evaluation scripts should record usage per run and estimate it as follows:
RUNS = 3PRICE_IN = 0.002 # Example price per 1K input tokens; replace with actual pricingPRICE_OUT = 0.008 # Example price per 1K output tokens (including chain-of-thought tokens)total_cost = 0.0for task in tasks:for i in range(RUNS):resp = client.chat(model=model_id, messages=[build_messages(task)])u = resp.usage # prompt_tokens / completion_tokens / total_tokenscost = u.prompt_tokens / 1000 * PRICE_IN + u.completion_tokens / 1000 * PRICE_OUTtotal_cost += coststorage.save(task.id, i, resp.text, cost) # Archive the output and cost together
Also record the proportions of prompt tokens, output tokens, and chain-of-thought tokens. This ratio helps judge whether the model is “overthinking” on a given task.
4.2 The relationship between chain-of-thought length and capability
In practice, people often equate “longer chain of thought” with “stronger capability,” which isn’t rigorous. Complex tasks do need more reasoning steps, but reasoning length isn’t always positively correlated with success rate. Some models have concise chains yet clear, effective arguments; others reason at length and are more prone to drifting off course through repeated second-guessing.
A more sensible approach is to plot a “chain-of-thought length vs. accuracy” scatter plot and observe the relationship:
- Moderate length with high accuracy indicates both efficiency and capability;
- Very long length without a matching accuracy gain suggests over-reasoning.
4.3 Latency and stability
In interactive settings, response time directly affects the experience. Evaluation should record separately:
- Time to first token;
- End-to-end time for full generation.
In addition, run the same task 3–5 times and measure the variance in success rate. A task that succeeds in only one run has limited reliability and is not enough to prove the model reliably possesses the capability.
5. The Evaluation Pipeline: Turning Subjective Feelings into Observable Data
A single manual evaluation is hard to scale. Build an automated pipeline with the following flow:
[Task bank] → [Inference run] → [Automated checks] → [Expert blind review] → [Metric aggregation] → [Evaluation report]↑ ↓ ↑[Log & usage collection] [Problem-sample feedback] [Dynamic difficulty calibration]
Responsibilities of each stage:
- Inference run: Call the model in batch per task card; record raw output, usage, and latency;
- Log collection: Write the full process of each call to a logging service, including the input prompt, model output, a chain-of-thought summary (if any), token usage, latency, and task ID. This is the base data for later retrospectives and cost attribution;
- Automated checks: Problems that can be graded automatically go straight into the statistics; those that can’t go into the expert queue;
- Expert blind review: Reviewers score against a consistent standard, with a third person arbitrating divergent samples;
- Metric aggregation: Compute success rates per tier, the distribution of composite scores, cost and latency distributions, and consistency across runs;
- Problem feedback and difficulty calibration: Samples judged incorrect feed back into the task bank for improvement; problems with too-high or too-low pass rates have their difficulty adjusted, keeping the set discriminative.
When implementing in the cloud, you can pick general-purpose capabilities by responsibility:
- Object storage to hold the task bank, model outputs, and evaluation reports;
- A container platform to host compute tasks like inference runs and automated checks, for elastic scaling on demand;
- A logging service to collect call records centrally and support task-level retrieval for retrospectives;
- Monitoring and alerting to set thresholds on metrics like single-run cost and call failure rate, and to notify promptly when spending crosses a certain percentage of the monthly quota.
For example, general products like Baidu AI Cloud’s object storage, container service, logging service, and cloud monitoring can serve as building blocks for such an evaluation pipeline. Building your own scripts works too—what matters is that the data is complete and traceable.
BUDGET_RATIO = 0.05 # A single evaluation task should stay under 5% of the monthly quotadef after_run(task, cost, monthly_quota):metrics.report("task_cost", cost, tags={"task_id": task.id, "difficulty": task.difficulty})if cost / monthly_quota > BUDGET_RATIO:alert.warn("Single-task cost exceeded the budget threshold", task_id=task.id, cost=cost)
6. Five Practical Principles for Model Selection
- Evaluate in tiers; don’t draw a conclusion from one problem. Cover at least three difficulty tiers, with enough problems per tier, and describe results with confidence intervals rather than a single number.
- The process matters as much as the result. A correct conclusion doesn’t equal a reliable proof; rigor and completeness belong in the required dimensions.
- Evaluate cost and capability together. Between two equally correct lecture notes, the one that costs less and runs faster has more engineering value.
- Tune for the scenario. Teaching settings value readability and style; research-support settings value rigor and traceability; batch-processing settings should prioritize cost and stability.
- Keep running regression evaluations. When the model version, prompt template, or business needs change, re-run the baseline evaluation set rather than “switching models on a hunch.”
7. Conclusion
Back to the opening question: that proposition with no existing proof is a good entry point for evaluating an LLM’s math ability. It has no standard answer to copy, so it requires the model to genuinely complete the whole process—from understanding the problem, to organizing an argument, to expressing it properly—and it clearly surfaces cost.
The point of evaluation isn’t to rank models by a simple label, but to build a reproducible, comparable, and continuously updatable method. When task design is sound, scoring dimensions are complete, and cost data is transparent, “which model is stronger at math” stops being a question answered by feel and becomes an engineering question you can discuss on data.
