How to Evaluate the Mathematical Reasoning Ability of Large Language Models: Methods and Practice

    Large language models are increasingly used for demanding work such as mathematical derivation, theorem proving, and algorithm correctness analysis—yet it is hard to tell whether a model can genuinely reason mathematically or is just recalling memorized answers. This article presents a systematic, reproducible methodology built around two ideas: problems that have no publicly available proof, and lecture-note-style long outputs that make errors hard to hide. It covers a five-tier capability scale (from basic computation to research-frontier conjectures), problem design, a five-dimension scoring rubric, a seven-step implementation workflow, prompt design, cost control, and turning evaluation into an automated pipeline. It also includes reusable prompt templates and a review checklist. The goal is to answer three questions about any model: can it do it, is it trustworthy, and is it worth the cost.

    Introduction: From Solving Problems to Constructing Proofs

    Large language models have been widely adopted in code generation, document summarization, data analysis, and many other scenarios. When they start taking on harder work—mathematical derivation, theorem proving, and algorithm correctness analysis—a key question arises: how do you judge what level a model’s mathematical reasoning ability is actually at?

    The intuitive answer is “give it more problems and see who gets more right.” But for complex reasoning tasks, this standard is far from enough. An answer may happen to be correct while the reasoning behind it is riddled with holes; a problem may simply reproduce publicly available material rather than involve genuine derivation.

    One test approach worth borrowing works like this: pick mathematical propositions that have no ready-made proof in public material, let the model derive them on its own, and then require it to produce a well-structured, detailed document. Such tasks carry high information density and make errors hard to conceal, which makes them a valuable observation window. But a single experience usually comes with considerable resource consumption, and the conclusions tend to be subjective. To turn a “personal impression” into a “reliable conclusion,” you need a systematic, reproducible evaluation methodology.

    This article presents a complete practical approach across six areas—task tiering, problem design, scoring systems, process standardization, cost control, and engineering implementation—and closes with ready-to-reuse prompt templates and a scoring checklist.


    1. Problem Definition: Why Correctness Count Is Not Enough

    Mathematical tasks fall into two broad categories, and evaluation must treat them differently:

    Dimension Closed-form tasks Open-ended tasks
    Typical form Calculations, proof exercises, contest problems Propositions without ready-made proofs, open problems
    Answer Unique, automatically comparable Requires designing a proof path yourself
    Focus Correctness, speed Strategy selection, rigor, verifiability
    Typical failure mode Wrong calculation “Right conclusion, wrong process,” circular reasoning, citing results that don’t exist
    Main risk The problem may already be “seen” by the model Hard to judge correctness, high evaluation cost

    Open-ended tasks best reflect the reasoning needs of real research and engineering. For these tasks, “getting it right” is only the minimum bar—whether the proof holds is the core criterion. So evaluation cannot stop at answer matching; it must also examine the following:

    1. Is the reasoning self-contained? Does the model merely restate existing conclusions, or does it give a complete derivation?
    2. Is the process rigorous? Is each step justified, or are there “obviously follows” leaps?
    3. Are the details reliable? Are domains, boundary conditions, and prerequisite assumptions left out?
    4. Is the presentation standardized? Is the notation consistent, and can others read, verify, and reuse the document?
    5. Is the cost controllable? How many resources does one successful derivation consume, and can it be reproduced stably?

    2. Building a Capability Scale: Distinguishing Real Reasoning from Strong Retrieval

    The biggest misconception in evaluation is mistaking memorization ability for reasoning ability. I recommend first building a tiered model, and having every evaluation result note which tier it belongs to, so as to avoid cross-tier comparison.

    Tier Task characteristics Focus Main confounders
    L1 Basic computation Arithmetic, algebraic simplification, solving equations Accuracy Long numbers, accumulated error across many steps
    L2 Standard proofs Textbook theorems, problems with standard solutions Correctness, method selection The answer may be recorded, making memory hard to distinguish from reasoning
    L3 Combinatorial reasoning Multiple knowledge points intersect, idea must be designed Decomposition ability, path search Tends to stall midway when hints are insufficient
    L4 Open proofs Propositions without ready-made proofs Rigor, strategy, long-chain consistency High evaluation cost, hard to judge correctness
    L5 Research frontier Open problems, conjectures Proposing new ideas and constructions Requires long-term tracking by domain experts

    This leads to several practical points:

    • A “good” score at L2 does not directly indicate L4 ability. Public problems are widespread in training data, and a correct answer may be recall rather than reasoning.
    • What is truly discriminative is the combined performance at L4 and each tier below it. L4 tests whether the model can construct an argument from scratch, and here “whether the strategy is viable” and “whether the execution is rigorous” should be viewed separately: a model may offer the right proof idea yet make frequent mistakes on details, which is a different problem from “completely unable to find an idea” and points to a different optimization direction.
    • Tier and version are the prerequisites for interpreting results. After a model version is updated, past conclusions may quickly become invalid, and tiers should not be simply ranked against each other. The evaluation system should support periodic regression testing.

    3. Problem Design: What Kind of Problems Are Worth Using

    The key to L4 evaluation lies in problem selection. A good evaluation problem should satisfy four conditions at once:

    1. No ready-made answer. The problem has no published solution, or its published solution is hard for the model to retrieve directly, which rules out memorization. You can use propositions from niche literature, or have domain staff write problems themselves.
    2. Moderate difficulty. It should be achievable within the current model’s capability yet still able to expose common errors. Problems that are too hard only yield “cannot judge,” while problems that are too easy have no discriminative power.
    3. Verifiable correctness. The results the reasoning relies on must themselves be rigorously proven, and human review must be able to judge right from wrong. Self-authored problems especially need this, otherwise you may get “the model’s answer happens to be wrong, but the reviewer thinks it is right.”
    4. Domain diversity. Cover algebra, analysis, number theory, combinatorics, and other directions, to avoid over-generalizing from one domain’s conclusions.

    Beyond forward problems, I recommend two kinds of control problems to calibrate review standards:

    • False-premise problems: deliberately give propositions that do not hold or have insufficient conditions, and observe whether the model blindly complies and forces a “proof,” or can identify the problem and provide a counterexample.
    • Equivalent-variant problems: change the phrasing or notational system to test whether the reasoning conclusion is stable.

    4. Evaluation Vehicle: Amplifying Information with Lecture-Note-Style Long Output

    Compared with “answering only right or wrong,” requiring the model to output a structured reasoning lecture carries far more information.

    First, long reasoning chains amplify errors. The more complex a proof, the higher the chance of an error. If the model still stays internally consistent across the whole document, that shows its ability to maintain consistency over a long context is strong.

    Second, the lecture structure makes item-by-item review easy. A qualified proof document usually contains:

    • Notation and conventions: unify symbol meanings to avoid front-to-back inconsistency.
    • Lemmas and facts: each citation notes its source; if a reliable source cannot be given, prove it rather than cite out of thin air.
    • Step-by-step derivation: every key step has a clear logical basis.
    • Boundary and degenerate cases: special cases are discussed separately, explaining why the general case can be reduced.
    • Conclusion: clearly state that the original proposition holds.

    Third, a lecture is naturally suited to evaluating writing ability. Mathematical writing is itself a professional skill; whether notation is standardized, whether the reasoning has clear layers, and whether readers can follow it through are all important parts of technical-document generation ability. Requiring the output in English also incidentally tests the model’s cross-language professional writing.

    Of course, lecture-style evaluation comes at a cost. The longer the derivation chain, the more resources it consumes. So it is best suited as a capability probe and an acceptance check, not a task to run in full on every regression round.


    5. Scoring System: Five Dimensions to Quantify a “Good Proof”

    To make scoring reproducible, I recommend an “admission gate + multi-dimensional scoring” approach. First set three admission conditions; failing any one judges that reasoning attempt as a failure, and it does not proceed to scoring:

    1. The final conclusion is correct;
    2. The key reasoning steps contain no substantive logical errors;
    3. The cited results genuinely exist and can be verified in authoritative sources.

    Reasoning that passes the gate is then scored on five dimensions (each with a maximum of 5 points):

    Dimension Points Core question Reference scoring points
    Correctness 5 Do the conclusion and each reasoning step hold? 5: no errors; 1: only a few correct fragments
    Rigor 5 Are boundaries and premises missing, any leaps? 5: key cases fully discussed; 1: key premises missing
    Verifiability 5 Is each citation traceable and reproducible? 5: source clear, location accurate; 1: many citations without sources
    Organization 5 Is the structure clear and the layering reasonable? 5: unified notation, clear layers; 1: chaotic structure
    Writing quality 5 Is the expression accurate and readable? 5: precise terminology, smooth prose; 1: frequent ambiguity and grammatical errors

    Weighting and summing the five dimensions lets you place evaluation results for different models and versions in the same coordinate system. For example, you can start with weights of 30%, 30%, 15%, 15%, and 10%, then adjust according to your team’s own priorities.

    Two points deserve emphasis:

    • Organization should be examined last. “Looking rigorous” is not the same as “being rigorous.” Logical correctness is the foundation of everything, and polished prose cannot make up for gaps in a proof.
    • Verifiability is the key to suppressing hallucination. Fabricating authoritative-looking citations is the most hidden and also the most dangerous failure mode in reasoning tasks.

    6. Implementation Process: From a Single Experience to Reproducible Experiments

    A persuasive evaluation usually contains seven steps:

    Step 1: Fix the experimental environment. Record the model name and version, generation parameters such as temperature, the system prompt and user prompt, the maximum output length for a single call, and the evaluation date. Silent model updates are a common cause of distorted evaluation conclusions; none of this information can be omitted.

    Step 2: Design problems and a baseline. Prepare several L4 problems plus control problems, and write human reference proofs in advance as a scoring baseline.

    Step 3: Control the number of samples. Under the same prompt, a large model’s output is random. I recommend sampling each problem independently multiple times (e.g., 3 to 5), and separately tracking the best and average performance.

    • “Can do it once” and “can do it stably” are two different abilities.
    • The variance of the success rate also matters: high variance means the cost of repeatedly investing in the same task will rise significantly.

    Step 4: Automatic pre-screening. Filter first with automated means beyond human review:

    • Whether the formula syntax parses correctly;
    • Whether the cited notation and terminology are consistent throughout;
    • Whether the output is truncated, and whether the argument chain is complete.

    Step 5: Expert review. Domain staff review the reasoning item by item. When multiple reviewers are involved, use blind review, meaning the reviewer does not know which model a result came from.

    Step 6: Cost recording. Record per problem the number of processing units consumed, response time, and number of retries, and compute the unit cost of success, i.e., “the average resource input needed to complete one qualified proof.”

    Step 7: Result archiving. Save the full inputs and outputs along with scoring details, supporting multi-dimensional retrospective comparison by model version, problem difficulty, and technical domain.


    7. Prompt Design: Making Requirements Executable and Checkable

    Prompt quality directly affects the comparability of evaluation results. I recommend including five parts: role setting, input content, task objective, output requirements, and quality requirements. The following template can be used as a reference:

    1. You are a rigorous mathematical researcher. Please complete the following task:
    2. [Input Proposition]
    3. {proposition}
    4. [Task Requirements]
    5. 1. First judge whether the proposition holds; if it does not, provide a counterexample or point out the error.
    6. 2. If the proposition holds, provide a complete, self-contained proof; you must not treat the proposition to be proved as known.
    7. 3. All external results you use must genuinely exist and be attributed to a source; for results whose source you cannot confirm, prove them yourself.
    8. [Output Requirements]
    9. 1. Write in English.
    10. 2. Organize according to the following structure:
    11. - Notation and Conventions
    12. - Auxiliary Results (with proofs)
    13. - Proof of the Proposition
    14. - Boundary Cases and Remarks
    15. 3. Define the meaning of each symbol at first use and keep it consistent throughout.
    16. [Quality Requirements]
    17. 1. Every key step must give an explicit reason; do not use hand-waving expressions like "obviously" or "clearly."
    18. 2. Prefer self-contained elementary arguments; when you must cite an external result, first give a verifiable source.
    19. 3. Discuss degenerate, limiting, and special-parameter cases separately.

    Prompt design has three key points:

    • Judge first, then prove. Having the model assess whether the proposition holds before starting can, to some extent, suppress blind compliance with false propositions.
    • Prohibit citations without sources. Writing the “verifiability” requirement directly into the instructions noticeably reduces the chance of fabricated sources.
    • Fix the output structure. A uniform structure lets different outputs be compared directly on the same scale, and also makes automatic pre-screening easier.

    8. Case Analysis: A Complete Breakdown of a “Proof-Free Proposition” Evaluation

    The following example uses an open-ended proof problem to show how each step lands in practice. Let T be a connected graph, and consider this proposition:

    Proposition: A connected graph T with n vertices and n−1 edges contains no cycle.

    (Note: a connected graph means there is a path between any two vertices, and a cycle means a closed path of length at least 3. This problem is illustrative and far easier than a real evaluation problem.)

    Reference argument: Use induction.

    • When n = 1: there are 0 edges, so no cycle can exist; the proposition holds.
    • Assume the proposition holds for all connected graphs with fewer than n vertices.
    • For any vertex v in T, first show deg(v) ≥ 1: otherwise T has only 1 vertex, contradicting n ≥ 2.
    • Then show T contains a vertex with deg(v) = 1: suppose every vertex has degree at least 2; then the sum of degrees is ≥ 2n, so the number of edges is ≥ n, contradicting the n−1 edges.
    • Take such a vertex v, whose only neighbor is u. After deleting v and the edge uv, the resulting graph T′ is connected with n−1 vertices and n−2 edges. By the induction hypothesis, T′ contains no cycle.
    • If T contained a cycle C, then since deg(v) = 1, we must have v ∉ C, so C would also be a cycle in T′, a contradiction. Therefore T contains no cycle.

    Automatic pre-screening can be implemented like this:

    1. def prescreen(doc: dict) -> dict:
    2. # 1) Notation: are all symbols appearing in the text defined?
    3. undefined = [s for s in doc["symbols"] if not doc["defined_symbols"].get(s)]
    4. # 2) Verifiable citations: is a source given, and is it on the whitelist?
    5. unverifiable = [c for c in doc["citations"] if not c.get("source")]
    6. unknown = [c for c in doc["citations"] if c.get("source") and c["source"] not in TRUSTED_SOURCES]
    7. # 3) Hand-waving: does it contain vague expressions that need review?
    8. handwave = [s for s in doc["sentences"] if any(w in s.lower() for w in ("obviously", "clearly", "trivially"))]
    9. # 4) Structural completeness: are all required sections present?
    10. missing_sections = [k for k in REQUIRED_SECTIONS if k not in doc["sections"]]
    11. return {
    12. "undefined_symbols": undefined,
    13. "unverifiable_citations": unverifiable,
    14. "unknown_sources": unknown,
    15. "handwave_steps": len(handwave),
    16. "missing_sections": missing_sections,
    17. }

    During review, judge in the following order:

    1. Judgment: is the conclusion consistent with the reference answer.
    2. Admission: are key steps such as the degree derivation and the connectivity after vertex deletion valid; is there circular reasoning, i.e., does it secretly use “trees contain no cycles” itself.
    3. Scoring: are boundary cases discussed, is the organization clear, is the expression accurate.
    4. Error-resistance comparison: if the model directly gives a wrong conclusion on the same problem yet can point out the counterexample “there exists a connected graph with 2 vertices and 1 edge that contains a cycle,” it shows the model has grasped the relevant concepts, and the failure may stem from a slip in expression or calculation rather than a misunderstanding. This kind of distinction is very helpful for localizing problems.

    Cost and success rate are the ultimate landing point of the evaluation. Suppose the model passes the admission gate in 2 of 5 samples, consuming about 60,000 processing units per call on average; then the unit cost of success is about 60,000 × 5 ÷ 2 = 150,000 processing units.

    If you introduce a two-stage strategy—first have the model briefly outline the proof idea, and require a full lecture only for answers whose idea is correct—resource consumption can drop substantially. But you must retain enough context between the two stages to prevent the outline from diverging from the expansion. For the final selection, weigh three metrics together—capability level, stability rate, and unit cost of success—rather than looking only at single-run performance.


    9. Cost Control: Making High-Value Tasks Sustainable

    Complex reasoning tasks often consume large amounts of resources on deep thinking and long-text output. A single trial may use only a small fraction of a monthly quota, but batch evaluation, multiple sampling, and cross-model comparison stacked together can easily blow the budget. I recommend the following measures:

    1. Tiered scheduling: use lower-tier models for routine tasks, and hand complex mathematical derivations and code reviews to higher-capability models, paired with human review.
    2. Budgeted sampling: set an upper limit on the number of samples for evaluation, and when it fails, adjust the prompt first rather than blindly adding samples.
    3. Reuse of intermediate results: write reusable lemmas and passed proofs into a knowledge base, and call them directly in later reasoning to avoid repeated derivation.
    4. Input trimming: remove redundant context to prevent long inputs from inflating cost and crowding out output space.
    5. Usage monitoring: set threshold alerts for metrics such as number of calls, resource consumption, and success rate, to catch anomalies in time.

    10. Engineering Implementation: Turning Evaluation into a Pipeline

    A single experience is hard to support long-term decisions. To keep evaluation running continuously, you can draw on the common cloud capabilities to build a pipeline:

    • Container platform: run reproducible evaluation tasks, fixing the code and runtime environment.
    • Message queue: deliver propositions into the evaluation queue one by one, smoothing peaks and filling valleys, handling failure retries, and avoiding large-scale evaluation from impacting online business.
    • Object storage: centrally store prompts, model outputs, review records, and version information, for easy auditing and traceability.
    • Log service: record each call’s parameters, duration, and resource consumption to locate anomalies.
    • Monitoring and alerting: track changes in success rate, unit cost of success, and average response time, and promptly trigger regression tests after a model version update.

    These are all common cloud architecture patterns in the industry. For example, on Baidu AI Cloud, users can combine products such as object storage, message service, container service, log service, and monitoring and alerting on demand to build an evaluation pipeline, avoiding repeated investment in infrastructure.

    The sample code for continuous evaluation is as follows (the model call is represented by a generic function chat):

    1. import hashlib
    2. import json
    3. SYSTEM_PROMPT = """You are a rigorous mathematical researcher."""
    4. PROMPT_TEMPLATE = """Please prove the following proposition and produce a structured proof document as required:
    5. Proposition: {proposition}
    6. Requirements: {requirements}"""
    7. def run_eval_batch(propositions, model, prompt_version, sample_times=3):
    8. results = []
    9. for p in propositions:
    10. user_prompt = PROMPT_TEMPLATE.format(
    11. proposition=p.text, requirements=p.requirements
    12. )
    13. # Derive a cache key from "prompt template + problem" to avoid re-investing on the same problem
    14. key = hashlib.sha256(
    15. (prompt_version + p.text).encode("utf-8")
    16. ).hexdigest()
    17. samples = []
    18. for _ in range(sample_times):
    19. doc = chat(model, system=SYSTEM_PROMPT, user=user_prompt)
    20. report = {
    21. "model": model.name,
    22. "model_version": model.version,
    23. "prompt_version": prompt_version,
    24. "difficulty": p.difficulty,
    25. "input_tokens": doc.usage.input_tokens,
    26. "output_tokens": doc.usage.output_tokens,
    27. "latency_ms": doc.latency_ms,
    28. "prescreen": prescreen(doc),
    29. "scores": {}, # to be filled by reviewers with the five-dimension scores
    30. "cache_key": key,
    31. "timestamp": doc.timestamp,
    32. }
    33. samples.append(report)
    34. results.append(samples)
    35. # Evaluation tasks can be written to logs and monitoring to track success rate and cost changes
    36. return results

    Once the pipeline is built, evaluation can be upgraded from a “personal experience” to an “organization-level capability”:

    • Regression is triggered automatically after each model version update;
    • Different teams share the same set of problems and scoring standards;
    • Evaluation data accumulates by domain and difficulty, gradually forming a capability profile suited to your own business.

    11. Common Pitfalls and Avoidance Checklist

    1. Drawing conclusions from single-run performance: reasoning results are random, so base conclusions on multiple samples.
    2. Looking only at whether the conclusion is right or wrong: you must review the whole process to guard against “right conclusion, wrong process.”
    3. Using public problems for open evaluation: this easily mistakes memory for reasoning.
    4. Using different prompts for different models: comparison loses fairness, so test under the same prompt system.
    5. No blind review, no unified standards: subjective bias can mask real gaps.
    6. Ignoring resource cost: high capability, if accompanied by high consumption and low stability, has equally limited practical value.
    7. Comparing across versions directly: after a model update, old conclusions need to be re-verified.

    Conclusion

    Evaluating a large model’s mathematical reasoning ability essentially answers three questions: can it do it, is it trustworthy, and is it worth it.

    • Can it do it: can it construct a complete proof when there is no ready-made answer.
    • Is it trustworthy: does every reasoning step hold up to scrutiny, and are citations genuine and traceable.
    • Is it worth it: how much resource cost does reaching an equivalent level of reasoning require.

    With “propositions without public proofs + lecture-note-style long output” as the core, multi-dimensional scoring as the yardstick, and a reproducible pipeline as the support, we can arrive at stable, objective, and actionable judgments in an environment of rapid model iteration—so that a large model’s mathematical ability can truly serve research and engineering practice.


    Appendix: Evaluation Prompt and Checklist Quick Reference

    Single-invocation reasoning prompt (to be used with the template above):

    1. As a rigorous mathematical researcher, prove the proposition: {proposition}
    2. Requirements:
    3. 1. First judge whether the proposition holds;
    4. 2. If it holds, provide a complete, self-contained proof;
    5. 3. Any external results you use must genuinely exist and be attributed to a source;
    6. 4. Write in English;
    7. 5. Output in the structure "Notation and Conventions / Auxiliary Results / Proof of the Proposition / Boundary Cases."

    Review checklist (tick off each item for every reasoning attempt):

    • Conclusion is correct
    • All key reasoning steps hold, with no circular reasoning
    • Citations genuinely exist and sources are verifiable
    • Boundary and special cases are discussed separately
    • Notation is uniformly defined and consistent throughout
    • No obvious hand-waving
    • Structure is complete and the output is not truncated
    • Resource consumption, response time, and retry count are recorded