How to Systematically Evaluate LLMs on Advanced Mathematical Proofs

    Evaluating large language models on advanced mathematics requires more than checking whether the final answer looks complete. This article presents a structured framework for assessing proof understanding, lemma usage, reasoning gaps, counterexample handling, mathematical presentation, and self-correction. It also proposes a reproducible multi-stage testing workflow, explains why longer reasoning is not necessarily stronger reasoning, and distinguishes textbook-style proof tasks from open mathematical problems. The goal is to help researchers and developers build more objective, verifiable, and repeatable evaluations of LLM mathematical proof capabilities.

    1. Why a Complete-Looking Answer Is Not Enough

    When evaluating a large language model’s mathematical ability, especially its advanced mathematical ability, it is not enough to observe whether it can produce a proof that is structurally complete, notationally standard, and fluently written. For elementary calculation problems, it is often relatively easy to verify whether the answer is correct. In fields such as abstract algebra, algebraic geometry, topology, number theory, or category theory, however, a proof may contain many implicit assumptions. A single incorrect use of a definition, or a lemma applied outside its valid scope, can invalidate the entire argument.

    Therefore, evaluation of mathematical models should distinguish at least the following capabilities:

    1. Knowledge recall: Can the model accurately explain existing definitions, theorems, and common proof patterns?
    2. Proof reconstruction: Can it organize scattered known results into a complete proof based on provided material?
    3. Independent reasoning: Can it propose a new argument path when no ready-made proof is available?
    4. Formal expression: Can it use rigorous mathematical language, notation, and typesetting to generate readable text?
    5. Error self-checking: Can it identify gaps in its own proof and revise them proactively?
    6. Open-problem handling: When facing a problem without a standard answer, can it continue to explore rather than generate plausible-looking hallucinations?

    These dimensions are not equivalent. A model may be good at organizing lecture notes without being able to solve open problems independently. It may also produce a correct conclusion without providing a sufficiently rigorous proof.

    2. Using “From Source Material to Lecture Notes” as a Test Task

    A representative testing method is to give the model a mathematical proposition and ask it to consult or understand the relevant background, organize the proof into English or Chinese lecture notes, and use standard mathematical typesetting.

    The value of this kind of task is that it examines multiple capabilities at once:

    • Whether the model understands the objects, conditions, and conclusion in the proposition;
    • Whether it knows which definitions are essential to the proof;
    • Whether it can fill in derivations that are not directly provided in the source material;
    • Whether it can correctly handle dependencies among multiple lemmas;
    • Whether it can convert informal intuition into rigorous argument;
    • Whether it can make the final text suitable for teaching, reading, and review.

    Particular attention should be paid to the following: if the reference material only states a proposition without giving a proof, the content generated by the model cannot simply be classified as “retrieval followed by paraphrasing.” The model may be deriving the result from existing theoretical background, or it may be assembling a superficially reasonable argument driven by language patterns. The two look very similar, but their reliability is completely different.

    Therefore, testing should clearly distinguish among:

    • Proofs already present in the source material;
    • Proofs completed by the model based on definitions and related theorems;
    • New proofs constructed independently by the model;
    • Content that the model cannot confirm and generates by speculation.

    Only after a proof has been verified step by step can one judge whether the model has genuinely completed mathematical reasoning.

    3. Six Core Criteria for Judging Proof Quality

    3.1 Accuracy of Problem Understanding

    Before beginning a proof, the model must clearly state the objects and assumptions in the proposition. For example, if a conclusion depends on whether a space is compact, whether a ring is Noetherian, or whether a morphism is flat, the model cannot arbitrarily omit these conditions during the proof.

    A high-quality response usually completes the following work first:

    • Restate the proposition;
    • Explain key terms;
    • List known assumptions;
    • Identify the target to be proved;
    • Explain the overall proof strategy.

    If the model proceeds directly into derivations without clarifying symbol meanings and the scope of conditions, it is difficult to confirm the rigor of the subsequent content, even if it appears formally complete.

    3.2 Whether Lemmas Are Applied Under Valid Conditions

    The most common problem in mathematical proofs is not that a theorem itself is wrong, but that the theorem is used incorrectly. For example, a model may know that a certain result “usually holds” while ignoring additional requirements such as finiteness, continuity, invertibility, or completeness.

    When checking lemma usage, one should ask item by item:

    • Does the current object satisfy the premises of the lemma?
    • Does the conclusion of the lemma actually imply the next step?
    • Is a local property being mistaken for a global property?
    • Are isomorphism, homomorphism, embedding, and equivalence being confused?
    • Is an existence conclusion being miswritten as a uniqueness conclusion?

    This step usually requires the reader to have some professional background. It can also be assisted by formal verification tools or independent symbolic computation programs.

    3.3 Whether the Reasoning Chain Contains Gaps

    Advanced mathematical proofs allow obvious steps to be omitted, but “obvious” must be based on something the reader can reproduce. Common gaps in model-generated proofs include:

    • Jumping directly from a local conclusion to a global conclusion;
    • Inferring the existence of a target object directly from the existence of some object;
    • Writing “can be constructed” as “therefore holds”;
    • Omitting the definition of a key map;
    • Ignoring boundary cases;
    • Switching between structures at different levels without explanation.

    Evaluation should not focus only on proof length. A shorter proof in which every step is accurate is usually more reliable than a long proof that depends on many vague statements.

    3.4 Ability to Handle Counterexamples

    A model that truly understands a proposition should be able to answer “what happens if a certain condition is removed?” Therefore, testing can ask the model to:

    • Explain the role of each assumption;
    • Give a counterexample when that assumption is missing;
    • Explain whether the proposition can be generalized;
    • Point out which conditions are merely technical requirements;
    • Compare the scope of applicability of different proof paths.

    If the model can only repeat the original proof but cannot explain why the conditions are necessary, it may have mastered only a surface template without truly understanding the mathematical structure.

    3.5 Presentation and Typesetting Quality

    Mathematical lecture notes must not only be correct; they must also be easy to read and review. Excellent output should achieve the following:

    • Consistent definitions of notation;
    • Clear hierarchy among theorems, lemmas, and corollaries;
    • Standard formula typesetting;
    • Textual explanations for key steps;
    • A clear endpoint for the proof;
    • Sources or usage explanations when citing results;
    • No mixing of natural-language description and formal derivation without distinction.

    It should be emphasized that typesetting quality is only a readability indicator and cannot replace correctness. Fluent language, attractive structure, and neat formulas do not prove that the mathematical content has no flaws.

    3.6 Self-Correction Ability

    After the first-round answer, one can continue to ask the model targeted questions, such as:

    • Please check the proof you just gave line by line;
    • Which step is most likely to contain a gap?
    • Does your proof depend on undeclared conditions?
    • If that lemma cannot be used, please use another method;
    • Please provide an independent verification approach.

    If the model can clearly identify problems, explain the reasons for errors, and provide effective corrections, it demonstrates some reflective ability. Conversely, if it merely rephrases the original answer or uses longer text to obscure the problem, then effective self-checking cannot be considered complete.

    4. How to Design a Reproducible Testing Workflow

    To avoid evaluation results being affected by accidental factors, testing can be conducted in stages.

    Phase 1: Basic Understanding

    Provide a proposition with clear definitions and a moderate proof length, and ask the model to explain terms, restate the goal, and give a proof plan. This phase mainly judges whether the model accurately understands the problem rather than immediately pursuing the final answer.

    Phase 2: Complete Proof

    Ask the model to complete the argument without directly providing a ready-made proof, and to explicitly list the lemmas used. At this point, record:

    • Whether the first answer succeeds;
    • How many unexplained gaps appear in the proof;
    • Whether conceptual confusion occurs;
    • Whether nonexistent or inapplicable results are cited;
    • Output length and reasoning complexity.

    Phase 3: Adversarial Follow-Up

    Ask counter-questions about key parts of the proof, requiring the model to explain conditions, fill in details, or construct counterexamples. This step can effectively distinguish genuine understanding from surface generation.

    Phase 4: Independent Verification

    Split the proof into several local propositions and ask the model to reprove them separately, or hand them to another independent system for checking. For content with a higher degree of formalization, one can try to convert it into machine-checkable expressions. For abstract theory, professional reviewers need to examine it step by step.

    Phase 5: Repeated Testing

    Change the problem, prompt, and output language to observe whether the model remains stable. Success on a single problem cannot represent overall ability, and in particular, it cannot be used to assert that the model can handle open mathematical problems.

    5. How to Interpret “Longer Reasoning” Versus “Stronger Capability”

    In practical use, some models generate very long reasoning processes containing many attempts, backtracking, and self-explanations, while others can provide clear proofs in a shorter space. One cannot simply assume that the former is stronger.

    Reasoning length may reflect:

    • The problem itself is complex;
    • The model is exploring multiple directions;
    • The model lacks a stable strategy;
    • The model tends to unfold all intermediate steps;
    • The model is correcting earlier errors.

    A concise proof may reflect:

    • The model has mastered a better structured method;
    • It can quickly identify key lemmas;
    • It has stronger abstraction ability;
    • It may also simply have omitted necessary steps.

    Therefore, evaluation should shift from “how long the reasoning is” to “whether the reasoning is effective.” More reasonable indicators include:

    • Correctness rate;
    • First-pass rate;
    • Amount of human modification required for the proof;
    • Number of critical gaps;
    • Self-correction success rate;
    • Generalization ability when conditions change;
    • Reusability of the output.

    6. Differences Between Open Mathematical Problems and Textbook-Style Proofs

    Textbook-style proofs usually have clear goals, known theory, and relatively stable solution methods. A model can combine existing knowledge to produce a relatively high-quality explanation. Open problems often lack a standard path, and it may even be unknown whether the problem itself is true.

    When facing open problems, a model must meet higher requirements:

    1. Distinguish conjectures, theorems, and unproved intuition;
    2. Judge whether existing methods are applicable;
    3. Design new definitions, constructions, or induction schemes;
    4. Identify failed paths and explain why they fail;
    5. Evaluate whether results can be generalized;
    6. Avoid disguising exploratory text as a complete proof.

    Therefore, a model’s strong performance on theorem-to-lecture-note tasks does not mean it can independently solve frontier problems. There is a clear capability gap between the two. In practical evaluation, “knowledge-integration tasks” and “open-exploration tasks” should be established separately, and performance on the former cannot replace the latter.

    7. Practical Recommendations

    For researchers and developers, large language models are suitable for the following work:

    • Organizing existing proofs;
    • Explaining abstract definitions;
    • Generating teaching versions at different difficulty levels;
    • Identifying potential gaps in proofs;
    • Providing possible proof directions;
    • Converting natural language into standard mathematical typesetting;
    • Helping build test problem and counterexample libraries.

    However, caution is required in the following scenarios:

    • Claiming that an unsolved problem has been solved;
    • Directly using model output for the core proof of a paper;
    • Relying on the model to determine whether a theorem is true;
    • Publishing mathematical conclusions without professional review;
    • Treating long reasoning text as proof of reliability.

    A sound workflow is: first ask the model to propose an approach, then require it to expand step by step; next conduct counterexample testing, condition review, and independent verification; finally, have someone with relevant domain knowledge complete the confirmation. For high-risk content, the original prompts, model versions, output records, and human modification traces should also be retained for review.

    8. Conclusion

    The core of evaluating advanced mathematical models is not to compare whose text sounds more like an expert, nor whose reasoning process is longer. The key is whether the model can stably complete the full loop of “understanding the problem—constructing a proof—checking conditions—responding to challenges—correcting errors.”

    Based on current practical performance, large language models already have substantial practical value in mathematical knowledge integration, proof explanation, and formal writing. They are especially suitable for assisting with lecture notes, organizing research materials, and exploring proof directions. However, on genuinely open problems that lack standard answers, model reliability still depends on the type of problem, verification tools, and human review.

    For serious comparisons, it is advisable not to use only one problem and not to focus only on the final answer. Instead, a testing system should be built across multiple dimensions such as correctness, stability, generalization, verifiability, and correction ability. Only in this way can one obtain conclusions that are more objective and reproducible than “it feels stronger.”