How to Systematically Evaluate LLMs' Mathematical Reasoning and Lecture Note Generation Capabilities

    This article explores a systematic approach to evaluating large language models (LLMs) for mathematical reasoning and lecture note generation. It highlights the critical distinction between surface-level text quality and actual proof correctness, categorizes mathematical tasks by complexity, and outlines a correctness-centric evaluation framework. Furthermore, it introduces a phased workflow for generating verifiable lecture notes, a reusable task prompt template, cost measurement strategies based on effective delivery, and guidelines for building a long-term, reliable evaluation dataset.

    I. From “Writing a Proof” to “Proving It Actually Holds”

    When evaluating the mathematical capabilities of large language models (LLMs), the most common pitfall is equating expression quality with reasoning quality. If a lecture note maintains consistent notation, clear structure, fluent language, and includes several seemingly appropriate lemmas, it often leaves a strong professional impression. However, whether a mathematical proof holds depends entirely on the validity of each key derivation, not on whether the text reads like a textbook.

    A representative application scenario is when a user selects a proposition from public mathematical materials that lacks a proof, asks the model to complete the proof, and formats it into an English lecture note. Such tasks simultaneously involve proposition comprehension, background knowledge organization, proof route selection, and technical writing, making them much closer to real-world research and teaching needs than simply answering computational questions.

    However, the fact that “the material page lacks a proof” only indicates that the model needs to provide the missing argument; it does not directly mean it has performed original research. The conclusion may already have a proof in other textbooks or papers, or it might be a direct corollary of a classic theorem. Even if the model does not invoke external retrieval, one cannot rule out the possibility of it utilizing existing knowledge based solely on its output.

    Therefore, a reasonable evaluation should be broken down into three questions:

    1. Did the model correctly understand the proposition and its validity conditions?
    2. Did the model provide a complete proof sufficient for independent verification?
    3. Did the model organize the proof into a lecture note suitable for the target audience?

    These three layers are interrelated but cannot substitute for one another.

    II. Distinguish Task Levels Before Discussing Capabilities

    The difficulty of mathematical tasks is not a simple linear progression. A short prompt may contain profound obstacles, while a long output might merely expand routine calculations. Therefore, task types must be clarified before evaluation.

    1. Standard Knowledge Explanation

    This category includes explaining definitions, illustrating theorem intuitions, demonstrating basic examples, and supplementing common derivations. Evaluation focuses on conceptual accuracy, complete premises, and the ability to avoid confusing similar concepts.
    For example, explaining the difference between continuity and uniform continuity requires more than restating definitions; it must explain how the order of quantifiers affects the conclusion and demonstrate the difference through appropriate examples.

    2. Proof Completion for Known Propositions

    The task provides a valid proposition and requires the model to derive the conclusion from explicit premises. Evaluation focuses on whether the proof is closed, whether citations are applicable, and whether hidden conditions exist.
    “Proof omitted” sections in public materials are suitable for testing this capability, but the prompt must provide sufficient context. Otherwise, the model might argue based on its familiar version of a definition, while the actual material uses a different convention.

    3. Complex Derivation and Auxiliary Lemma Construction

    These tasks require the model to break down the goal, design intermediate propositions, and handle multi-layer dependencies. This better reflects reasoning organization skills but is also more prone to “locally reasonable, globally invalid” situations.
    For instance, an auxiliary lemma might be flawless but require a condition that has not yet been established; if subsequent steps use the main conclusion to retroactively satisfy that condition, it constitutes circular reasoning.

    4. Exploratory Mathematical Research

    Exploratory tasks may include finding counterexamples, modifying conjectures, studying special cases, and proposing proof routes. Here, a valid model output is not necessarily a complete answer; it could be a verified local conclusion or a clear elimination of an erroneous route.

    5. Open Problem Solving

    Standards for open problems must be exceptionally strict. Generating an apparently complete proof cannot be considered as having solved the problem. For genuinely unsolved problems, independent review by experts in the relevant field is required; when applicable, formal verification tools can be added, though these tools also rely on correct proposition expression and assumption encoding.
    Therefore, a few successful cases are insufficient to support the conclusion that “a certain class of models can stably solve open problems.” A more prudent statement is: the model may provide valuable candidate ideas for the research process, but these candidates must undergo independent verification.

    III. Establish a Correctness-Centric Evaluation Framework

    A single total score easily masks severe issues. No matter how clearly a proof is written, if a critical step fails, it should not receive an “excellent” rating based on formatting and language scores. A more appropriate method is to set a correctness threshold first, then evaluate other dimensions.

    1. Proposition and Assumption Understanding

    First, check whether the model accurately retains the research object, quantifiers, and boundary conditions, paying special attention to:

    • Miswriting “there exists” as “for all”;
    • Treating a sufficient condition as a necessary one;
    • Arbitrarily adding assumptions like finiteness, commutativity, compactness, or regularity;
    • Ignoring empty sets, zero objects, or degenerate cases;
    • Confusing local properties with global properties.
      If the model proves a weaker proposition by adding extra conditions, this must be explicitly noted and cannot be treated as if the original proposition has been proven.

    2. Proof Correctness and Closure

    During review, dependency relationships should be established for key steps: what assumptions, definitions, lemmas, or theorems does this step use? Are these bases already established?
    Particularly be wary of phrases like “obviously,” “easily obtained,” or “by standard results.” They are not inherently wrong, but they often mask the very steps that most need explanation.
    A practical principle: after deleting all decorative language, the remaining logical chain should still be able to support the conclusion.

    3. Verifiability of Theorem Citations

    A model might remember the general form of a conclusion but omit its applicable conditions. Therefore, one should not just accept theorem names; the model should be required to write out the specific version used in this instance.
    For example, if a step involves exchanging limits and integrals, it must specify under what convergence conditions. One cannot assume the exchange is legal simply because the expression forms look similar.
    Unverifiable citations should be marked as dependencies to be verified, not automatically accepted as facts.

    4. Pedagogical Quality of Lecture Notes

    Once the proof passes the correctness check, evaluate whether the lecture note is suitable for reading. High-quality lecture notes should typically include:

    • Necessary definitions and notation conventions;
    • Complete statement of the proposition;
    • Explanation of proof strategy and key obstacles;
    • Layered, rigorous proof;
    • Examples or counterexamples demonstrating the role of assumptions;
    • Appropriate summary and extension discussions.
      Whether the English expression is natural and the mathematical formatting is standard falls under this layer. They are important but cannot replace mathematical correctness.

    5. Stability and Reproducibility

    A single success might be influenced by the model’s familiarity with the problem, context length, and generation randomness. Evaluation should use multiple problems and record the results of each attempt.
    Observable metrics include first-generation pass rate, number of manual revisions, number of critical errors, and whether the conclusion remains consistent after altering the phrasing. For the same problem, you can also ask for different proof routes, but multiple similar answers do not equal multiple independent verifications, as they might share the same erroneous premise.

    IV. Why Longer Output Doesn’t Mean Stronger Reasoning

    Lengthy derivations easily create the impression of “thorough thinking,” but output length has no necessary correlation with correctness. A model might repeatedly restate the same idea in a long text, or continuously introduce new lemmas, complicating the dependency graph.

    Three factors deserve more attention during evaluation:

    First, is the critical path clear? Readers should be able to quickly identify the main thread of the proof, rather than searching for the core argument among numerous tangents.

    Second, are the auxiliary conclusions necessary? Every new lemma increases the verification burden; if it doesn’t reduce the complexity of the main proof, it might just be shifting the difficulty.

    Third, is the final argument compressible? A mature proof usually forms a relatively concise logical skeleton. “Concise” here doesn’t mean omitting critical steps, but removing repetition and ineffective exploration.

    Therefore, for evaluation, models should be required to submit verifiable proofs, bases, and uncertainties, rather than treating lengthy exploratory text as proof of capability.

    V. Break Lecture Note Generation into a Verifiable Engineering Workflow

    Asking for “a proof written as an English lecture note” in one go is convenient but mixes reasoning with writing. A more reliable process is phased delivery.

    Phase 1: Fix Task Boundaries

    The input should include the complete proposition, necessary definitions, allowed background conclusions, and the knowledge level of the target audience. If a definition comes from a specific source, it’s best to provide its text directly rather than just a reference number.
    Also state: if the proposition lacks conditions, point out the issue first; if the proof cannot be completed, clearly report the obstacle, and do not fill the gap with vague wording.

    Phase 2: Generate Proof Skeleton

    First, require the output of the main steps, auxiliary lemmas, and dependencies, without rushing to polish. The goal of this phase is to discover early whether the route holds, avoiding investing heavy writing costs in the wrong direction.

    Phase 3: Expand Rigorous Proof

    After the route passes preliminary review, expand it step by step. For every external conclusion invoked, require checking the premises; for special cases and boundary conditions, check separately.

    Phase 4: Conduct Adversarial Review

    The review task should not just be “checking for errors,” but more specific:

    • Look for the steps most likely to fail;
    • Try to construct examples that satisfy the assumptions but not the conclusion;
    • Check if the conclusion to be proven is used prematurely;
    • Delete assumptions one by one and observe at which step the proof fails;
    • Check whether the mappings, domains, and codomains between objects match.
      Model self-checks can only serve as aids, as generation and review might share the same blind spots. Critical conclusions should still be independently verified by someone with the appropriate background.

    Phase 5: Format as Lecture Notes

    Only after the core proof passes review should the English rewriting and formatting take place. During rewriting, the proposition and mathematical expressions should be locked to avoid altering quantifiers, conditions, or conclusion strength for the sake of linguistic fluency.
    After completion, cross-check against the original proof, not just checking grammar.

    VI. A Reusable Task Specification Template

    The following template is suitable for proof completion and lecture note writing:

    Please complete the task based on the following proposition and definitions.
    First, check if the proposition statement is complete, listing potential ambiguities and missing conditions.
    Second, propose a proof route, listing the required auxiliary lemmas and their dependencies.
    Third, provide a step-by-step verifiable rigorous proof, explicitly stating the applicable conditions for each external conclusion.
    Fourth, attempt to find counterexamples, circular reasoning, and boundary cases.
    Fifth, after the proof passes inspection, format it into an English lecture note, including definitions, the proposition, proof strategy, complete proof, and necessary explanations.
    If you cannot complete a step, clearly mark the unresolved parts; do not write guesses as proven facts.

    The value of this template lies not in guaranteeing success, but in making failures visible and problems locatable. For complex mathematical tasks, being able to clearly report “which step is not yet proven” is usually more useful than generating a seemingly complete but unverifiable answer.

    VII. Measure Cost by Effective Delivery, Not Just Token Consumption

    Consuming fewer tokens in a task indicates limited resource usage for that interaction, but it is insufficient to infer overall cost-effectiveness. Different tasks vary greatly in input length, generation length, retry counts, and manual review costs, and subscription quotas may not directly convert to per-task costs.

    A more reasonable statistical metric is:

    Unit Cost of Qualified Delivery = Total Task-Related Cost ÷ Number of Deliveries Passing Acceptance.

    Here, total cost should include model invocation, material preparation, manual verification, revisions, and repeated formatting. For high-difficulty mathematical content, expert review is often a non-negligible part.

    Additionally, stopping conditions can be set by phase: if proposition understanding is already wrong, immediately correct the input; if the proof route has fundamental flaws, pause lecture note generation; if it’s just an expression issue, proceed to text polishing. This is more conducive to controlling rework than repeatedly generating the entire content.

    VIII. Build a Small, Reliable Evaluation Dataset for Long-Term Use

    Individual researchers or technical teams can maintain a task collection covering different difficulties, including basic explanations, proof completion, erroneous proof diagnosis, counterexample construction, and exploratory problems.

    Each problem should save the proposition version, necessary background, reference proof or acceptance criteria, common errors, and manual judgment records. For exploratory problems, rather than simply marking “success” or “failure,” record what verified new conclusions were generated, what routes were eliminated, and what obstacles remain.

    The evaluation should also include a certain number of false propositions and propositions with insufficient conditions. Testing only true and provable conclusions tends to reward the model’s inclination to “give a proof regardless.” Adding invalid tasks allows observation of whether it refuses false premises and seeks counterexamples.

    IX. Conclusion: The Core of Reliability is the Verification Loop

    The practical value of LLMs in mathematical work includes assisting derivation, as well as knowledge organization, proof expression, and lecture note drafting. However, any single brilliant output should not be directly elevated to an overall capability ranking, let alone replace the verification of specific proofs.

    A professional usage approach is to break the task into five phases: proposition understanding, route design, rigorous proof, independent review, and lecture note delivery, setting clear acceptance criteria for each phase.

    What is truly worth relying on is not the performance of “always providing an answer,” but the ability to provide verifiable bases when correct, accurately expose gaps when uncertain, and keep manual verification costs controllable. Only by establishing such a closed loop can mathematical reasoning assistance transition from an impressive demo to a continuously usable research and teaching workflow.