How to Evaluate Advanced Mathematical Capabilities of Large Language Models

    This article explains why simple math-question accuracy is not enough for evaluating advanced mathematical capabilities in large language models. It proposes a practical evaluation framework based on asking a model to generate a self-contained lecture note that proves a proposition whose full proof is not publicly available. The framework covers task construction, output constraints, layered scoring, cost measurement, mutation testing, capability tiers, cautions about rankings and comparisons, and how to build a repeatable evaluation loop.

    1. Problem Background: Why We Need to Redefine “Mathematical Capability” Evaluation

    In the early stages of large language model adoption, judging whether a model was “good at math” often meant checking whether it could correctly solve a few science exam questions or competition problems. This kind of “answer accuracy” metric mainly reflects how well the model has learned common problem templates.

    However, once the use case shifts from “solving a problem with a standard answer” to “fully arguing a conclusion within a mathematical branch” or “writing a deliverable academic lecture note,” the original metric quickly becomes insufficient. There are three main reasons.

    1. The output is no longer a single value, but a structured proof.
      The correctness of a proof depends on a causal chain among reasoning steps. If any link in the chain breaks, the whole conclusion fails. In addition, the final result often does not have a simple “right” or “wrong” answer.

    2. Longer reasoning does not necessarily mean higher quality.
      A model with longer or more complex chain-of-thought reasoning is not always stronger. A lengthy derivation may simply be the result of repeated trial and error or circular self-reasoning. Using reasoning length as a proxy for capability may systematically overestimate such models.

    3. Knowledge boundary issues cannot be solved by ordinary retrieval alone.
      When a proposition is clearly stated in public mathematical materials, but its proof is not publicly available, or is only briefly sketched, the model must actually carry out the derivation instead of “finding the original text and copying it.” This type of task naturally resists memorization and retrieval-based shortcuts, making it suitable for testing genuine reasoning ability.

    Therefore, a more meaningful evaluation approach is to use “generating a deliverable academic lecture note” as the task carrier, and design the evaluation system around three dimensions: verifiability, completeness of reasoning structure, and awareness of knowledge boundaries. The following sections start from a concrete evaluation example and provide an actionable solution.

    2. Evaluation Example: Proving a Proposition Without a Full Public Proof and Writing It as a Lecture Note

    2.1 Task Construction Principles

    The key design point of this example is the selection of the proposition. The proposition should satisfy the following conditions:

    • Clear statement and meaning: The proposition itself should be a public mathematical fact. Readers and reviewers should be able to understand its meaning, avoiding an evaluation of “how well the task statement is written.”
    • Public sources lack a complete proof: Materials often provide only a conclusive statement. The proof may be omitted, reduced to a hint, or scattered across cross-references in multiple documents. The model must fill in the key steps itself.
    • Difficulty falls within the “hard reasoning” range: The task should require the use of multiple prerequisite lemmas across sections, the combination of several standard techniques, and a reasoning chain long enough to expose intermediate errors.
    • The conclusion can be independently verified: Through reverse construction, boundary-case checks, or inspection of key intermediate quantities, there should be a clear criterion for judging whether the result is correct or not.

    Note: The specific proposition is often chosen from a particular mathematical database or public lecture notes in a given branch. However, the selection criterion is transferable: any proposition with a clear statement but no publicly available full proof can be used as an evaluation item. It does not need to be tied to one specific source.

    2.2 Prompt and Output Constraint Design

    The task instruction should precisely control the output format. Otherwise, the evaluation results may contain a large amount of noise unrelated to mathematical ability. A typical instruction structure is as follows:

    1. [Task]
    2. Prove the proposition below and write the result as an English lecture note.
    3. [Proposition]
    4. <Insert the precise statement of the proposition here>
    5. [Constraints]
    6. 1. The output must be a self-contained lecture note that can be understood without external materials.
    7. 2. Use standard mathematical typesetting syntax so that it can be compiled directly into a document.
    8. 3. Explicitly state the notation definitions, assumptions, and context used in the proof.
    9. 4. Do not include omission phrases such as “this conclusion has already been proved,” “omitted here,” or “left as an exercise.”
    10. 5. If the proof requires additional assumptions or carries the risk of counterexamples, these situations must be discussed explicitly.
    11. [Output]
    12. The complete English lecture note text, plus a brief explanation of key steps.

    The purpose of these constraints is to separate “expressive ability” from “reasoning ability” as much as possible. The model cannot hide gaps by omitting reasoning steps, and reviewers do not need to guess what was left out.

    2.3 Scoring Rubric: From Format Compliance to Content Correctness

    The evaluation should not rely on a single total score. A layered rubric is recommended, where the criteria at each layer can be independently reviewed.

    Layer Criteria Check method
    L1 Format compliance Correct syntax, directly compilable, complete section structure Automatically check syntax trees and compilation results
    L2 Symbolic consistency Unique notation definitions, no duplicate definitions, no undefined symbols Manual reading plus keyword extraction
    L3 Reasoning completeness Every derivation step has a clear basis, with no logical jumps Manual item-by-item verification, or reverse verification with a third-party reasoning engine
    L4 Independent value The lecture note can be understood and reproduced by readers without reading the original source Blind reading feedback from domain readers

    In practice, the most common failure point is L3. A model-generated proof may look reasonable at every step, but it may contain conclusion skipping at key points. For example, it may assume a lemma holds without proving it, or quietly change a premise within a certain parameter range. Such errors are difficult to detect through simple keyword retrieval and must be verified manually, item by item.

    3. Cost Perspective: Measuring Resource Consumption per Unit of Reasoning Ability

    When building or calling inference services, cost is unavoidable. It is useful to break cost into measurable categories:

    • Input-side cost: The longer the prompt, the higher the cost per call. Instructions should therefore be structured and concise.
    • Output-side cost: The longer the lecture note, the higher the generation cost. If the evaluation goal includes generating a complete lecture note, resources must be budgeted for long outputs.
    • Retry cost: Because a single sample is random, a robust evaluation may need to run the task k times and take the majority result. This cost is approximately k times the cost of a single run.
    • Human review cost: This is the most easily overlooked category. L3 and L4 almost always require manual review or an additional reasoning engine, making the cost much higher than automated checks.

    A practical approach is “coarse screening plus fine evaluation.” First use automated checks, such as L1 and L2, together with a small amount of manual spot-checking, to remove obviously unqualified outputs. Then perform complete review only on samples that pass coarse screening.

    It is also possible to use “quota consumption” as a quantitative cost-side indicator, meaning the quota share required to complete the same lecture note under a fixed task. This indicator can reflect a model’s resource efficiency on a given task. However, it is affected by task difficulty, output length requirements, and sampling count, so it should not be treated as a cross-task comparable capability metric.

    4. Distinguishing Real Reasoning from Pattern Matching: Mutation Testing

    This is the most important, yet most frequently skipped, step in the entire evaluation process.

    Core idea: Given a proposition with a known answer, ask the model to produce a proof. Then systematically modify one condition of the proposition by strengthening it, weakening it, or replacing a parameter, and observe whether the model’s output changes reasonably.

    Three types of mutation can be designed:

    1. Condition-weakening mutation: Remove one assumption from the proposition.
      Correct behavior: Point out that the conclusion no longer holds and provide a counterexample.

    2. Parameter-replacement mutation: Replace a constant or exponent in the proposition with another value.
      Correct behavior: Adjust the key proof steps that depend on that parameter.

    3. Structurally equivalent rewrite mutation: Restate the proposition in an equivalent but differently worded form.
      Correct behavior: Recognize the equivalence rather than producing a proof that is almost identical to the original wording.

    If a model shows extremely high similarity in the third type of mutation but cannot adjust its conclusion in the first two types, it is more likely performing pattern matching than reasoning. The value of mutation testing is to separate “getting the right answer” from “being able to reason.” This is something that no single accuracy metric can achieve.

    5. Capability Tier Framework

    Combining the above dimensions, the advanced mathematical capabilities of large language models can be divided into four tiers. The purpose of layering is not ranking, but helping users decide which tier of capability is needed for a specific task.

    • Tier 1: Memory and Retrieval Tier
      The model can restate known theorems and cite standard conclusions. Its characteristic is fast response to familiar propositions, but failure when the wording changes.

    • Tier 2: Format and Expression Tier
      The model can output structured lecture notes according to conventions and use terminology basically accurately. Most models can reach this tier, so it is not a differentiating advantage.

    • Tier 3: Complete Derivation Tier
      The model can independently complete a logically closed proof, with intermediate steps that can be verified item by item and without relying on “looking up the original text.” This tier is the dividing line between “usable” and “highly usable.”

    • Tier 4: Knowledge Boundary Exploration Tier
      The model can propose testable conjectures, construct counterexamples, and point out missing premises in proofs when facing open problems or vaguely stated problems. Currently, few models demonstrate this tier, and performance is highly dependent on the specific problem domain.

    An important judgment is this: The gap between Tier 3 and Tier 4 is usually greater than the gap between Tier 2 and Tier 3. Therefore, when users complain that “the model is bad at math,” the issue is often not that the model fails to reach Tier 2, but that the task itself requires Tier 4 capability.

    6. How to Treat “Rankings” and “Comparisons”

    In public discussions, comparisons of model capability are easily distorted. There are four main reasons:

    1. Evaluation set contamination: Public questions may already be present in the training data. If the model has “seen the answer,” the ranking reflects memory rather than reasoning.
    2. Prompt sensitivity: The same model may perform differently under different instruction styles. The difference may even exceed the gap between models. Therefore, the conclusion that “Model A is stronger” depends heavily on prompt engineering.
    3. Sampling randomness: A single result is not enough to support a conclusion. Repeated sampling and statistical analysis are necessary.
    4. Incomparable task difficulty: Comparing “high accuracy on easy problems” with “medium accuracy on hard problems” can lead to misleading conclusions.

    A safer way to phrase the result is: “On this type of task and under these constraints, Model A performs better than Model B.” It is better to avoid making a global strength judgment. For users, the more meaningful question is not “which model is the strongest,” but “which tier of model is sufficient for my task and controllable in cost.”

    7. Practical Recommendations: How to Build Your Own Evaluation Loop

    1. Build a private evaluation set: Select propositions from public mathematical materials that have clear statements but lack publicly available full proofs. The number does not need to be large, around 20 to 30, but the set should cover different branches and difficulty levels.
    2. Fix the evaluation protocol: Keep the prompt template, sampling parameters, and scoring rubric fixed so that results from different evaluation rounds are comparable.
    3. Make mutation testing mandatory: Each task should have at least one mutation version. Use the mutation pass rate as a core metric.
    4. Retain raw outputs: Archive the model’s complete outputs, including reasoning processes, for later review and error attribution.
    5. Build an error taxonomy: Classify failures into categories such as “logical skipping,” “missing premise,” “symbol confusion,” “format non-compliance,” and “hallucinated citation.” Use the distribution of errors, rather than a single total score, to guide optimization.
    6. Perform regular regression testing: After a model version is updated, rerun the evaluation set to observe whether capability has degraded or improved. Avoid a “one-time model selection with no long-term re-evaluation” approach.

    8. Conclusion

    Evaluating the advanced mathematical capabilities of large language models should not stop at the level of “how many questions it got right.” A more reliable path is as follows: Use a task with anti-memorization and anti-retrieval properties as the carrier; use a layered scoring rubric to separate format compliance from content correctness; use mutation testing to separate real reasoning from pattern matching; and use cost metrics to constrain model selection. This method does not depend on any single product’s feature description, and it does not require unverified performance data. Any team can quickly build an evaluation loop based on its own tasks.

    Finally, one point must be made clear: there is currently no evidence that any model can reliably complete full solutions to open mathematical problems. Equating “being able to write a well-structured lecture note” with “having research-level mathematical ability” is a clear overestimation of current technical capabilities. For users, the correct expectation is this: at Tier 3, models can already serve as efficient draft-generation and structure-organization tools; at Tier 4, they are better used as assistants that propose candidate ideas rather than as authorities that provide final conclusions.