How to Evaluate the Mathematical Proof Capabilities of General-Purpose LLMs
Judging whether a general-purpose large language model can handle real mathematical work requires more than noticing fluent answers or attractive formatting. This article presents an evaluation framework built around one demanding task: asking a model to independently prove a mathematical statement that has no publicly available standard proof, and to write it up in the style of English mathematical lecture notes. It covers how to set evaluation objectives, design reproducible prompts, account for cost, run adversarial verification, compare models across dimensions, and turn one-off successes into a reliable production pipeline.
To judge whether a general-purpose large language model has reliable mathematical ability, you cannot stop at surface impressions such as “fluent answers,” “clean formatting,” or “looks like it can write proofs.” A far more revealing test is this: give the model a mathematical statement for which only the claim itself is publicly known—no standard proof exists—and ask it to complete the proof independently, written up as English mathematical lecture notes. This task simultaneously tests conceptual understanding, reasoning and planning, lemma organization, formal expression, notational consistency, and cost discipline. Throughout evaluation, treat the model as a “candidate proof generator,” not an authoritative source; trustworthiness must come from a verifiable chain of logic, not from the model’s confident tone.
1. Define the Evaluation Goal First: Generation Is Only the Starting Point—A Valid Proof Is What Counts
The biggest difference between mathematical tasks and ordinary Q&A is that correctness is strongly structural. A proof can be elegant and complete in appearance while silently swapping a condition at a critical step; it can also be plainly written yet logically airtight. Evaluation should therefore be broken into four layers:
- Parsing the problem: Does the model correctly identify the objects, quantifiers, morphisms, limit conditions, and local-versus-global properties in the statement?
- Proof strategy: Can it choose an appropriate path—induction, contradiction, construction, approximation, stratification, commutative diagrams, or spectral sequences?
- Closing the details: Is every assertion backed by stated premises? Are unproven lemmas explicitly flagged?
- Expression and delivery: Can it produce readable English lecture notes, including definitions, notation, propositions, proofs, remarks, and follow-up exercises?
In particular, distinguish “retrieval and reproduction” from “independent construction.” If a statement comes from a common textbook, the model may simply be reciting a memorized proof. Only when the model produces a complete argument for a statement lacking a public proof does it genuinely demonstrate generalized reasoning. To avoid misjudgment, prioritize obscure variants, statements with modified conditions, cross-chapter synthesis problems, and boundary cases known to hide traps.
2. Design Reproducible Prompts, Not Improvised Ones
A high-quality prompt is not about length—it is about writing the task contract clearly. Recommended elements: set the role as a rigorous mathematical writing assistant; provide the precise statement and usable assumptions; require English lecture notes as output; forbid citing nonexistent theorems or faking unproven steps; require a self-contained but not bloated style; and ask for a list of key lemmas, a dependency outline, and risk points. An illustrative template:
Prove the following statement rigorously.Output format: English lecture notes in mathematical typesetting syntax.Requirements:1. State all assumptions and notation.2. Prove from first principles or clearly label any standard fact used.3. If a lemma is needed but not proved, mark it as [Gap] and stop.4. End with a dependency outline and potential counterexample checks.
The value of this kind of template is that it builds honesty into the output. Allowing the model to say “I don’t know” is safer than forcing it to fill in gaps. For high-stakes statements, ask the model to produce two independent routes: a main proof and a verification summary. Then have the same model redo the task in separate sessions and check whether conclusions remain stable. If repeated samples contradict each other on a key lemma, the output should not be adopted directly.
3. Bring Cost into the Engineering Metrics
Experience such as “about 2% of a monthly quota consumed to produce one set of lecture notes” can be abstracted into a cost-ratio metric: quota consumed per task divided by total periodic quota. This should not be read as a fixed price but as a relative budget signal. Mathematical proofs typically involve long chains of reasoning, long context, and multiple rounds of revision, so cost can grow nonlinearly with proof length. Engineering-wise, track four numbers: input token scale, output page count, number of iterations, and human repair time. If the model generates quickly but human review is extremely slow, total cost remains high; conversely, a rough first draft that reliably exposes gaps can still be valuable as an exploration tool.
Consider a tiered invocation strategy: use a low-cost mode first to screen proof routes, then a high-intensity reasoning mode to attack critical lemmas, and finally a lightweight mode for language polish and format consistency. For batch processing of statements, early-stopping rules matter—for example, switch to human review or downgrade the goal after two consecutive rounds fail to close the core lemma.
4. Build an “Adversarial” Verification Process for Correctness
Acceptance of mathematical output should never rest on a single model’s self-assessment. A recommended process includes:
- Restatement consistency check: Confirm no conditions were added or dropped when restating the proposition.
- Dependency audit: List where every invoked theorem is used.
- Counterexample injection: Deliberately weaken assumptions and see whether the proof still “holds”—if it does, there is likely a hole.
- Dimension and category checks: Especially for statements involving algebraic geometry, topology, or category theory, verify that objects live in the correct category.
- Notation normalization: Prevent the same symbol from drifting in meaning across sections.
- Human deep reading: Manually review the two or three most critical steps.
You can also introduce machine-checkable approximation signals: convert the proof into a semi-structured outline and verify that every node’s incoming edges have supporting premises; scan with scripts for high-risk phrases like “obviously,” “clearly,” and “similarly”; count the number of unproven lemmas; and run small-scale experiments for numerical or combinatorial claims. Formal proof assistants would be most valuable in an ideal world, but fully formalizing natural-language proofs is itself expensive—today they are better suited to local verification of key lemmas.
5. Avoid Fan-Style Rankings in Model Comparison
Differences among general-purpose models in mathematics typically show up in chain-of-thought length, conciseness, robustness, writing quality, and honesty. Comparisons should not assign absolute rankings; instead, build a dimensional scorecard: hard-problem solve rate, verifiability rate, hallucinated-lemma rate, average iteration count, output per unit cost, long-context consistency, and diagnosability of failures. Some models produce very long reasoning chains—broader coverage but more noise; some write concisely—good for final drafts but prone to omitting premises; others occasionally shine on open research problems but lack stability. For enterprises and research teams, selection should match the task: lecture generation favors readability and formatting; theorem work favors depth and self-consistency; question-bank production favors throughput and cost; formal paper assistance demands rigorous auditing.
6. Operational Advice: From One-Off Impressions to Pipeline Capability
To use these capabilities for lecture notes, research memos, or internal training, accumulate three types of assets: a prompt template library, a failure case library, and a review checklist. The template library stores fixed structures for different mathematical fields; the failure case library records classic pitfalls such as swapped quantifiers, circular reasoning, invalid limit exchanges, and category errors; the review checklist shifts responsibility from “trusting the model” to “verifying the model.” In addition, label every generated item with a provenance status: proven, cited as a standard fact, unproven lemma, or speculative intuition. Mature teams do not chase a perfect proof in one shot—they pursue making uncertainties explicit as quickly as possible.
In summary, the most convincing evidence of a general-purpose LLM’s mathematical proof ability is not whether it can talk about mathematics like a human, but whether it can produce auditable, reproducible, cost-controlled reasoning artifacts on unknown propositions. Combining prompt contracts, independent proofs, adversarial verification, cost-ratio tracking, and human review upgrades evaluation from subjective impression to an engineering process—and delivers stable value in lecture production, research support, and knowledge base construction.
