Evaluating Large Language Models on Mathematical Proof and Lecture Note Generation
This article presents a task-driven, reproducible methodology for evaluating how well large language models can independently prove unproved propositions from public mathematical references and express them as well-formatted English lecture notes. It defines the evaluation task and prompt design, outlines eight assessment dimensions including correctness, rigor, completeness, originality, cost, and reproducibility, discusses common misinterpretation pitfalls, and proposes an engineering-grade evaluation pipeline with automated screening, expert blind review, and cost accounting.
Evaluating Large Language Models on Mathematical Proof and Lecture Note Generation
1. Background: Why Test Models with “Unproved Propositions”
General-purpose large language models are already fairly mature at routine Q&A, code completion, and text polishing. Mathematical proof, however, is a much harder litmus test that surface fluency cannot easily mask. The reason is that a proof task demands not only a correct final conclusion but also clear definitions, valid lemmas, a closed chain of reasoning, complete boundary conditions, and no reliance on unverified implicit assumptions. If the task further requires writing the proof as English lecture notes, additional requirements stack up: accurate terminology, formula typesetting, section organization, and pedagogical readability.
Using “unproved propositions” from public mathematical reference materials as input is a valuable evaluation scenario. Such tasks are not simple retrieval and restatement: the reference material provides only the proposition statement, and the model must organize its own proof path. This tests whether the model has independent derivation ability, whether it can identify required prerequisites, whether it skips critical steps, and whether it can express results in standard English lecture note format. One caveat: a single success does not imply general capability, and a single failure does not rule the model out. Evaluation must introduce a task set, scoring criteria, and a reproducible process.
2. Task Definition and Prompt Design
A reproducible evaluation task should specify inputs, constraints, and output format explicitly. For example: given an unproved proposition from public reference material, the model must independently produce a proof and write it as English lecture notes using formula typesetting markup. The notes must include the problem statement, prerequisite definitions, the main proof, explanations of key lemmas, and a conclusion. Fabricated citations are prohibited; any external result used must be labeled with its source. Prompts should remain neutral—avoid hinting that “a specific known proof must be given,” and avoid asking the model to imitate a particular author’s style.
Illustrative pseudocode:
Task record = {task_id,proposition_statement,allowed_prerequisite_knowledge,output_format_requirements,prohibited_actions,scoring_dimensions}For each task:Generate multiple candidate outputsAutomatically check format completeness and formula typesettingHave domain experts blind-review correctness, rigor, and completenessRecord error types, human revision effort, and quota consumptionAggregate into task-level and model-level reports
Prompts should also fix variables such as the number of samples, output length limits, and whether multi-turn revision is allowed. Otherwise, different interaction entry points, different workflow entry points, and different retry strategies make results incomparable. Community discussions often feature feedback like “one task worked well” or “one task clearly failed”—such anecdotes are better treated as leads for investigation than as grounds for horizontal ranking.
3. Evaluation Dimensions: From “Looks Capable” to “Auditable Capability”
Correctness. The final conclusion, key lemmas, and every derivation step should be verifiable. Evaluations should distinguish between “correct conclusion with skipped steps,” “correct proof framework with missing boundary conditions,” and “correctly cited result with mismatched applicability conditions.”
Completeness. The notes must cover necessary definitions, notation, the main proof chain, and a conclusion that ties back to the statement. Missing prerequisites make the material impossible for readers to reproduce.
Rigor. Focus on circular reasoning, hidden assumptions, quantifier order errors, and misuse of limits and existence conditions.
Originality (non-copying). For an unproved proposition, merely rewriting the reference material’s statement into fluent text does not demonstrate reasoning ability. Assess this through similarity to the reference material, diversity of proof paths, and expert judgment.
Expression quality. Whether English terminology is accurate, formulas are cleanly typeset, and the structure is readable are important metrics for engineering deployment—but fluency cannot substitute for mathematical correctness.
Reproducibility. Record whether repeated sampling with the same prompt is stable, whether performance degrades through a different interaction entry point, and whether long tasks tend to lose constraints midway.
Cost and quota. Some users accessing complex tasks via subscription plans have reported that a single task can consume a low single-digit percentage of a monthly quota; such data serves only as anecdotal cost reference and cannot be extrapolated into general billing rules. Enterprise evaluations should combine generation cost, retry cost, human review cost, and rework cost.
Safety and compliance. Mathematical lecture notes may involve citations, copyright, and data sources. Outputs must avoid fabricated references, and sensitive material should be authorized and anonymized before use.
4. Common Pitfalls in Interpreting Results
First, single-run experience cannot replace systematic evaluation. A complex proof task may perform well because the prompt is clear and the proposition happens to match the model’s training distribution, or fail due to symbol ambiguity. Second, a longer chain of thought does not mean higher quality. Long reasoning may contain more self-checking, but also more redundancy and error propagation; what matters is whether key steps are verifiable. Third, “writing that sounds more human” is not evidence of mathematical correctness. Natural style and fluent English are pluses, but experts still need to verify line by line.
Moreover, open mathematical problems have a different difficulty structure from standard textbook exercises, competition problems, or course assignments. Stable scores on standard problems do not imply the ability to handle unproved open-type propositions; conversely, one successful independent proof does not imply general open-problem solving ability. Horizontal comparisons must also control prompts, sampling counts, human revision effort, and scoring criteria—otherwise rankings easily become “impression-based rankings.”
5. Recommendations for an Engineering-Grade Evaluation Pipeline
Enterprises and research teams can build a layered evaluation pipeline: layered task sets, templated prompts, multi-candidate generation, automated pre-screening, expert blind review, error attribution, cost accounting, and regression testing. Automated pre-screening can check format, formula closure, citation placeholders, and length; expert blind review handles correctness and rigor; error attribution classifies problems into categories such as missing knowledge, skipped reasoning, unclear expression, formatting errors, and inaccurate citations. For enterprise deployment, the pipeline can be assembled from general-purpose capabilities such as computing, object storage, and logging/monitoring offered by cloud providers such as Baidu AI Cloud—the concrete implementation should remain neutral, auditable, and replaceable.
6. Conclusion
When evaluating a large model’s advanced mathematical capability, adhere to task orientation, multiple metrics, and reproducibility. Mathematical proof and English lecture note generation are excellent stress-test scenarios, but correctness, rigor, completeness, expression quality, originality, cost, and human review must be measured within the same framework. Single-run experiences, community rankings, and chain-of-thought length are leads only, not substitutes for systematic evaluation. For developers, the safer path is to build a small but high-quality task set, fix prompts and scoring procedures, continuously record error types and human revision effort, and then make decisions in light of business cost.
