LLM-Assisted Mathematical Proofs: Evaluation Methods and Engineering Practices
This article explores the unique challenges of evaluating LLM-generated mathematical proofs and provides a comprehensive engineering practice guide. It covers evaluation dimensions, test sample construction, a four-step verification process, large task management, and common pitfalls to help teams assess the reliability and scope of LLMs in proof tasks.
1. Why Proof Tasks Require Separate Evaluation
Standard Q&A tasks typically only check whether the final answer is correct, but proof tasks are different. The value of a proof lies in the verifiability of every step; a correct conclusion does not guarantee a correct derivation process. Common issues in model-generated proofs include:
- Skipping Steps: Key assertions are stated directly without explaining the basis.
- Circular Reasoning: Using the conclusion to be proved as an intermediate premise.
- Distorted Citations: Citing non-existent theorems or applying conditions that do not match the original theorem.
- Omitted Conditions: Applying a conclusion in scenarios where its assumptions are not met.
- Generalizing from Special Cases: Claiming a statement holds for all objects after only verifying a simple example.
These problems are more likely to occur in lengthy proofs, as errors can be masked by fluent expressions later in the text. Therefore, evaluation requires step-by-step verification rather than just an overall impression.
2. Evaluation Dimension Design
Evaluating a proof can be approached from the following five dimensions.
| Dimension | Core Check Question | Typical Failure |
|---|---|---|
| Proposition Alignment | Is the model proving the required proposition, or has it been replaced with a weaker one? | Substituting conditions, proving only special cases |
| Dependency Results | Do the cited prior conclusions actually exist, and are their assumptions met? | Citing wrong versions, ignoring applicable conditions |
| Logical Completeness | Can every step be traced back to known premises or proven conclusions? | Skipping steps, circular reasoning |
| Symbols and Quantifiers | Are variable definitions consistent throughout, and is the quantifier order correct? | Confusing existential and universal quantifiers, symbol reuse |
| Source Distinction | Can it distinguish between existing conclusions in literature and newly added arguments? | Packaging known results as original conclusions |
Each dimension should yield a clear pass or fail judgment. Providing only a vague total score fails to pinpoint specific problems and makes it difficult to guide subsequent revisions.
3. Constructing Test Samples
The quality of test samples determines the credibility of the evaluation conclusions. When building samples, note the following points.
3.1 Differentiate Between Propositions With and Without Standard Answers
If a proposition has a complete proof in public reference materials, the model’s output can be directly compared with the standard proof to check if key steps are covered. If a proposition only has a statement but no proof in the literature, direct comparison is impossible, and manual review or formal verification is required. Such samples should be tagged separately in the dataset to avoid mistaking “incomparable” for “incorrect.”
3.2 Grading by Difficulty
It is recommended to divide samples into four levels:
- Definition and Basic Property Reproduction: Check if the model can accurately use definitions.
- Standard Proofs: Require combining two to three known conclusions.
- Multi-step Composite Proofs: Require constructing auxiliary lemmas and handling multiple cases.
- Open-ended Problems: No known conclusions for reference; only the soundness of reasoning can be evaluated.
After grading, the pass rate of each level can be calculated separately to determine the model’s capability boundaries.
3.3 Record Sources and Versions
Reference materials get updated, and the numbering and phrasing of the same conclusion may change. Each sample should record the source, version, and access date of the standard answer to prevent misjudgment caused by material changes.
4. Verification Process
It is recommended to divide verification into four steps, progressing from easy to difficult.
4.1 Formal Check
First, check layout and syntax issues. For example, whether formulas can be compiled by the typesetting system, whether environments are closed, and whether numbering and citations exist. This step has the lowest cost and can quickly filter out obviously unqualified outputs.
4.2 Split Lemmas and Verify Step-by-Step
Split the proof into several statements, annotating the basis for each statement: is it a problem assumption, a known theorem, or a conclusion already derived in this proof? Statements missing a basis are directly marked for review.
Here is an illustrative checking process; the specific implementation can be adjusted according to the team’s toolchain:
function verify_proof(proof):lemmas = split_into_statements(proof)results = []for lemma in lemmas:refs = extract_citations(lemma)refs_ok = all(lookup_reference(r) and conditions_satisfied(r, lemma) for r in refs)logic_ok = check_inference(lemma) # Manual review or formal proof toolsresults.append({ "id": lemma.id, "pass": refs_ok and logic_ok })return results
The key here is the conditions_satisfied item. Many errors are not about citing non-existent references, but about ignoring the theorem’s prerequisite conditions during citation.
4.3 Counterexample Testing with Weakened Assumptions
Weaken the assumptions actually used in the proof one by one to observe whether the conclusion still holds. If deleting a certain assumption allows a step of the proof to pass without any modification, it suggests the assumption might be redundant, or it might indicate a hidden error in the proof. If the conclusion clearly fails after deleting the assumption, but the proof remains unchanged, it means the proof process ignored the role of that assumption.
4.4 Sampled Expert Review
Automated checks cannot replace the judgment of domain experts. It is recommended to sample a certain proportion of proofs that passed the first three steps for review by personnel familiar with the field, focusing on the output of open-ended problems. The review results should be fed back into the sample library to refine the evaluation standards.
5. Splitting and Process Management for Large Tasks
A complete, lengthy proof or lecture note usually requires a long generation time. Directly submitting a large task incurs a high cost if it fails and needs to restart. Here are a few common engineering practices.
5.1 Split into Subtasks
Divide large tasks by chapter or theorem, with clear inputs and outputs for each subtask. For example, the first subtask organizes definitions and notations, the second proves the main lemmas, and the third assembles the conclusions. Subtasks pass results via intermediate files, making it easy to rerun them individually.
5.2 Set Checkpoints
After completing each subtask, write the intermediate results, parameters used, and timestamp to a log. After an interruption, you can resume from the nearest checkpoint instead of starting from scratch.
5.3 Pre-run Estimation and In-run Monitoring
Long-running tasks should estimate resource consumption before starting and set an upper limit. An illustrative configuration is as follows:
task:name: Proposition Verification and Lecture Note Generationmax_runtime_minutes: 120checkpoint_interval_minutes: 15usage_guard:enabled: truestop_when_remaining_below: 20 # Pause when remaining quota is below this percentage
During execution, progress should be checked periodically. If the task deviates from the goal, such as repeatedly generating the same content or producing no effective output for a long time, it should be stopped promptly to avoid wasting quota.
5.4 Segmented Usage Statistics
Quotas are usually calculated over different time windows, and short-term and long-term limits may apply simultaneously. It is recommended to record task consumption daily and weekly, and schedule large tasks during periods with sufficient remaining quota. This prevents tasks from being interrupted midway due to window limits, which affects overall progress.
6. Common Pitfalls
- Treating Length as Rigor: The longer the derivation, the easier it is to hide skipped steps. Length itself is not a basis for correctness.
- Only Looking at the Final Conclusion: A proof with a correct conclusion but erroneous basis will introduce risks in subsequent applications.
- Repeatedly Tuning on the Same Batch of Samples: After test samples are overused, results will be artificially inflated, losing discriminative power. A portion of the samples should be rotated regularly.
- Ignoring Presentation Quality: A proof with messy symbols and inconsistent definitions is hard to verify, even if the logic is generally correct, and should be penalized.
- Taking Model Output as Final Verdict: For open-ended problems, model output is better suited as a draft or source of ideas; the final conclusion still requires manual verification.
7. Implementation Suggestions
- Establish an internal evaluation set, clarifying the source and version of the standard answer for each sample.
- Write verification results into structured reports, recording the location, type, and revision suggestions for each failed step.
- Maintain skepticism towards model outputs, treating them as drafts that need verification.
- Regularly review evaluation results and adjust sample distribution and verification rules based on failure types.
By following this process, teams can determine the usable scope of the model in proof tasks at a lower cost, focusing manual review efforts on the areas most prone to error.
