Workflow Design and Quality Evaluation for LLM-Assisted Mathematical Proofs

    This article explores how to leverage Large Language Models (LLMs) to assist with mathematical work, focusing on workflow design and quality evaluation. It details task decomposition, structured generation of lecture notes, multi-layer verification methods, credibility grading, evaluation dimensions, and resource management. By positioning the model as a drafter and an independent verification process as the arbiter, researchers can improve efficiency while maintaining mathematical rigor.

    1. Suitable Stages for LLM Assistance in Mathematical Work

    Large language models perform relatively stably in text understanding, symbolic transformation, and structured writing. Therefore, they can undertake certain stages of mathematical work, such as organizing definitions, constructing proof frameworks, filling in intermediate steps, and generating lecture note drafts. However, the reliability requirement for mathematical conclusions is much higher than general text generation; model outputs can only serve as drafts to be verified. Positioning the model as a drafter and an independent verification process as the arbiter is the premise of the entire workflow.

    2. Task Decomposition: Definitions First, Proofs Second

    Before submitting a complete proposition to the model, four aspects should be clarified:

    1. Research objects and all definitions, including symbol meanings and applicable scopes;
    2. Known premises: which conclusions can be directly cited, with sources noted;
    3. Target statements: distinguishing the conclusion to be proved from its sufficient conditions;
    4. Output requirements: such as step-by-step numbering and annotating the basis for each step.

    When decomposing into a lemma chain, each lemma can only depend on previously confirmed results. Dependencies can be represented using a directed graph, so that once an error is found, the failing stage can be quickly located, and it can be determined which downstream conclusions need to be rechecked.

    3. Structured Generation of Lecture Notes

    Mathematical lecture notes usually consist of five parts: proposition statement, prerequisites, main proof body, remarks, and references. During generation, the model can be required to use uniform mathematical typography markup and place each proof step into an independent paragraph for easy step-by-step verification later.

    If the original literature does not provide a proof, the lecture notes should clearly state, “This proof is newly written and has not been confirmed by the original source.” This prevents readers from mistakenly thinking the proof originated from the literature and clearly indicates which content requires further verification.

    4. Verification Methods for Results

    Verification includes at least three levels:

    • Formal level: Are symbols consistent, are variable scopes clear, and is the quantifier order correct?
    • Logical level: Is each step directly derived from premises, and are there implicit unannounced assumptions?
    • Literature level: Are the cited theorem conditions satisfied, and do the numbers and statements match the original text?

    In practice, another independent session or a human expert can be asked to review the draft step by step, requiring reviewers to point out specific steps and specific issues rather than giving general evaluations. For propositions that can be tested by special cases, small-scale counterexamples or numerical verifications should also be constructed.

    The verification process is illustrated as follows:

    1. def generate_and_verify_proof(proposition):
    2. lemma_list = decompose_to_lemma_chain(proposition)
    3. passed = []
    4. for lemma in lemma_list:
    5. draft = call_model_generate_proof(lemma, passed)
    6. verification_result = verify_step_by_step(draft)
    7. if verification_result == "failed":
    8. record_error_location()
    9. retry_once_if_still_fails_route_to_human()
    10. else:
    11. passed.append((lemma, draft))
    12. review_feedback = submit_to_independent_reviewer(passed)
    13. return passed, review_feedback

    5. Credibility Grading for Conclusions

    It is recommended to divide the output into three levels:

    • Verified: Every step has been verified, and citations can be traced back to the original text;
    • To be reviewed: The logic is generally sound, but some parts have not been fully checked;
    • Untrustworthy: There are unrepairable logical breaks or citation errors.

    Only content reaching the “Verified” level can enter formal documentation. For open problems, new ideas provided by the model should be treated as research leads and cannot be used as conclusions until independent verification is completed.

    6. Evaluation Dimensions and Comparison Methods

    When comparing different models or different configurations of the same model, a unified task set and unified scoring criteria should be used. Common dimensions include:

    1. Correctness: Are the final conclusion and key steps valid?
    2. Completeness: Are necessary conditions or boundary cases omitted?
    3. Verifiability: Are the steps clear and can they be independently checked by others?
    4. Expression quality: Are symbols and terms standardized, and is the structure easy to read?
    5. Resource cost: Response time, number of calls, and quota consumption for a single task.

    During the evaluation process, the input, output, and verification results of each task should be recorded to ensure others can reproduce them. Sorting based solely on personal subjective impressions makes it difficult to reach reliable conclusions.

    7. Resource and Quota Management

    Long-running derivation tasks can easily hit rate and quota limits. The following measures can be taken:

    • Select configurations based on task complexity: use lightweight configurations for simple organizing tasks, and high-performance configurations for critical steps;
    • Split long tasks into subtasks with checkpoints, saving intermediate results at each step so they can be resumed from the breakpoint if interrupted;
    • Set warning thresholds for quotas, pausing low-priority tasks when approaching the limit;
    • Record consumption at each stage for subsequent scheduling and cost estimation.

    The checkpoint mechanism is illustrated as follows:

    1. progress = read_saved_checkpoint()
    2. for subtask in pending_subtask_list:
    3. if subtask marked_as_completed_in_progress:
    4. continue
    5. if current_quota_usage_rate > warning_threshold:
    6. pause_task_and_notify_owner()
    7. break
    8. result = execute(subtask)
    9. save_checkpoint(subtask, result)

    8. Common Pitfalls

    1. Treating linguistically fluent text as a correct proof;
    2. Only looking at the final conclusion without checking intermediate steps;
    3. Treating model-generated content as literature citations and using it directly without confirming with the original text;
    4. Comparing the effects of different configurations without unified standards and drawing conclusions based on that.

    9. Conclusion

    The key to LLM-assisted mathematical work lies in workflow design: clear task decomposition, step-by-step verification, graded credibility labeling, and manageable resource control. Using the model as an efficient drafting and checking assistant can improve work efficiency while ensuring the rigor of conclusions.