A Practical Guide to Evaluating LLMs on Math Proofs and Lecture Note Generation

    This guide explains how to evaluate large language models on two related tasks: writing mathematical proofs and generating lecture notes. It covers how to check process quality and output quality separately, how to design benchmarks around statements without published proofs, how to structure prompts and output contracts, which quality and cost metrics to track, how to group models into capability tiers, and how to set up a repeatable evaluation workflow.

    1. Defining the Problem: Proof Generation and Lecture Note Generation Are Composite Tasks

    Evaluating the mathematical ability of a large language model cannot stop at whether it gives a correct final answer. Proof generation and lecture note generation are related but distinct abilities. Proof generation emphasizes logical correctness, complete steps, and reliable citations. Lecture note generation also requires a clear structure, accurate definitions, natural English, mathematical markup source that compiles, and a sensible teaching sequence. Evaluating both together is closer to real research and teaching scenarios, but it also places higher demands on the evaluation design.

    A common mistake is to check only whether the conclusion is correct. For proof problems, a correct conclusion does not mean the proof is valid. The reasoning may skip steps, argue in a circle, rely on hidden assumptions, cite sources incorrectly, or look plausible without being verifiable. Evaluation should therefore cover both process quality and output quality. Process quality includes whether assumptions are stated explicitly, whether the lemmas are sufficient, and whether counterexamples are ruled out. Output quality includes the structure of the notes, whether the markup compiles, terminology consistency, citation credibility, and readability for the target audience.

    1. Benchmark Design: Prioritize Propositions Without Published Proofs

    If a benchmark statement already has a complete proof in public sources, a model may reproduce the answer from memory or retrieval. This makes it hard to separate reasoning ability from recitation. A more valuable benchmark uses statements from a public mathematical resource that give only the statement, with no proof attached. The model must construct a proof within the given definitions and permitted references, then organize it into English lecture notes and mathematical markup source. This reduces the risk of reproduction while keeping the results verifiable. Domain experts can review the proof against a scoring rubric, the markup source can be compiled, and each citation can be checked.

    Task design should set clear boundaries. These include the statement, the known definitions, the standard theorems that may be cited, the output language, the markup format, whether external materials are allowed, and whether the model must list all assumptions. For difficult tasks, ask the model to produce structured notes in this order: summary, definitions, lemmas, main proof, self-check, and risk notes. The self-check section should require the model to flag uncertain steps, conditions that may be missing, and places that need human review. This supports both automatic parsing and manual review.

    1. Prompts and Output Contracts

    Prompts should avoid vague instructions such as “prove this and write it up as lecture notes.” Instead, define an explicit contract. The system prompt can require the model to avoid fabricating citations, mark uncertain points, list all implicit assumptions, make each proof step checkable line by line, and include compilable mathematical markup source. The task prompt should state the goal, the known conditions, the permitted citations, the output structure, and the points that scorers will focus on. The output contract can be described as structured fields, for example: task summary, definitions and notation, list of lemmas, main proof, self-check list, citation list, and markup source.

    Pseudo-code example:

    1. function build_prompt(task):
    2. return {
    3. role: "Rigorous mathematical author",
    4. goal: "Prove the statement under the given conditions and generate lecture notes in English",
    5. constraints: [
    6. "Do not fabricate citations",
    7. "State assumptions explicitly",
    8. "Mark uncertain steps",
    9. "Output structured lecture notes and mathematical markup source"
    10. ],
    11. input: task.statement + task.definitions + task.allowed_theorems,
    12. output_schema: [
    13. "summary", "definitions", "lemmas", "proof",
    14. "self_check", "citations", "math_markup"
    15. ]
    16. }
    1. Quality Dimensions and Quantitative Methods

    Evaluation should cover at least correctness, rigor, originality and non-recitation, readability, markup compilability, citation reliability, cost, latency, and stability. Experts judge whether the main proof holds, using a scoring rubric. Rigor covers assumptions, lemmas, boundary conditions, and whether counterexamples are excluded. Originality checks whether the answer clearly reproduces a known solution, and whether a proposition without a published proof was actually constructed by the model. Readability covers the structure of the notes and terminology consistency. Markup compilability is tested by compiling the source in an isolated environment and recording the failure rate and error types. Citation reliability requires checking each citation to see whether it is fabricated or mismatched.

    Cost should not be measured only by the price of a single call. Include rework rounds, human review time, compilation retries, and quota consumption. The total cost of a task can be defined as: generation cost + retry cost + compilation repair cost + human review cost. Practical feedback suggests that a medium-difficulty lecture note task may use a single-digit percentage of a monthly subscription quota, but results vary widely across tasks. Log the input size, output size, retry count, and human intervention time for every task in a consistent format.

    1. Capability Tiers and Selection Principles

    To avoid ranking specific products, use capability tiers. L0 covers basic Q&A and restating standard conclusions. L1 covers reproducing common standard proofs. L2 covers handling medium-difficulty propositions without published proofs and producing checkable proofs. L3 covers completing difficult constructions and producing complete, compilable lecture notes. L4 covers proposing verifiable research directions for open problems.

    When choosing a model, classify tasks by risk. Low-risk tasks, such as rough drafts, can use a lower tier with human review. High-risk proofs and teaching materials should require a higher tier, along with two-person review, citation checks, and compilation verification.

    Longer chains of reasoning do not guarantee better quality, and more natural writing does not mean a proof is correct. Evaluation should follow these principles: the process can be checked, the output can be compiled, citations can be verified, and cost can be tracked over time. Do not rely on an impression from a single problem or on subjective style preferences.

    1. Engineering an Evaluation Workflow

    Build a layered evaluation set with basic definition questions, standard theorem reproduction, medium problems without published proofs, difficult constructive problems, and lecture note generation tasks. Each category needs its own scoring rubric and pass threshold. The workflow includes blind review, two-person review, version regression, archiving of failure cases, compilation in an isolated environment, and logging with monitoring alerts. General-purpose container platforms, logging services, and monitoring tools can record the inputs, outputs, latency, cost, and failure reasons of each evaluation run. Store evaluation data with version control so that different model versions and prompt versions can be compared.

    Pseudo-code example:

    1. function evaluate(task, model):
    2. prompt = build_prompt(task)
    3. result = model.generate(prompt)
    4. parsed = parse_structured_output(result)
    5. compile_ok = compile_in_isolated_env(parsed.math_markup)
    6. proof_score = expert_review(parsed.proof, task.rubric)
    7. citation_score = check_citations(parsed.citations)
    8. cost = measure_quota_and_time(result)
    9. return record(task.id, proof_score, compile_ok,
    10. citation_score, cost, parsed.self_check)
    1. Common Pitfalls and Conclusion

    Common pitfalls include judging by a single problem, looking only at the final answer, ignoring retrieval contamination, ignoring markup compilation, skipping citation checks, ignoring rework costs, and mistaking writing style for mathematical rigor. A more reliable approach is to use propositions without published proofs as a high-value benchmark, build multi-dimensional scoring rubrics, and combine automatic parsing, isolated compilation, and expert review. This produces an evaluation system that is reproducible, suitable for regression testing, and sustainable over time. Only then can you objectively assess the real capability limits of large language models in math proof and lecture note generation.