Evaluating LLM Mathematical Proofs Under a Verifiable Research Workflow

    This article introduces a rigorous evaluation framework for large language models in research settings, shifting the focus from "plausible answers" to "independently verifiable artifacts." It covers task definition and verification levels, metric design beyond simple success rates, observations from public experiments—including girth-5 graph constructions and tightened coding bounds—and engineering guidance for building reproducible evaluation pipelines with a human-model-verifier loop.

    1. Background: From “Generating Answers” to “Reviewable Artifacts”

    Large language models continue to improve at natural language QA, code generation, and mathematical reasoning. But research scenarios demand something fundamentally different from casual conversation. What a research task truly requires is not a conclusion that “looks reasonable,” but an object that can be independently reviewed: a checkable proof, a construction that satisfies given constraints, an executable program, or a verifiable parameter bound. Without a verification loop, model-generated content easily stays at the level of “plausible but possibly wrong,” making it hard to integrate into a trustworthy research workflow.

    A more rigorous evaluation approach therefore places the problem definition, attempt budget, verifier, artifact records, and failure analysis inside a single verifiable system. The system does not just look at final scores; it records how many attempts the model used, in what manner, and on which tasks it produced reviewable results. This kind of evaluation is much closer to real research collaboration: a human poses a well-defined problem, the model searches for candidate solutions under a constrained budget, a verifier returns pass/fail feedback, and researchers then assess the value and novelty of the outcome.

    2. Framework Design: Verification as the First Principle

    2.1 Task Definition and Artifact Types

    A verifiable evaluation task should contain, at minimum: a problem statement, constraints, a reference baseline, a verifier specification, an attempt budget, and a logging format. Model output is not limited to natural-language answers; it should be constrained to reviewable artifacts, such as:

    • Formal or semi-formal proofs, including key lemmas and reasoning steps;
    • Combinatorial constructions, such as point sets, edge sets, matrices, or sequences;
    • Executable programs with their optimization objectives;
    • Proof-based arguments for parameter upper/lower bounds;
    • Counterexamples or infeasibility statements.

    The verifier should be isolated from the model under test and able to independently check whether the artifact meets the task specification. For mathematical arguments that cannot be fully auto-verified, a layered mechanism of “executable checks + human audit” can be used, but the verification level must be made explicit to avoid mixing subjective human judgment into automated scores.

    2.2 Task Types and Verification Levels

    Evaluation tasks can be tiered by verification strength:

    • Fully formal verification: proof steps can be mechanically checked by a proof checker;
    • Executable verification: constructions or programs can be checked by scripts against constraints and objective values;
    • Semi-formal review: proof logic is audited by domain experts against a checklist;
    • Statistical reproduction: results are stable across multiple runs, and failure modes can be classified.

    Different verification levels should not be naively combined into one total score; they should be reported separately. Otherwise, a model that succeeds on strongly verified tasks may be incorrectly compared with one that scores on weakly verified tasks.

    2.3 Metrics: Coverage and Improvement Beyond Success Rate

    A single success rate can mask differences in task difficulty and attempt budgets. More reasonable metrics include:

    • First-attempt success rate;
    • Within-budget success rate;
    • Number of distinct tasks covered (solved at least once);
    • Independent completion rate—the success rate without intermediate hints or lemma hints;
    • Reviewability rate—the proportion of outputs that can be fully checked by the verifier or auditors;
    • Improvement magnitude—the gain over the reference baseline in edges, bounds, runtime, etc.;
    • Stability—the variance across repeated experiments and the distribution of failure types.

    For example, the within-budget success rate can be written as:

    1. Success rate = solved tasks / total tasks
    2. Coverage = number of distinct tasks solved at least once
    3. Independent completion rate = tasks solved without hints / total tasks

    The triple of “attempts – successes – coverage” should also be recorded, rather than summarizing capability with a single percentage.

    3. Observations from Public Experiments: High Scores Do Not Mean Across-the-Board Dominance

    In one public verifiable evaluation of this kind, the top overall score was 71.9, with the second-place model at 57.4. The top model completed 14 out of 20 theorems without intermediate lemmas, a 70.0% success rate; the second-place model succeeded 23 times in 60 attempts, covering 9 distinct theorems, for a 38.3% success rate; the remaining six agents all stayed at or below 11.7%. These numbers illustrate two points. First, given well-defined problems, sufficient budget, and verifiable feedback, some models can already complete a meaningful proportion of proof and construction tasks. Second, capability is unevenly distributed across models: leading the overall leaderboard does not mean leading every subtask.

    A typical counterexample is an alphabet reliability task: the second-place model completed it in all three attempts, while the top-scoring model failed to solve it at all. This shows that evaluation cannot rely on total-score rankings alone; task-level coverage and failure modes matter. Some models may be stronger at proof search, while others are more consistent at constraint satisfaction or constructive search.

    Two categories of results are particularly concrete. The first is a classic extremal graph theory task—constructing a graph of girth 5: one model produced a graph with 124 vertices and 630 edges containing no triangles or 4-cycles, exceeding the 629-edge reference value used by the task. The second is a coding-parameter upper bound task: one model tightened a published upper bound from 84 to 82. The significance of these results is not that “models replace mathematicians,” but that models can perform candidate search, local improvement, and proof attempts under verifiable objectives, and hand the results to a verifier for checking.

    4. Capability Boundaries: Verifiable Feedback Is an Amplifier, Not Autonomous Research

    Public experiments are better read as a capability map than as a conclusion of “surpassing humans across the board.” Models can already complete some proofs, program optimizations, and construction improvements when given clear problems, sufficient budgets, and verifiable feedback. But autonomously choosing research directions, judging the value of results, and posing major open problems still require humans.

    Specifically, models are good at:

    • Generating large numbers of candidates under given constraints;
    • Locally modifying and searching over candidates;
    • Translating natural-language problems into checkable proof drafts or programs;
    • Iterating under verifier feedback until the objective is met or the budget is exhausted.

    Models are still weak at:

    • Judging whether a problem is worth researching;
    • Recognizing the novelty and theoretical significance of a result;
    • Guaranteeing rigor when no verifier is available;
    • Sustaining a long-term research agenda and handling open-ended subproblems.

    Therefore, human-AI collaboration should be engineered as a closed loop of “humans define problems and audit, models search and generate, verifiers adjudicate and record.” Humans are responsible for defining problems, selecting verification levels, and auditing the verifier itself; models are responsible for producing reviewable candidates within budget; the system is responsible for logging attempt trajectories and failure causes.

    5. Engineering Guidance: Building a Reproducible Evaluation Pipeline

    A verifiable evaluation pipeline can be abstracted as the following pseudocode:

    1. for task in task_set:
    2. for attempt in range(task.budget):
    3. output = model.solve(task.statement, task.hints)
    4. verdict = verifier.check(output, task.spec)
    5. log(task.id, attempt, output, verdict, cost)
    6. report = aggregate(task.id, logs)
    7. report.add_failure_modes(classify(logs))

    Practical considerations when implementing it:

    1. Deploy the verifier independently, so the model under test cannot access reference answers;
    2. Unify attempt budgets and resource metering, so “unlimited retries” do not make results incomparable;
    3. Preserve raw artifacts, verification logs, and failure classifications for every task;
    4. Establish audit checklists for semi-formal tasks to reduce subjective drift;
    5. Report success rate, coverage, independent completion rate, and improvement magnitude separately—do not rely on total score alone.

    6. Conclusion

    The value of a verifiable research workflow lies in pushing large models from “generating answers” to “generating reviewable objects.” Public experiments show that, with clear problems, sufficient budgets, and verifiable feedback, models can already complete some theorem proofs, construction improvements, and program optimizations. But a leading total score does not equal across-the-board dominance; task-level differences, failure modes, and human audit remain critical. For developers and enterprises, the more pragmatic path is to build reproducible, auditable evaluation systems with per-dimension reporting, and to write the boundaries of human-AI collaboration into the process—rather than misreading a single high score as autonomous research capability.