Where AI Stands in Mathematical Research: A Verifiable Evaluation Perspective

    This article examines how large language models perform on research-level tasks such as mathematical proofs, constructions, and code optimization when evaluated under a fully verifiable framework. It explains why machine-checkable outputs matter, how a deterministic evaluation pipeline is designed, what the results reveal about model strengths and weaknesses, and where human judgment remains essential. It also offers practical guidance for engineering teams considering AI-assisted research workflows.

    1. Background: Why Verifiability of Research Outputs Matters

    Over the past couple of years, large models have been applied to research tasks such as mathematical proof, algorithm construction, and program optimization. Most discussions focus on whether a model can produce an answer. For research work, the more critical question is whether the result can be independently verified. A plausible-looking proof that no one can review within a reasonable timeframe offers limited value to the research process.

    For developers, this is closely analogous to software testing. Whether a piece of code is correct depends on whether tests can execute and return a conclusion—not on how readable the code is. AI-generated research outputs should follow the same logic: define how to verify first, then evaluate what the model generates.

    2. Core Design of the Evaluation Framework

    A joint effort by a university research team and researchers from an AI company attempts to place the outputs of the research workflow into a single verifiable system. Its design can be summarized in three points.

    1. Outputs must be reviewable artifacts. The model delivers a constructed graph, a set of parameters, or a machine-checkable proof—not a natural-language conclusion.
    2. Evaluation conditions are fixed. Every participating system receives the same problem descriptions, the same number of attempts, and the same budget, so results are comparable.
    3. Judgment is performed by programs. A verifier makes a deterministic pass/fail decision on each output, avoiding bias from subjective human scoring.

    The following code illustrates the basic structure of this kind of evaluation, for explanatory purposes only:

    1. from dataclasses import dataclass
    2. from typing import Any, Callable, Dict, List
    3. @dataclass
    4. class Attempt:
    5. problem_id: str
    6. artifact: Any # constructed object, parameters, or proof text
    7. def verify(attempt: Attempt, checkers: Dict[str, Callable[[Any], bool]]) -> bool:
    8. """Invoke the deterministic checker for the given problem and return pass/fail."""
    9. return checkers[attempt.problem_id](attempt.artifact)
    10. def success_rate(attempts: List[Attempt], checkers) -> float:
    11. if not attempts:
    12. return 0.0
    13. passed = sum(verify(a, checkers) for a in attempts)
    14. return passed / len(attempts)

    The value of this structure is that evaluation results do not depend on subjective reading of the outputs. As long as the checker is reliable, anyone can reproduce the conclusions with the same inputs.

    3. Reading the Results: Overall Rankings vs. Per-Task Differences

    On the aggregate score, the top-ranked model (referred to as Model A) scored 71.9, while the runner-up (Model B) scored 57.4. The gap is clear, but the total score alone can mask task-level differences.

    At the per-problem level, Model A solved 14 out of 20 theorems without being given intermediate lemmas, a success rate of 70.0%. Model B succeeded 23 times in 60 attempts, covering 9 distinct theorems, for a success rate of 38.3%. The remaining six systems all had success rates no higher than 11.7%.

    The differences do not all point in the same direction. On a PCP-related alphabet reliability task, Model B completed all three of its attempts, while Model A failed to solve the problem. This shows that different models excel at different problem types, and a single ranking cannot capture the full picture.

    More tangible results came from construction tasks. On an extremal graph problem with girth 5, Model C produced a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals. The reference value was 629 edges, so this result exceeded it. In another case, Model A proved that A(22,10) ≤ 82, tightening the published upper bound of 84 down to 82. Both kinds of results can be checked directly by programs, so their credibility is high.

    4. Conclusion: The Current Boundary of Capability

    Taken together, these results suggest dividing model capability on research tasks into two parts.

    What models can already do:

    • Complete certain proofs when the problem statement is clear, the budget is sufficient, and feedback is verifiable.
    • Optimize the performance of existing programs.
    • Improve constructions toward a given goal, such as finding better graph structures or tighter upper bounds.

    What still depends on humans:

    • Independently choosing research directions.
    • Judging the academic value and practical significance of results.
    • Solving major open problems.

    The key premise of this division is “verifiable.” Once verification conditions disappear, model outputs require substantially more human review, and the efficiency advantage declines accordingly.

    5. Practical Takeaways for Engineering Teams

    If your team plans to adopt AI-assisted research or analysis, consider the following practices.

    1. Write the checker first, then let the model generate. Task definitions should include explicit acceptance criteria; otherwise there is no way to judge whether a result is usable.
    2. Record the full attempt history. When reporting success rates, also provide the number of attempts, the number of problems covered, and failure samples—avoid presenting only the single best result.
    3. Evaluate by task type. Performance on proofs, constructions, and optimization can vary widely; merging them into one score loses useful information.
    4. Keep human review checkpoints. Before conclusions enter reports, papers, or production systems, they should be confirmed by someone with the relevant background.
    5. Control the budget and record costs. The number of attempts and compute resources directly affect results; use the same budget when comparing different systems.

    6. Limitations and Scope

    This evaluation has a limited sample size, and the problems are concentrated in mathematics and theoretical computer science. Success rates are sensitive to both problem selection and budget settings, so the conclusions should not be extrapolated directly to other research fields. In addition, the correctness of the checkers themselves requires independent verification; otherwise, a result could pass verification while still being wrong.

    Overall, given well-defined problems and verifiable feedback, AI can already take on part of the research workload. Choosing research directions, judging the value of results, and cracking major open problems still require humans today.