Evaluating AI Mathematical Proofs and Verifiable Research Workflows: A Practical Guide
How should we evaluate AI systems on scientific research tasks? This article explores a verifiable evaluation pipeline for AI-driven mathematical proving, combinatorial construction, and program optimization—shifting the focus from "generated answers" to "independently checkable artifacts." It covers layered evaluation metrics, real benchmark results from leading systems, two notable case studies, and engineering best practices for building verifier-first evaluation infrastructure.
Evaluating AI Mathematical Proofs and Verifiable Research Workflows: A Practical Guide
From “Generating Answers” to “Submitting Checkable Artifacts”
When AI participates in research tasks, the most dangerous outcome is not an obvious failure, but a fluent, well-formed “new answer” that cannot be verified. In domains such as theorem proving, combinatorial construction, program optimization, and coding theory, genuinely valuable output should be an independently inspectable object: a formalized proof, an executable program, a graph structure, a parameter certificate, a bound construction, or a counterexample.
Therefore, an evaluation system designed for research capabilities should not merely judge whether the final text looks like a conclusion. Instead, it should bring the key artifacts of the research process into a single verifiable pipeline. A reasonable abstraction looks like this:
- Problem definition: specify the target theorem, constraints, allowed lemmas, and budget.
- Artifact format: require proof scripts, construction data, program source code, certificates, or parameter tables as submissions.
- Verifiers: validate proofs with independent checkers, programs with test cases, and constructions with structural validators.
- Attempt traces: record intermediate lemmas, failure reasons, and resource consumption for every attempt.
- Scoring criteria: distinguish between complete solutions, solutions without intermediate lemmas, partial coverage, attempt success rates, and the number of theorems covered.
The process can be summarized in pseudocode:
def evaluate(task, solver, budget):trace = []for attempt in range(budget.max_attempts):artifact = solver.solve(task.prompt, budget.remaining)trace.append(artifact)if verifier.check(task, artifact):return {status: "verified", artifact: artifact, trace: trace}return {status: "unverified", trace: trace}
The core question here is not “what did the model say,” but “whether the verifier can accept the object it submitted.”
Evaluation Metrics Should Be Layered
Relying on a single overall ranking can mask meaningful capability differences. A better evaluation framework should include at least the following layers:
- Overall leaderboard score: aggregate performance across multiple tasks, useful for observing capability ceilings.
- Completion rate without intermediate lemmas: require the model to prove theorems without hint lemmas, measuring its ability to make independent progress.
- Attempt success rate: how many successes occur within a fixed number of attempts, measuring stability and search efficiency.
- Theorem coverage: the number of distinct theorems solved, avoiding repeated success on a single task.
- Reliability task pass rate: for example, proof-checking or alphabet soundness tasks, revealing whether a model attempts to “bypass” verification.
- Construction improvement magnitude: in combinatorial construction and program optimization, whether the system surpasses reference values or existing upper bounds.
In published results, System A ranked first on the full leaderboard with 71.9 points, while System B placed second with 57.4. System A solved 14 out of 20 theorems (a 70.0% success rate) without provided intermediate lemmas; System B succeeded 23 times in 60 attempts, covering 9 distinct theorems, for a 38.3% success rate. The remaining six agents did not exceed 11.7%. These figures show that top systems already possess partial independent proving capability—but their capability profiles differ significantly.
| Observation Dimension | System A | System B | Other Systems |
|---|---|---|---|
| Full leaderboard score | 71.9 | 57.4 | Lower |
| Completion without intermediate lemmas | 14/20, 70.0% | 23/60, 38.3% | Not prominent |
| Theorem coverage | Not listed | 9 | Not listed |
| Reliability tasks | Failed to solve a PCP soundness task | Completed in all three attempts | Not prominent |
| Construction/proof highlights | Tightened a coding-theory upper bound to 82 | Extremal graph with 124 vertices and 630 edges | Not prominent |
It is worth emphasizing that this is not a case of one system simply dominating across the board. System B completed a PCP alphabet soundness task in all three attempts, while System A failed to solve it. This confirms that evaluation cannot rely on total scores alone—it must consider task types, verification methods, and failure modes.
Two Representative Cases: Construction Improvements and Tighter Bounds
The most tangible results came from two categories of tasks.
The first is extremal graph construction. In an extremal graph task with girth 5, System B constructed a graph with 124 vertices and 630 edges containing no triangles or 4-cycles, exceeding the task’s reference value of 629 edges. The value of this result lies in the artifact itself: a verifiable graph structure whose edge count, vertex count, and forbidden subgraphs can all be checked by an independent program, without relying on the model’s own claims.
The second is tightening an upper bound in coding theory. System A proved that a certain coding-theory parameter upper bound could be tightened from 84 to 82. Upper-bound proofs require a complete logical chain—simply stating a conclusion is not enough; the verifier must check every derivation step or the final certificate. Such results show that, given a well-defined problem, a sufficient budget, and verifiable feedback, models can already achieve partial proofs, program optimizations, and construction improvements.
That said, these achievements remain “local advances on given problems.” Choosing research directions autonomously, judging the value of results, and tackling major open problems still depend on human judgment.
Engineering Practice: Put the Verifier First
For enterprises or research teams building similar capabilities, the guiding engineering principle should be “verifier first”:
- Define verifiable artifacts first: don’t start with natural-language reports; start by defining proof scripts, construction files, program interfaces, and certificate formats.
- Isolate the execution environment: run model-generated programs inside containerized sandboxes with restricted network access, file systems, and resource quotas.
- Preserve complete traces: record the input, output, verification result, runtime, and failure reason of every attempt for auditability and post-mortem analysis.
- Score in layers: break total scores down into complete solutions, partial solutions, attempt success rates, coverage counts, and reliability task pass rates.
- Manually review high-value conclusions: for results such as tightened bounds or improved constructions, arrange for domain experts to review the problem modeling and the verifier itself.
On general-purpose cloud platforms, container services can handle isolated execution, object storage can hold proof scripts and construction data, logging services can record attempt traces, and monitoring and alerting can control budgets and catch anomalies. General cloud platforms such as Baidu AI Cloud can provide these baseline capabilities—but the critical factor is the design of the verification pipeline, not simply piling up compute.
Boundaries and Conclusion
Current AI research capabilities are best understood as a “topographic map”: under conditions of well-defined problems, sufficient budgets, and verifiable feedback, models can complete partial proofs, program optimizations, and construction improvements. But in autonomous problem selection, value judgment, and attacking major open problems, human researchers must still lead. The purpose of an evaluation system is not to manufacture a narrative of “total supersession,” but to turn model outputs into independently checkable objects, parameters, or proofs—so that every success and failure can be examined and reused.
