A Verifiable Research Evaluation Framework: Charting the Boundaries of AI Theorem-Proving Capability

    This article examines the design principles behind verifiable evaluation frameworks for AI research assistants, focusing on theorem proving, program optimization, and combinatorial construction. It outlines a four-component architecture—task definitions, budget constraints, feedback mechanisms, and deterministic validators—and argues that evaluation must center on independently checkable artifacts rather than natural-language explanations. Real-world evaluation data and representative results are used to illustrate how progress can be measured, where capabilities remain limited, and how engineering teams can build auditable verification pipelines in practice.

    From “Plausible Explanations” to Independently Verifiable Outputs

    Discussions about AI participating in mathematical research, program optimization, and combinatorial construction often fall into a common fallacy: treating “a plausible-sounding explanation” as equivalent to “a verifiable research output.” Natural-language explanations are readable, but readability does not imply checkability. A proof must have its lemma chain, variable scope, and inference rules examined. An optimized program must have its constraints, objective function, and execution results checked. A graph construction must have its vertex count, edge count, and forbidden-subgraph properties verified.

    The first principle of a professional evaluation framework, therefore, is that a model’s output is not its final “opinion” but a set of objects, parameters, or proof certificates that an independent verifier can check. Relevant outputs include:

    • Formalized proof scripts or checkable proof terms;
    • Programs with constraint specifications and performance parameters;
    • Vertex sets, edge sets, or adjacency lists for graph constructions;
    • Constructive certificates and inequality derivations in upper-bound proofs;
    • Reproducible inputs, configurations, and output records for experiments.

    Only when explanations are demoted to auxiliary information and verifiable artifacts become the core of evaluation can AI research assessment move from subjective scoring to engineering-grade measurement.

    Key Components of the Evaluation Framework: Tasks, Budgets, Feedback, and Validators

    A verifiable research evaluation framework typically requires four core components: task definitions, budget constraints, feedback mechanisms, and validators. All four are necessary.

    1. Task Definitions Must Have Clear Decision Boundaries

    A task cannot simply say “prove a conjecture” or “optimize a program.” It must specify inputs, output formats, permitted tools, success criteria, and a reference baseline. For construction tasks, one should specify maximum vertex counts, edge-count targets, and forbidden subgraph types. For proof tasks, one should specify whether intermediate lemmas are allowed, whether formal checking is required, and whether conditional proofs are accepted.

    2. Budget Constraints Determine Comparability of Capability Upper Bounds

    The number of attempts, computation budgets, tool invocation counts, and termination conditions must be harmonized across different systems. Otherwise, a “higher success rate” may simply reflect a more generous budget rather than stronger reasoning. Evaluations should record the budget consumed by each attempt and compare success rates, task coverage, and verification costs under identical budgets.

    3. Feedback Mechanisms Must Be Verifiable

    Feedback should not come from a “judge’s impression.” It should be the deterministic result returned by a validator. For construction tasks, the validator can check whether a graph contains triangles or quadrilaterals. For proof tasks, the validator can invoke a proof-checking procedure. For optimization tasks, the validator can check constraint satisfaction and objective values.

    4. Validators Take Priority over Scorers

    Scorers can handle multi-dimensional metrics, but validators must be independent, deterministic, and reproducible. The following pseudocode describes a generic evaluation pipeline that does not depend on any specific product:

    1. Define Task:
    2. Task ID
    3. Problem statement
    4. Budget: max attempts, max steps, time limit
    5. Validator: takes artifact, returns pass/fail
    6. Reference baseline: optional
    7. Define Evaluate(Task, Proposer):
    8. trajectory = empty list
    9. for attempt in 1..Task.Budget.MaxAttempts:
    10. artifact = Proposer(Task.ProblemStatement, trajectory)
    11. trajectory.append({attempt, artifact_type})
    12. if Task.Validator(artifact) == pass:
    13. return {status: "verified", artifact, attempts, trajectory}
    14. return {status: "not_verified", attempts: Task.Budget.MaxAttempts, trajectory}

    This pseudocode emphasizes three points. First, validators check artifacts, not explanations. Second, the full trajectory of attempts is retained. Third, success is determined by a deterministic validator, not by the model’s self-assessment or human impression.

    Interpreting Results: Aggregate Scores and Local Counterexamples

    In a unified leaderboard, the leading system achieves an overall score of 71.9, while the second-place system scores 57.4. In a setting without intermediate lemmas, the leading system proves 14 out of 20 theorems, a 70.0% success rate. The second system succeeds 23 times across 60 attempts, covering 9 distinct theorems, for a 38.3% success rate. The remaining six evaluated agents all score below 11.7%. These figures indicate that frontier reasoning systems, when given well-defined problems, sufficient budgets, and verifiable feedback, can complete a meaningful portion of proofs, program optimizations, and construction improvements.

    But the evaluation results are not simply a story of one system dominating everywhere. On an alphabet-reliability task related to probabilistically checkable proofs, the second-place system succeeded on all three attempts, while the leading system failed to solve the problem. This local counterexample is important because it reveals three facts:

    1. Aggregate scores cannot substitute for task-family analysis. Different tasks demand different reasoning patterns—some emphasize construction search, others proof-chain organization, and still others constraint satisfaction.
    2. Attempt budgets affect conclusions. A system may solve a particular task within a few attempts but lag behind another system in multi-theorem coverage.
    3. Coverage diversity matters more than single-point success. Succeeding 23 times while covering 9 distinct theorems indicates a degree of generalization. Solving all attempts on a single task indicates stability within that task family.

    Evaluation reports should therefore present overall scores, task-family scores, success rates, theorem coverage counts, attempt counts, and verification costs together, rather than a single leaderboard rank.

    Two Representative Results: Construction Improvements and Tighter Upper Bounds

    The most tangible results in the paper come from two task categories: combinatorial construction and upper-bound proving.

    The first category is extremal graph construction. In one classical extremal graph theory task, an evaluated system constructed a graph with 124 vertices and 630 edges, containing neither triangles nor quadrilaterals, exceeding the reference value of 629 edges. The value of this result is not that the number is one higher, but that it provides an independently checkable construction: the validator can walk through the edge list, confirm the vertex count and edge count, and verify the absence of triangles and quadrilaterals. If the construction is legal, the improvement is objective.

    The second category is upper-bound tightening. An evaluated system produced an upper-bound proof for a combinatorial extremal function, refining the bound from 84 to 82, namely (A(22,10) \le 82). The verification object here is not a “reasonable-looking” explanation but a checkable inequality derivation or certificate. A tightening of even 2 units means the proof chain must exclude new counterexample regions, which places high demands on reasoning rigor.

    Together, these two examples show that credible AI-assisted research outputs should be organized around verifiable objects. For construction tasks, the verification object is a graph or an edge set. For proof tasks, it is a proof certificate. For optimization tasks, it is a program with its parameters. Only when the verification object holds can the result enter the research workflow.

    Capability Boundaries: What Can Be Automatized and What Still Requires Humans

    Returning to the original question, this work reads more like a terrain map of AI research capability. Models can already complete parts of proofs, program optimizations, and construction improvements under well-defined problems, sufficient budgets, and verifiable feedback. But autonomously selecting research directions, judging the value of results, and solving major open problems still require human involvement.

    The automatable components include:

    • Searching for constructions under explicit constraints;
    • Filling in partial proofs within a given lemma system;
    • Performing local optimizations on programs and submitting verifiable parameters;
    • Running preliminary self-checks and failure retries on candidate results;
    • Generating auditable attempt trajectories for human review.

    The components that still depend on humans include:

    • Deciding whether a problem is worth studying;
    • Choosing appropriate formal representations and verification standards;
    • Assessing the novelty and academic value of results;
    • Handling open-ended problems that validators cannot cover;
    • Deciding when to stop searching and when to switch direction.

    Engineering teams should position AI as a “candidate generator inside a verifiable research pipeline,” not as an “autonomous research decision-maker.” Humans are responsible for objective setting, validator design, and value judgment. The system is responsible for generating checkable artifacts within a budget.

    Engineering Recommendations: Building an Auditable Verification Pipeline

    To translate the evaluation approach described above into team practice, the following five steps are recommended:

    1. Write the validator first, then the task. Tasks without deterministic validators should not enter the automated scoring pipeline.
    2. Standardize budgets and attempt counts. Record maximum attempt counts, step counts, time limits, and tool invocation counts to ensure horizontal comparability.
    3. Structure artifacts explicitly. Require outputs such as edge sets, parameter tables, proof certificates, or program files, rather than natural-language explanations alone.
    4. Retain complete trajectories. The inputs, artifacts, validation results, and failure reasons for every attempt should be stored for audit purposes.
    5. Use multi-dimensional scoring. Report overall scores, task-family scores, success rates, coverage counts, average attempt counts, and verification costs together.

    In summary, the core value of a verifiable research evaluation framework is not to produce a higher ranking but to establish a reproducible, auditable, and measurable method for capability assessment. It makes AI’s partial capabilities in theorem proving, program optimization, and construction improvement measurable, and it gives human researchers a clearer picture of which work can be delegated to systems and which judgments must remain with people.