Verifiable Research Evaluation Systems and the Boundaries of AI Proving Ability

    Evaluating AI's research capability requires verifiable outputs rather than plausible-sounding text. This article explains how to design a verifiable research evaluation system—covering task contracts, objectified artifacts, layered verification, experimental controls, and multi-dimensional scoring—and uses anonymized benchmark results to illustrate how such systems reveal structural capability differences between models and define where human researchers remain essential.

    Verifiable Research Evaluation Systems and the Boundaries of AI Proving Ability

    1. From “Does the Answer Look Right” to “Can the Result Be Independently Verified”

    The hard part of evaluating AI’s research capability is not getting a model to produce professional-looking text—it is determining whether the model has actually produced a research artifact that can be independently verified. Traditional Q&A benchmarks rely on text similarity, human preference, or reference-answer matching, which makes it easy to mistake “plausibly phrased” for “correct.” For tasks such as mathematical proving, algorithmic construction, and program optimization, a better approach is a verifiable research evaluation system: a framework that places the key outputs of the research process into something checkable, reproducible, and auditable.

    The core of such a system is not asking the model to “answer the question again,” but requiring it to submit explicit artifacts: a proof that can be checked step by step, a serializable graph structure, a computable parameter, a runnable optimization program, or a construction that satisfies stated constraints. A verifier then independently judges whether the artifact holds, based on the task contract. Only when outputs can be independently rechecked—by machines or experts—does evaluation shift from a “language ability test” to a “research output test.”

    Anonymized results from related research illustrate the value of this approach. On the overall leaderboard, Model A ranked first with 71.9 points and Model B second with 57.4 points. Without intermediate lemmas, Model A completed 14 out of 20 theorems (a 70.0% success rate), while Model B succeeded 23 times across 60 attempts, covering 9 distinct theorems (a 38.3% success rate). The remaining six agents all stayed at or below 11.7%. These numbers do not represent a simple sweep: on the PCP alphabet reliability task, Model B completed all three attempts, while Model A failed to solve it at all. Verifiable evaluation reveals structural differences in capability, not a single-dimensional ranking.

    2. Design Principles for a Verifiable Research Evaluation System

    A practical verifiable research evaluation system should include at least five components: task contracts, objectified artifacts, layered verification, experimental controls, and multi-dimensional scoring.

    First, the task contract must clearly specify inputs, outputs, and acceptance criteria. A task should not simply say “prove this conclusion” or “improve this construction”; it should define the output format, allowed budget, number of attempts, whether intermediate lemmas are provided, timeout limits, and scoring rules. Otherwise, models are compared under unequal conditions and the results lack explanatory power.

    Second, artifacts must be objectified. A proof should have checkable steps or an executable proof script; a graph construction should specify vertex and edge sets; parameter optimization should provide recomputable values and the computation process; program optimization should run and reproduce its metrics. Only objectified artifacts let a verifier judge independently, rather than relying on another model’s “sense” of correctness.

    Third, the verifier should be layered. The lowest layer performs syntax and type checks—for example, whether an edge list is valid or vertex indices are out of range. The middle layer checks constraints—for example, whether a graph contains no triangles or 4-cycles, or whether a proof step uses an undeclared lemma. The top layer performs independent rechecking and adversarial testing—for example, random re-runs, boundary cases, and counterexample search. Layered verification distinguishes “format errors,” “constraint violations,” “incomplete proofs,” and “wrong conclusions,” providing a basis for failure attribution.

    Fourth, experimental controls must be reproducible. Whether intermediate lemmas are provided, how many attempts are allowed, how much compute budget is given, and whether random seeds are fixed all significantly affect success rates. In the referenced results, “no intermediate lemmas” is a key control variable: it raises task difficulty and better reflects independent exploration. Likewise, “23 successes in 60 attempts, covering 9 theorems” reminds us that evaluation should consider not just single-attempt success rates, but also attempt cost and theorem coverage.

    Fifth, scoring should be multi-dimensional. It should include at least final correctness, independent verifiability rate, average attempts, number of tasks covered, failure type distribution, and human review cost. A model may score low overall yet excel on certain subtasks, or lead overall yet fail on specific tasks. Multi-dimensional scoring produces a “capability terrain map.”

    Illustrative pseudocode:

    1. task = {
    2. id: "graph-girth-5",
    3. statement: "Construct a triangle-free, 4-cycle-free graph maximizing the number of edges",
    4. output_schema: {vertices: "int", edges: "list[(int,int)]"},
    5. verifier: [
    6. "check_simple_graph",
    7. "check_no_3_cycle",
    8. "check_no_4_cycle",
    9. "count_edges"
    10. ],
    11. budget: {attempts: 60, timeout_s: 600},
    12. hints: {intermediate_lemma: false}
    13. }
    14. result = solver.solve(task)
    15. report = verify(task, result.output)
    16. score = score_task(task, report, result.cost)
    17. audit_log.append({task, result, report, score})

    This pseudocode captures the essentials of a verifiable system: tasks have contracts, artifacts have structure, verification has steps, scoring has a basis, and the process has an audit log.

    3. Capability Differences Behind the Anonymized Results

    At the leaderboard level, Model A’s 71.9 points versus Model B’s 57.4, with the other six agents at or below 11.7%, shows a clear divergence between leading and average models. But the subtask structure is more informative.

    On theorem-proving tasks without intermediate lemmas, Model A completed 14 of 20 theorems (70.0%), while Model B succeeded 23 times in 60 attempts, covering 9 distinct theorems (38.3%). The former demonstrates strong per-attempt completion ability; the latter demonstrates search ability—covering different theorems through repeated attempts. These are not strengths and weaknesses on the same dimension: one model is better at producing verifiable proofs within fewer attempts, while the other may depend more on attempt budget and search strategy.

    The PCP alphabet reliability task further shows that “first place overall” does not mean “best at everything.” Model B completed all three attempts while Model A never solved the task. This demonstrates that verifiable evaluation can expose model blind spots on specific formal tasks, and it warns practitioners not to let a single aggregate score replace task-level analysis.

    The two most striking results in the paper are both independently verifiable. On a classic extremal graph task, Model B constructed a graph with 124 vertices and 630 edges containing no triangles or 4-cycles, exceeding the task’s 629-edge reference value. Model A proved A(22,10) ≤ 82, tightening the published upper bound from 84 to 82. These results matter not because the prose is novel, but because the construction can be checked, the parameter recomputed, and the proof re-examined. The models produced not “plausible-sounding new answers” but objects, parameters, and proofs that can be independently audited.

    4. The Capability Terrain Map of AI Research and the Human–Machine Boundary

    Taken together, a more accurate assessment is this: AI can already complete some proofs, program optimizations, and construction improvements given well-defined problems, sufficient budget, and verifiable feedback. But autonomously choosing research directions, judging the value of results, and solving major open problems still require humans.

    Specifically, models show strong search ability within closed loops of “given goal, given acceptance criteria, unlimited retries.” They remain weak in open-ended phases that require proposing new conjectures, deciding which directions are worth pursuing, and recognizing long-term value. Verifiable systems are good at decomposing research problems into checkable subtasks; they are not good at replacing scientists in defining what is worth studying. The sensible engineering practice is therefore human–machine collaboration: humans handle problem definition, acceptance criteria, and value judgments; models handle candidate constructions, proof drafts, program optimization, and counterexample search; verifiers handle independent rechecking; and audit logs handle reproduction and attribution.

    For developers and enterprise users, the priority when implementing this approach is building a verifiable task library, unified artifact formats, automated verifiers, and audit logs. For research teams, open problems should be decomposed into verifiable subtasks, with budget, attempt counts, and hint conditions treated as control variables. For evaluation platforms, reporting should include total score, coverage, attempt cost, and failure types together, avoiding the illusion of a “total sweep” created by a single leaderboard. The significance of verifiable research evaluation systems lies in turning AI research capability from a vague impression into an auditable terrain map—one that clearly marks the boundaries where humans must remain in charge.