Verifiable Research Evaluation: AI Capability Boundaries from Proof to Construction

    This article examines verifiable research evaluation for large language models in scientific tasks, shifting the focus from plausible answers to independently checkable evidence. It describes a pipeline that formalizes problems, defines evidence objects such as proofs, constructions, parameters, and programs, and decouples generation from verification. It also discusses budgeted attempts, success rates, and task-level differences, using reported results to show complementary strengths rather than uniform superiority. The article concludes with engineering lessons for developers and enterprises and clarifies the boundary between verifiable capability and autonomous research.

    Verifiable Research Evaluation: AI Capability Boundaries from Proof to Construction

    1. From “Does the Answer Look Right?” to “Can the Evidence Be Reviewed?”

    When large language models enter research scenarios, the most easily overestimated capability is “generating a plausible-looking answer,” while the most easily underestimated capability is “producing evidence that can be independently reviewed.” In tasks such as mathematical proof, algorithm optimization, and program construction, natural-language answers are often misleading: a derivation chain may skip steps, a construction may satisfy only part of the constraints, or an optimization result may lack a verifiable certificate. Therefore, the key to evaluating a model’s research capability should not stop at the fluency of the answer. It should turn to verifiability: what the model provides is not text that “looks like a new answer,” but an object, parameter, program, or proof that can be independently reviewed.

    Around this idea, a verifiable research evaluation framework usually splits the research workflow into several checkable stages: problem formalization, candidate lemma generation, proof search, construction generation, program optimization, counterexample search, and final certificate submission. Each stage requires the model to output evidence that can be automatically checked or manually reviewed. For example, a proof task requires a complete lemma chain or a proof object that can be checked by a proof assistant; a construction task requires submitting vertices, edges, parameters, or program output; an optimization task requires providing constraint satisfaction before and after improvement. Only when the evidence passes review does the result count as a success.

    The value of this kind of evaluation is not to create a “leaderboard,” but to draw a capability map: on which tasks the model can stably produce reviewable results, on which tasks it can only give directional suggestions, and on which tasks humans are still needed to propose key lemmas or judge research value.

    2. How to Build a Verifiable Research Task System

    2.1 Task Normalization: Turning Open Problems into Checkable Objects

    Research problems are usually open, but evaluation must be repeatable. In engineering terms, the problem first needs to be converted into a clear input-output contract. For example, a proof task needs to define the axiom set, target theorem, allowed lemmas, and proof format; a construction task needs to define the object type, constraints, and objective function; a program optimization task needs to define correctness assertions, performance constraints, and test cases. Task normalization is not about lowering difficulty; it maps a “research workflow” to a “verifiable pipeline.”

    A general verifiable pipeline can be abstracted as follows:

    1. class ResearchTask:
    2. def __init__(self, spec, verifier, budget):
    3. self.spec = spec # formal problem description
    4. self.verifier = verifier # independent verifier
    5. self.budget = budget # attempt/time/compute budget
    6. def run_agent(self, agent):
    7. evidence = agent.propose(self.spec, self.budget)
    8. # Evidence must be an independently checkable object, parameters, or proof
    9. result = self.verifier.check(self.spec, evidence)
    10. return {
    11. "success": result.ok,
    12. "reason": result.reason,
    13. "evidence": evidence,
    14. "cost": result.cost
    15. }

    The pseudocode above emphasizes two points: first, model outputs must land on structured evidence; second, the verifier is decoupled from the generator to avoid “model self-verification.” When deployed on cloud infrastructure, object storage can persist tasks and evidence, container platforms can isolate verifier execution, and logging and monitoring can support end-to-end traceability. These general capabilities help turn evaluation from a one-time experiment into a sustainable engineering system.

    2.2 Evidence Objects: Proofs, Constructions, and Parameters

    Different tasks require different evidence objects. Theorem-proving tasks focus on lemma dependencies and proof steps; combinatorial construction tasks focus on whether a concrete object satisfies constraints; program optimization tasks focus on correctness tests and performance metrics; upper-bound improvement tasks focus on whether a new bound is strictly better than an existing bound and whether the proof chain has no gaps. The evaluation system needs to define a checker for each type of evidence and record failure reasons, such as “missing intermediate lemma,” “construction violates constraints,” “proof step cannot be reviewed,” or “upper bound is not strictly improved.”

    2.3 Budgets and Attempts: Success Rate Is More Informative Than a Single Hit

    Research is not a one-shot generation. Verifiable evaluation should allow the model to make multiple attempts within a budget and count success rate, theorems covered, and number of attempts. A single hit may come from luck; stable success across multiple attempts better reflects capability. For example, in a unified evaluation, the leading model scored 71.9 and the runner-up model scored 57.4; without providing intermediate lemmas, the leading model completed 14 out of 20 theorems, with a success rate of 70.0%, while the runner-up model succeeded 23 times in 60 attempts, covering 9 different theorems, with a success rate of 38.3%; the other six agents all stayed at or below 11.7%. These metrics show that different models differ clearly in “single-problem depth” and “multi-problem coverage,” and a single total score is not enough.

    3. Interpreting Results: Complementary Capabilities, Not Across-the-Board Dominance

    The most noteworthy point in the public results is that the leading model is not superior on every task. For example, on one type of proof-reliability task, the runner-up model completed all three attempts, while the leading model did not solve that problem. This shows that model capabilities differ across task distributions: some models are good at long-chain proof search, while others are more stable on specific constraint satisfaction and construction tasks. Evaluation should encourage showing this complementarity rather than masking differences with a single total score.

    In an extremal graph theory construction task, one model constructed a graph with 124 vertices and 630 edges containing neither triangles nor quadrilaterals, exceeding the 629-edge reference value used by the task; another model tightened a combinatorial upper bound from 84 to 82. These two types of results have different meanings: the former is a “construction improvement,” while the latter is an “upper-bound improvement.” The former needs to satisfy multiple constraints and provide a checkable object; the latter needs to provide a gap-free proof and be strictly better than the existing bound. Together they show that, under clear problems, sufficient budgets, and verifiable feedback, models can already complete some proofs, program optimization, and construction improvements.

    But the boundary must also be recognized: models still cannot autonomously choose research directions, have difficulty judging the value of results, and still rely on humans to propose key lemmas, design proof strategies, and assess research significance when facing major open problems. Verifiable evaluation provides a “capability map,” not a completion certificate for “autonomous research.”

    4. Engineering Implications: What Developers and Enterprises Can Learn

    For developers, the design ideas of verifiable research evaluation can be transferred to scenarios such as intelligent programming, formal verification, configuration optimization, and test generation. The key is not to have the model provide a final answer in one shot, but to have it output checkable intermediate artifacts: test cases, patches, constraint satisfaction reports, performance comparisons, and failure reproduction steps. For enterprises, the evaluation pipeline should be engineered: task versioning, evidence archiving, verifier isolation, budget control, failure attribution, and monitoring and alerting. Only by running the “generate-review-attribute-iterate” loop can model capabilities move from demonstration to production.

    5. Conclusion

    Verifiable research evaluation moves large language models from “text generators” toward “evidence producers.” Under clear problems, sufficient budgets, and verifiable feedback, models can already complete some proofs, construction improvements, and program optimization; however, in autonomous topic selection, value judgment, and solving major open problems, humans remain the primary drivers of the research workflow. The key in the future is not simply pursuing a higher total score, but establishing finer-grained evidence standards, more transparent failure attribution, and more reliable human-AI collaboration mechanisms.