Verifiable Evaluation: Mapping the Research Capabilities of Large Language Models

    Verifiable evaluation changes how we assess the scientific research capabilities of large language models. Instead of asking whether an answer looks plausible, it asks whether the output can be independently checked. This article explains the core logic of verifiable evaluation, distinguishes coverage from success rate, outlines three major types of scientific tasks—theorem proving, extremal and combinatorial construction, and program or algorithm optimization—and summarizes what the resulting capability map reveals. It also offers practical guidance for building or reusing such evaluation systems, emphasizing that large models can serve as reliable research copilots while human researchers remain essential for setting directions and judging significance.

    1. Introduction: Why “Verifiable” Is the Foundation of Research Evaluation

    When we evaluate whether a model has scientific research capability, the real difficulty is not whether it can generate fluent text. The difficulty is whether its output can survive independent review. Traditional question-and-answer benchmarks mainly test whether an answer looks correct. Scientific research tasks require something stronger: whether the result is actually correct. The gap between these two requirements is the starting point of this article.

    In recent years, the academic community has increasingly placed research workflows such as theorem proving, graph construction, and algorithm optimization into verifiable evaluation systems. The core value of this approach is that the model does not merely produce a seemingly reasonable new answer. Instead, it produces an object that can be independently checked by a machine or an expert—a proof, a set of parameters, or a runnable program. This “reviewability” fundamentally changes how we measure AI-assisted scientific research capability.

    2. What Counts as an Independently Verifiable Result

    The most essential difference between scientific evaluation and traditional benchmarks lies in the constraint on output form. In mathematics and theoretical computation, a valid result usually has one of the following forms:

    • A proof that can be checked line by line by a formal verifier.
    • A constructed object that satisfies specific constraints, such as a graph structure, a combinatorial design, or an encoding scheme.
    • A program that can be compiled, run, and validated against given test cases.

    This means that judging correctness no longer depends on subjective impression. It depends on a deterministic checker. The core logic of the evaluation can be expressed with the following pseudocode:

    1. def evaluate(task, agent, budget, checker):
    2. for attempt in range(budget):
    3. result = agent.solve(task) # The model produces a candidate object
    4. if checker.verify(result): # Independent verification, not dependent on model self-assessment
    5. return success(result, attempt)
    6. return failure()
    7. # Key metrics
    8. pass_at_k = probability of at least one success within k attempts # coverage
    9. success_rate = successful attempts / total attempts # stability

    There are two important but often confused dimensions: coverage and success rate. Coverage measures whether the model can find a correct result within several attempts. Success rate measures whether the model can reproduce success consistently. A model may occasionally get a small number of tasks right through massive sampling, yet lack stable reproducibility. Therefore, a serious evaluation must report both types of metrics and explicitly state the attempt budget, such as the value of k. Otherwise, scores are not directly comparable.

    3. Three Levels of Scientific Research Tasks

    Verifiable scientific research tasks can usually be grouped into three categories. They differ in both difficulty and verification method.

    1. Theorem Proving Tasks

    These tasks require the model to construct a complete proof chain from scratch, without being given existing intermediate lemmas as hints. This type of task has the strongest verifiability: a proof either passes the checker or does not pass, with no middle ground. When designing such evaluations, it is important to specify whether hinting lemmas are allowed, because “with hints” and “without hints” measure very different levels of capability.

    2. Extremal and Combinatorial Construction Tasks

    A typical form is “construct as large an object as possible under certain forbidden-substructure constraints.” For example, under constraints such as no triangles and no quadrilaterals, the model should construct a graph with as many vertices and edges as possible. The value of these tasks lies in the fact that the result is a concrete, enumerable, and cross-verifiable object. Evaluations usually set a reference baseline value. If the model can construct an object that exceeds the baseline, this constitutes a verifiable improvement.

    3. Program and Algorithm Optimization Tasks

    These tasks require the model to produce runnable code that achieves better performance on a given dataset, or to tighten the upper bound of a combinatorial parameter—for example, improving a bound from 84 to 82. Verification for such tasks depends on the execution environment and the evaluation set. These tasks also place high demands on the clarity of computational budgets and judging criteria.

    4. The Capability Map: Strengths and Blind Spots

    When the above tasks are placed into the same evaluation system, a capability map emerges. This map is often more valuable than the simple claim that “one model leads across the board.”

    First, ranking by score does not mean complete dominance. Different models may win or lose on different task types. While one model may complete all theorem proving tasks, another model may solve the construction problem that the first model failed. This shows that scientific research capability is multidimensional. A single overall score can easily hide structural differences. Evaluation should therefore focus on the distribution of capabilities rather than a single ranking.

    Second, under clear conditions, models can already produce real results. When the problem is clearly defined, the attempt budget is sufficient, and feedback can be verified by machines, models can indeed complete some proofs, match or exceed reference baselines in construction tasks, and perform program-level optimization. These are outputs that can be independently reviewed, not merely self-reported claims.

    Third, the boundaries are equally clear. Selecting valuable research directions, judging whether a result is worth pursuing, and solving long-standing major open problems still depend heavily on the taste and judgment of human researchers. Models are good at searching within given coordinates, but they are not good at deciding which region to explore.

    5. Practical Lessons for Building a Credible Research Evaluation System

    If you want to build or reuse such an evaluation system, the following engineering practices are transferable:

    • Prefer tasks that can be verified by machines. Only when the verifier is reliable can the scores be credible. Avoid introducing stages that require subjective grading, otherwise the evaluation falls back into the old problem of “seemingly reasonable” answers.
    • Explicitly declare budgets and sampling strategies. Record the number of attempts, computational cost, and whether hints are provided to ensure that different models are compared under fair conditions.
    • Report both coverage and success rate. Use pass@k to measure whether the model can get a result right, and use success rate to measure whether it is stable. Both are necessary.
    • Use reference baselines as anchors. For construction and optimization tasks, set recognized reference values so that “improvement” becomes a quantifiable and explainable event.
    • Preserve intermediate outputs for later analysis. Record lemmas, constructed objects, or programs produced by the model. This supports cross-verification and makes it easier to analyze failure modes.

    6. Conclusion: Treat AI as a Research Copilot

    Overall, this kind of verifiable evaluation is more like an honest map of the current landscape than an exaggerated scorecard. It shows that, under clear problems, sufficient budgets, and verifiable feedback, large models can already take on some proof, optimization, and construction work. They can become reliable copilots in the research process. However, when it comes to setting research directions, evaluating the value of results, and breaking through major challenges, human researchers remain irreplaceable. Understanding the shape of this capability map is the prerequisite for making good use of this additional assistance.