AI Research Capability Map: What Verifiable Evaluations Reveal

    This article explains how to evaluate AI systems on research tasks when research outputs are hard to check. It covers three elements of a reliable evaluation design: precise task definitions, a fixed attempt budget, and automated verification. It walks through a checker for a graph construction task, explains how to read ranking results alongside per-problem outcomes and failure cases, and outlines where current models perform well and where human judgment is still essential. It closes with practical steps for research and engineering teams.

    Why Research Tasks Need Verifiable Evaluation

    Research outputs are often hard to judge quickly. A proof or conclusion that looks plausible may take an expert several days to review. For that reason, the most useful design for evaluating research-oriented AI is to make every output an object that can be checked independently, such as a graph, a set of parameters, a runnable program, or a formal proof.

    When an output is an object, a checker can determine whether it satisfies the conditions of the problem. The judgment does not depend on the reader’s subjective impression. This approach separates “sounds correct” from “is correct,” and it makes results from different teams comparable.

    Three Elements of an Evaluation Design

    A reliable research evaluation usually has three elements.

    1. A precise task definition. The problem must state the input, the constraints, and the goal. For example, “Construct a graph with a fixed number of vertices that contains no triangles or 4-cycles, with as many edges as possible” is far easier to judge than “Study this class of graphs.”
    2. A sufficient attempt budget. When comparing systems, the number of attempts per problem, the time window, and the prompting conditions must be the same. Otherwise, score differences may only reflect differences in resources.
    3. Automated, verifiable feedback. The checker should reach a conclusion without relying on an explanation from the submitter.

    Take a construction task as an example. The checker can first reject any graph that contains a triangle or a 4-cycle, then count the edges. The following is an illustrative checking logic, intended only to explain the idea:

    1. from itertools import combinations
    2. def build_adj(n, edges):
    3. adj = {v: set() for v in range(n)}
    4. for u, v in edges:
    5. if u == v:
    6. raise ValueError("Self-loop detected")
    7. adj[u].add(v)
    8. adj[v].add(u)
    9. return adj
    10. def has_triangle(adj):
    11. # An edge u-v forms a triangle if its two endpoints share a common neighbor
    12. return any(adj[u] & adj[v] for u in adj for v in adj[u])
    13. def has_4cycle(adj):
    14. # Two distinct vertices that share at least two common neighbors form a 4-cycle
    15. return any(len(adj[a] & adj[c]) >= 2 for a, c in combinations(adj, 2))
    16. def check_candidate(n, edges, reference_edges):
    17. adj = build_adj(n, edges)
    18. if has_triangle(adj) or has_4cycle(adj):
    19. return "Invalid"
    20. if len(edges) > reference_edges:
    21. return "Valid, exceeds the reference value"
    22. return "Valid, does not exceed the reference value"

    The value of this kind of checker is that anyone can reproduce its result. It also limits the room for a system to justify its own answers.

    How to Read the Results

    The source material reports a ranking. The top-ranked system completed 14 of 20 theorems without being given intermediate lemmas, for a success rate of 70.0%. The second-ranked system succeeded 23 times in 60 attempts, covering nine distinct theorems, for a success rate of 38.3%. The remaining six systems each had success rates of no more than 11.7%.

    An overall ranking does not represent performance on every problem. On one reliability-related proof task, the second-ranked system completed the proof in all three attempts, while the top-ranked system did not solve it. Readers should look at per-problem results, attempt counts, and failure cases together to judge which system suits which kind of work.

    The following figures come from the source material and have not been independently reproduced. Cite them with that caveat.

    • On an extremal graph problem with girth 5, one system constructed a graph with 124 vertices and 630 edges that contains no triangles or 4-cycles. The reference value for this task is 629 edges, so the result exceeds it.
    • Another system proved that A(22,10) ≤ 82, tightening the published upper bound of 84 to 82.

    Both results are verifiable as objects. The first can be checked directly with the logic above. The second requires a mathematical expert to review the proof.

    Current Capability Boundaries

    Based on this set of evaluations, models already show useful performance in the following areas:

    • Filling in parts of a proof when the goal is clear and the success criteria are well defined;
    • Improving the performance of a program under a given metric;
    • Improving the parameters of a combinatorial construction under known constraints.

    The following tasks still depend heavily on humans:

    • Independently choosing a research direction that is worth pursuing;
    • Judging whether a result is important or novel;
    • Solving major open problems.

    At this stage, AI is better suited to clearly defined subtasks. Researchers still need to make the decisions about which problems to pursue and what results are worth.

    Practical Recommendations for Developers and Research Teams

    If your team plans to introduce a similar research assistance workflow, consider the following practices.

    • Rewrite research questions as testable specifications. Define the input, constraints, objective function, and acceptance criteria explicitly.
    • Design a checker for each output type. Use constraint checks for construction tasks, formal tools or independent review for proof tasks, and fixed benchmarks for optimization tasks.
    • Record all attempts, not only the best result. Report success rates, failure modes, and the number of problems covered so that different approaches can be compared fairly.
    • Standardize the evaluation protocol. Run the same problem set with the same budget, the same prompts, and the same time window to avoid distorted comparisons.
    • Keep human review in the loop. Have domain experts review problem selection, result interpretation, and anything published externally.

    The usefulness of a research assistance tool depends on whether the task is verifiable, whether the evaluation is reproducible, and how humans oversee the process.