Verifiable Benchmarks: A New Way to Gauge AI Research Capability

    As large language models advance rapidly, a pressing question emerges: can AI truly participate in scientific research, or merely answer questions and generate code? This article explores why traditional evaluation methods fall short in measuring research capability and examines a new class of verifiable benchmarks designed to quantify AI's scientific output objectively. Covering theorem proving, constructive optimization, and bound-tightening tasks, the article analyzes recent evaluation results that reveal a stark capability gap between leading models and the rest. It also highlights two landmark achievements—a graph construction exceeding reference baselines and a tightened combinatorial bound—while discussing current limitations and practical implications for engineering teams looking to integrate AI into research workflows.

    Why AI Research Capability Is Hard to Evaluate

    As large language models evolve rapidly, a question is drawing increasing attention: Can AI genuinely participate in scientific research—not just answer questions or generate code snippets?

    This question is difficult to answer because “research capability” itself is inherently complex. Traditional evaluation benchmarks mostly target closed tasks—selecting a clear option, finding a unique answer, or passing automated scoring. Real scientific research, however, has several fundamentally different characteristics:

    • Open-ended problems: Researchers must define problems themselves, break them into sub-problems, rather than passively receive a formatted question;
    • Long processes: A proof or an optimization often requires multi-step reasoning, where intermediate states cannot be simply scored as right or wrong;
    • Verifiable results but hard to generate: Once a mathematical theorem’s proof is given, its correctness can be checked step by step; but finding that proof in the first place is extremely difficult.

    This is precisely why the industry has begun exploring a new class of evaluation approaches: extracting the stages of the research process where “output can be independently verified by machines” and measuring them within a unified framework. The emergence of such verifiable benchmarks has, for the first time, given “AI research capability” a quantifiable, horizontally comparable landscape.

    Design Logic Behind Verifiable Benchmarks

    A well-designed research capability benchmark typically revolves around three types of tasks:

    1. Formal Theorem Proving

    The task provides a set of mathematical theorems, requiring the agent to produce complete proofs within a limited number of attempts. Proofs are typically submitted in a formal language and automatically verified by a proof checker—making scoring entirely objective, with no fuzzy space for “looks reasonable but is actually wrong.” Evaluation tracks two dimensions: the success rate on given problems, and the number of distinct problems covered. The latter distinguishes between “repeatedly getting lucky” and “consistently solving problems”—two fundamentally different capability profiles.

    2. Program and Construction Optimization

    A typical example is graph construction problems in extremal combinatorics: under specific constraints (e.g., avoiding certain substructures), approach a known optimal parameter as closely as possible. The scoring rules for such tasks are unambiguous—a constructed object either satisfies all constraints or it does not; a parameter either exceeds a reference value or it does not. Verification can be automated programmatically, making it naturally suited for machine scoring.

    3. Bound Tightening

    Given a published upper or lower bound for a certain mathematical quantity, the task requires the model to produce a tighter bound with a proof. Scoring is based on “whether the result is strictly better than the existing result,” which is likewise fully verifiable.

    One key design decision of such benchmarks: no intermediate lemmas are provided, and no hints are given. The agent must plan its proof path autonomously from the original propositions, just as a human researcher would. Additionally, each attempt typically comes with an independent compute and interaction budget, preventing results from being diluted by “unlimited retries.”

    What the Leaderboard Reveals: Capability Stratification

    Results from published evaluations show a clear, cliff-like distribution across models on the same verifiable benchmark:

    • The top-ranked model leads with approximately 72 points (out of 100), independently completing 14 of 20 theorem-proving tasks with a 70% success rate;
    • The second-ranked model scores around 57 points, succeeding 23 times across 60 attempts but covering only 9 distinct problems, with a combined success rate of roughly 38%;
    • All other evaluated agents scored no higher than 12%, a gap of an order of magnitude from the leaders.

    This distribution conveys two things. First, the top models have crossed the threshold of “being able to produce formal research output.” Second, the capability gap is not linear—it is concentrated between a very small number of frontier models and the long tail, indicating that frontier reasoning ability remains the province of a few players.

    Asymmetric Strengths: No Universal Domination

    It’s worth noting that leaderboard scores do not fully capture capability differences. The evaluation surfaced a classic “task-level reversal”:

    On a task centered on alphabet reliability, the second-ranked model succeeded on all three attempts, while the top-ranked model failed to solve it. This phenomenon carries important implications for evaluation methodology:

    • A single aggregate score masks structural differences. The two models’ areas of strength do not overlap, suggesting fundamentally different reasoning strategies and knowledge distributions;
    • Multi-dimensional reporting is essential. Beyond aggregate scores, results should be reported by task type, number of attempts, and coverage breadth; otherwise, model selection will be misled;
    • For users, “which model is stronger” is often a false question. The more accurate question is which model is more stable on their own task distribution.

    Two Landmark Results

    The most compelling findings from this round of evaluation come from both construction and proof directions:

    First, surpassing in extremal graph construction. In a classic extremal graph theory task (constructing a graph with as many edges as possible while containing no triangles or quadrilaterals), a model produced a construction with 124 vertices and 630 edges—exceeding the task’s reference value of 629 edges. The significance of such a result: it is not a selection among predetermined answers, but a new object that can be independently verified by a program and outperforms a reference baseline.

    Second, tightening a mathematical bound. Another model tightened a published upper bound for a combinatorial quantity from 84 to 82, providing a complete proof. Though the improvement is only 2, it represents AI output reaching the standard of “being testable by the mathematical community and updating known results.”

    The common thread: the models are no longer producing “plausible-sounding new answers,” but independently verifiable objects, parameters, or proofs. This is the core value that distinguishes verifiable benchmarks from traditional evaluations.

    Capability Boundaries: AI Research Remains Human-AI Collaboration

    Returning to the original question, these evaluation results read more like a topographic map of AI research capability than a declaration of the finish line. The current capability boundaries can be summarized in three points:

    1. What it can do: When problems are well-defined, budgets are sufficient, and feedback is verifiable, top models can complete some theorem proofs, construction improvements, and bound tightenings;
    2. What it cannot do: Autonomously selecting research directions, judging the academic value of results, or making sustained progress in domains lacking verification feedback;
    3. No indication of yet: Independently solving major open problems.

    In other words, AI currently plays the role of a “productive and rigorous executor,” while problem selection, taste, and value judgment remain human domains. For researchers, a sensible workflow is: humans define problems and acceptance criteria, machines scale production in high-verification-density stages, and both sides collaborate through verifiable interfaces.

    Implications for Engineering Teams

    For developers and enterprises focused on AI adoption, these verifiable benchmarks offer three transferable practical principles:

    • Prioritize scenarios with verifiable output: Tasks where correctness can be determined by programs, tests, or formal checks—code generation, formula derivation, configuration validation—are currently the areas where AI capability delivers fastest;
    • Use multiple independent attempts to hedge randomness: Evaluation data shows the same model’s success rate varies dramatically across tasks; engineering systems should design retries, ensembling, and human review mechanisms rather than relying on single-shot output;
    • Select models by task distribution, not aggregate score: Teams should build small verifiable evaluation sets based on their own business needs, measuring candidate models against real-world distributions rather than being misled by leaderboard totals.

    The significance of verifiable research benchmarks extends far beyond ranking models. They convert “can AI do research” from a debate of opinions into a reproducible, auditable engineering problem. This path is only just beginning.