Evaluating AI Research Capabilities with Verifiable Systems: A Terrain Map for Large Models

    This article examines a new evaluation paradigm that places the outputs of the full research process into a unified verifiable system, so that correctness can be checked programmatically rather than judged by text similarity or human scoring. It covers three task types—theorem proving, mathematical construction, and program optimization and reliability—and explains why independently reviewable outputs are central to suppressing hallucinated correctness. It also discusses the uneven capability distribution shown by the results, examples of new constructions that exceed reference results, the current boundaries between machine-executable research work and human-led problem selection and value judgment, and the practical implications for engineering teams building AI-assisted R&D workflows.

    Evaluating AI Research Capabilities with Verifiable Systems: A Terrain Map for Large Models

    1. Why AI Research Needs a Unified Yardstick

    Over the past few years, large models have improved rapidly on tasks such as code generation and mathematical competition problems. Yet there remains a clear gap between these results and genuine research capability. The reason is that research outputs—a proof of a new theorem, a high-performance code segment that has been optimized, or a better mathematical construction—must be independently reviewable, not merely plausible.

    Existing evaluation approaches have three systemic weaknesses:

    1. Answer leakage: benchmark problems often appear in public corpora, so a model may simply have memorized the answer.
    2. Unreliable verification: evaluation relies mainly on text similarity or human scoring, which cannot determine the strict correctness of a proof.
    3. Fragmented process: research includes problem selection, problem definition, construction, verification, and review. Most evaluations cover only one isolated step.

    A recent effort involving university research teams and a Turing Award laureate proposed a different approach: place the outputs of the entire research process into one verifiable system, so that the correctness of each task can be checked programmatically. The core contribution of this work is not the leaderboard itself, but the evaluation paradigm it establishes—replacing subjective judgment with formal, reproducible verification mechanisms.

    2. Design Logic of the Evaluation System

    2.1 Three Typical Research Stages

    The evaluation framework breaks research activity into three verifiable task types:

    • Theorem proving: Given a theorem statement and axiom system, the model must produce a complete proof, and the system verifies each reasoning step through a proof assistant or formal checker.
    • Mathematical construction: For example, finding combinatorial objects that satisfy specific constraints, such as extremal graphs, codes, or fault-tolerant structures. The verification program can directly check whether the construction satisfies all constraints and quantify how much it improves on existing results.
    • Program optimization and reliability: For algorithm implementations, the evaluation checks correctness verification and performance improvement. It examines not only whether the output is correct, but also whether the model can provide reviewable justification for key parameters.

    2.2 “Verifiable” Is the Key Word

    In traditional evaluation, a model can score by producing reasoning text that “sounds reasonable.” Under this framework, the output must be converted into an independently reviewable object—either a proof script that passes formal verification, a concrete construction that passes constraint checking, or a program that can be run repeatedly and reproduce results. This fundamentally suppresses “hallucinated correctness”: a model cannot pass by wrapping an answer in fluent language.

    This design places significant demands on engineering implementation. The system needs strict input-output protocols for each task type, automated verification pipelines, and reproducible evaluation environments. In cloud-based practice, such pipelines often rely on container platforms for environment consistency, object storage for task datasets, and logging and monitoring/alerting services to ensure traceability of the verification process. This is also a common technical path for enterprises implementing similar evaluation systems.

    3. Beyond the Leaderboard: Not Domination, but a Capability Terrain Map

    The results show a clear capability gap, but what is more valuable is the uneven distribution:

    3.1 Non-Overlapping Strengths Among Leading Models

    The top-ranked model leads the second-ranked model by more than ten points on most tasks and can complete most theorem-proving tasks even without intermediate lemma hints. However, the latter solved some specific construction tasks that the former did not. This mismatch shows that:

    • Different models vary widely in their coverage of research capabilities. A single large model does not comprehensively lead across all research subtasks.
    • Task difficulty is highly uneven. Most participating models have very low success rates on most tasks, indicating that this evaluation is far from saturated.

    3.2 The Most Convincing Result: Exceeding Existing Constructions

    Beyond the leaderboard numbers, the most intuitive results in the paper come from concrete construction tasks. One model produced a specific graph structure in an extremal graph construction task that exceeded the reference edge count while strictly satisfying short-cycle-free constraints. Another model tightened a long-standing parameter upper bound by several units. The significance of such results is that they are not restatements of training data, but new constructions that the mathematical community can directly check.

    This points to an important distinction: the value of a model is not in “giving a seemingly reasonable new answer,” but in producing outputs that can become objects of independent review. This is the paradigm shift brought by verifiable systems.

    4. The Real Boundaries of AI Research Capability

    Taken together, the evaluation results can be summarized as a “terrain map” of current large-model research capability:

    Areas already covered:

    • Under conditions of clear problems, sufficient verification feedback, and adequate compute budgets, models can complete some proof steps, program optimizations, and construction improvements.
    • Systematic search and iteration under a given objective function is a natural advantage of machines over humans.

    Areas still belonging to humans:

    • Autonomously choosing research directions: judging which problems are worth investing in requires domain intuition and long-term vision.
    • Judging the value of results: knowing whether an improvement is “important” involves understanding the development of a field.
    • Tackling major open problems: these often require conceptual breakthroughs rather than search within a given framework.

    For engineering teams, this conclusion has direct practical implications. At the current stage, it is more appropriate to position large models as verifiable execution units in research and development workflows—humans define problems and make value judgments, while models carry out iterative construction, proof, and optimization under explicit constraints. When building such collaborative workflows, priority should be given to automating and making verification reproducible; this determines overall output quality more than pursuing single-point model performance.

    5. Conclusion

    The real contribution of this work is that it provides a quantitative, reviewable framework for answering the vague question of whether AI can do research. Leaderboard rankings will continue to change as models iterate, but the methodology it establishes—replacing subjective evaluation with verifiable systems and replacing single-point testing with full-process coverage—will continue to shape AI research capability evaluation. It also offers a reference approach for enterprises building quality systems for AI-assisted R&D.