From Leaderboard Scores to Research-Grade Evaluation: What AI Can—and Cannot—Do in Theory

    A new benchmark for theoretical computer science evaluates AI across formal theorem proving, solver engineering, mathematical construction, and open conjectures. Its results show that advanced models can handle many machine-verifiable tasks, while still failing to independently solve problems that remain open to human researchers. The findings point toward AI systems built around formal specifications, automated verification, reproducible workflows, and clearly defined human oversight.

    From “Benchmark Chasing” to Research-Grade Evaluation: AI Needs a Harder Standard

    AI evaluation in mathematics and theoretical computer science has long faced an awkward problem: benchmark datasets are repeatedly reshuffled, most questions already have human-proven solutions, and common solution patterns are highly visible in training data. If a model has seen similar problem types, it may achieve impressive leaderboard scores.

    That kind of evaluation can be useful for technical demonstrations, but it provides limited insight into real research performance. Research problems often have no standard answer or even a clearly defined path to a solution. Failure is normal.

    A recently published preprint proposes a benchmark designed to more closely reflect the research environment. It evaluates 100 theoretical computer science tasks across four categories: proving established theorems, implementing algorithms and solvers, constructing mathematical objects that satisfy constraints, and addressing open conjectures that humans have not yet solved.

    The key contribution is not the number of tasks, but the task design and scoring mechanisms. The benchmark shifts the focus from “Can the model solve familiar problems?” to “Can it make progress on research-level problems under verifiable conditions?”

    Four Task Categories and the Difficulty Gradient

    The benchmark divides its 100 tasks into four categories, each targeting a different capability.

    1. Formal Theorem Proving

    There are 20 formal theorem-proving tasks. Each provides a mathematical statement and requires the model to produce a proof that can be checked by a machine.

    The central challenge is not explaining the proof in natural language. The model must express it as a formal script that a proof kernel can verify step by step. This tests mathematical intuition, proof search, familiarity with formal languages, and the ability to identify and repair errors.

    2. Solver Engineering

    The benchmark includes 30 solver-engineering tasks. Models must write or improve algorithms and solver programs that actually solve computational problems.

    This category is closer to software engineering than to conventional question answering. A model must not only develop a plausible idea but also produce code that runs reliably under test cases, resource constraints, and correctness checks. The tasks expose weaknesses in complex implementation, edge-case handling, and performance optimization.

    3. Mathematical Construction

    Another 20 tasks require the model to construct mathematical objects that satisfy a specified set of conditions.

    Compared with theorem proving, construction problems place greater emphasis on designing a search space, satisfying constraints, and eliminating counterexamples. Compared with engineering tasks, they demand a stronger understanding of abstract mathematical structures.

    4. Open Conjectures

    The remaining 30 tasks address theoretical problems that are still unresolved by humans.

    These tasks have no known answer and no proof template that a model can simply retrieve. The model may need to propose a candidate strategy, define auxiliary concepts, identify verifiable intermediate lemmas, or acknowledge that a particular approach is unlikely to work.

    The difficulty rises from proving known theorems to challenging unknown conjectures. This progression also makes the evaluation results more informative: success on a formalized problem and progress toward an open problem represent fundamentally different capabilities.

    Evaluation: Let Machines Decide What Can Be Verified

    One of the benchmark’s most important design choices is its effort to minimize subjective scoring.

    Human grading is expensive and inconsistent. Using another language model as a judge can introduce mutual praise, stylistic bias, and other forms of unreliable evaluation. Instead, the benchmark connects formal proofs, program correctness, and constraint-satisfaction tasks to automated verification tools.

    If a proof or program passes the relevant checker, it passes the task. If it fails, the system reports an error. This reduces ambiguity.

    The evaluation pipeline can be summarized with the following generic pseudocode:

    1. def evaluate(task, model):
    2. output = model.generate(task.prompt)
    3. if task.kind == "formal_proof":
    4. return formal_kernel.verify(task.statement, output.proof)
    5. if task.kind == "solver_engineering":
    6. return run_autotests(output.code, task.test_cases)
    7. if task.kind == "construction":
    8. return check_constraints(output.object, task.constraints)
    9. if task.kind == "open_conjecture":
    10. return machine_assisted_review(output.attempt, task.criteria)

    The first three categories can be evaluated with a relatively high degree of automation. Open-conjecture tasks are more difficult to assess fully automatically and require a combination of machine-assisted checking and expert review.

    The principle is straightforward: automate verification wherever possible, and clearly label the parts that cannot be fully automated. This is closer to the complexity of real scientific evaluation than a leaderboard based solely on a single aggregate score.

    Interpreting the Results: Around 70% on Theorems, Zero on Open Conjectures

    The reported results show that one frontier model achieved a success rate of approximately 70% on research-level theorem-proving tasks. However, it independently solved none of the 30 open conjectures.

    This contrast is highly informative.

    Established theorem-proving tasks are difficult, but their statements, definitions, and proof strategies often appear in public mathematical literature and formal libraries. A model can use pattern matching, lemma retrieval, proof search, and iterative correction to approach a solution.

    A success rate near 70% suggests that large models already provide useful assistance on proof tasks with clear objectives, machine-checkable outputs, and opportunities for iterative repair.

    Open conjectures require more than proof technique. They require the ability to propose new directions, formulate useful intermediate claims, and judge which paths are worth pursuing. This resembles research topic selection and breakthrough discovery more than solving a problem within a known framework.

    A zero-shot breakthrough does not mean that models have no value. It indicates that current systems are better at searching within a problem space defined by humans than at independently creating and exploring a new problem space.

    For developers, the practical implication is to position these models as candidate generators and verification assistants within a verifiable workflow—not as autonomous scientists.

    Implications for Developers and Enterprises: Build Verifiable AI Workflows

    The broader value of this benchmark is that it suggests how engineering teams can build more trustworthy AI systems.

    First, tasks should be decomposable. Large problems can be divided into definitions, lemmas, proofs, tests, counterexamples, and other units that can be checked independently.

    Second, specifications should be formalized. Whenever a constraint can be checked by a machine, it should not rely solely on a natural-language description.

    Third, workflows should be reproducible. Teams should use fixed environments, fixed test sets, and fixed verification tools, while recording every intermediate artifact generated during the process.

    Fourth, human–AI responsibilities should be explicit. The model can generate candidates and support rapid experimentation, while experts determine research direction and perform final acceptance.

    In production environments, enterprises can build evaluation and research pipelines on general-purpose cloud infrastructure. Container platforms can isolate execution environments; object storage can retain task definitions, proof scripts, and runtime logs; logging and monitoring services can track failure causes; and message queues can schedule batch evaluation jobs.

    Organizations using platforms such as Baidu AI Cloud should likewise focus on general capabilities including containers, storage, logging, and monitoring to create reproducible workflows. Evaluation should not depend on an unauditable “black-box” score.

    Limitations and the Road Ahead: From Exams to Research Assistants

    The benchmark still has limitations. Its task set is relatively small, open-conjecture evaluation cannot be fully automated, and static datasets may eventually be indirectly adapted to by future models.

    Future research could explore dynamic task sets, process-level scoring, interactive proof repair, multi-agent collaboration, tool use, and evaluation of long-term research trajectories.

    A genuinely useful AI research system must do more than produce an answer. It should preserve verifiable reasoning traces, failed attempts, and reproducible experimental paths.

    The conclusion is clear: AI can already perform some demanding, verifiable support tasks in theoretical computer science, but it remains far from independently solving open problems.

    For developers, the most practical path is not to pursue the narrative of an all-purpose model. It is to combine formal specifications, automated verification, reproducible engineering, and human–AI collaboration into a trustworthy workflow.

    Only through this combination can AI gradually move from being a “benchmark competitor” to becoming a reliable assistant in research and engineering.