How to Benchmark AI Research Ability: Lessons from a Full-Domain Theoretical Computer Science Evaluation

    Benchmarks for large language models keep climbing, but do high scores prove genuine research ability? This article examines a recent evaluation built around theoretical computer science (TCS), covering 100 machine-verifiable problems across concept understanding, technique application, full derivations, and open exploration. The results show that no model could independently solve open problems, though top models perform well on known-answer tasks. We break down the evaluation's design logic, why verification is the key bottleneck, and four practical lessons for developers assessing LLMs in production.

    How to Benchmark AI Research Ability: Lessons from a Full-Domain Theoretical Computer Science Evaluation

    Introduction: When AI Starts “Doing Research,” Who Grades the Exam?

    Over the past couple of years, large language models have raised their scores on various benchmarks roughly every month or two. The leaderboards keep multiplying, the numbers keep rising—but a fundamental question remains unanswered: what do these scores actually prove?

    A recent evaluation, conducted jointly by a research institution and an AI team, offers an illuminating answer. The researchers built a benchmark covering the full field of theoretical computer science (TCS) and used it to test the strongest current models. The conclusion is somewhat sobering: when faced with genuinely unresolved open problems, none of the tested models could independently produce a reliable solution.

    This result deserves careful consideration from developers and enterprise technical decision-makers who care about the true boundaries of AI capability. This article breaks down the evaluation’s design logic, its methodological value, and what it means for how we use large models day to day.

    1. Why Traditional Benchmarks Can’t Measure “Research Ability”

    1.1 Solving Problems vs. Doing Research

    Almost all mainstream benchmarks—math competition problems, code generation tasks, knowledge quizzes—share one common trait: the answer is known, and the path follows established patterns.

    Problem-solving is essentially “cooking from a recipe”—the steps can be imitated, the result has a standard answer, and grading can be automated. Even when a problem is hard, the model only needs to search and recombine within a well-explored solution space.

    Scientific research is fundamentally different:

    • No prescribed steps: researchers must formulate their own hypotheses and design their own proof strategies;
    • The outcome is unknowable in advance: for many open problems, even the person posing the question doesn’t know the answer—or whether an answer exists at all;
    • Value is hard to quantify: a “nearly correct” proof is, mathematically, completely wrong.

    A high score on standard leaderboards therefore only shows that a model is good at “reproducing and recombining already-solved problems.” It does not prove the ability to “explore unknown territory.”

    1.2 The Grading Problem: Who Verifies a Several-Hundred-Page “Proof”?

    Letting models directly tackle open problems immediately hits a second wall: verification cost.

    Suppose a model claims to have cracked a decades-old conjecture and outputs hundreds of pages of derivation. Confirming whether it’s right would require top domain experts to check it line by line over months—completely infeasible at scale. There have already been cases in the industry where an organization publicly released hundreds of AI-generated mathematical manuscripts in one batch, spanning number theory, geometry, mathematical physics, and other fields; the verification of those manuscripts still relies on the human community slowly digesting them.

    Mathematicians have pointed out that AI has a fatal weakness in almost every domain: it produces unverifiable errors. Mathematics is one of the few exceptions—a proof’s correctness can, in principle, be checked by machine. It’s a bit like Sudoku: filling it in is hard, but once filled in, one pass tells you whether it’s correct.

    This is precisely what makes theoretical computer science uniquely valuable.

    2. Why Theoretical Computer Science Makes an Ideal “Exam Hall”

    TCS sits at the intersection of mathematics and computer science. It doesn’t care about specific programming tricks or engineering implementation; it studies the nature of computation itself: where is the lower bound on the complexity of solving a problem? Is a class of problems decidable? How large is the theoretical gap between approximate and optimal solutions?

    A useful analogy: an ordinary programmer is like a driver, focused on the engine and the interior; TCS studies traffic itself—how fast one can theoretically get from A to B, and which routes are provably dead ends.

    Three technical reasons make TCS a sensible evaluation domain:

    1. Conclusions are machine-verifiable: TCS results consist mainly of rigorous proofs with objective correctness criteria, and some conclusions (such as constructed algorithms or lower-bound instances) can be checked directly by programs;
    2. Rich difficulty gradient: from standard-answer technique problems to open conjectures requiring new methods, the difficulty spectrum allows fine-grained differentiation of model capability levels;
    3. Broad coverage: complexity theory, algorithm design, cryptographic foundations, information theory, game theory, and more—a representative picture of “full-domain” research.

    In other words, TCS is one of the few fields that can simulate “real research” while bypassing the “human grading bottleneck.”

    3. Benchmark Design: A 100-Question “Research Ability Exam”

    The benchmark contains 100 problems, organized into four categories that mirror the daily workflow of theoretical computer scientists. This classification is itself instructive—it divides problems not by knowledge domain, but by type of research behavior:

    Category Corresponding Research Behavior What It Tests
    Concept understanding Reading literature, distinguishing definitions Depth of grasp of foundational theory
    Technique application Proving lemmas, constructing counterexamples Fluency with the toolbox
    Comprehensive derivation Completing full theorem proofs Multi-step reasoning and proof planning
    Open exploration Attacking unsolved conjectures Genuine original research ability

    Key Design Considerations

    First, every problem must be machine-gradable. This is the foundation of the entire benchmark. Each problem either has a clearly verifiable output (such as a concrete construction or an executable algorithm), or its proof can be checked through formal verification. This guarantees reproducibility—anyone using the same problems will reach consistent grading conclusions.

    Second, difficulty must extend into territory where “no one knows the answer.” Only by mixing in genuinely open problems can you find where the “ceiling” of model capability lies, rather than having all models compete in known territory over who has memorized more.

    Third, guard against data contamination. When evaluating research ability, you must confirm the problems haven’t appeared at scale in training corpora—otherwise you’re testing memory, not reasoning. This is a general best practice for constructing any serious benchmark.

    4. Core Findings: How Far Is AI from Being an “Independent Researcher”?

    The evaluation delivers several clear signals:

    1. Strengths on known problems: top models performed well on concept understanding and technique application, with some proofs showing impressively coherent reasoning chains;
    2. Total failure on open problems: faced with unresolved conjectures, no model could independently produce a correct and complete solution;
    3. “Plausible-looking” errors remain common: models often output “pseudo-proofs” that are structurally complete and professionally worded but contain gaps at critical steps—highly misleading for users without domain expertise.

    This matches widespread industry observation: today’s large models are more like “well-read assistant researchers”—capable of quickly retrieving knowledge, completing routine derivations, and validating ideas under human guidance—but there remains a fundamental gap before they can independently formulate and solve new problems.

    For trending claims like “AI will make mathematicians unemployed,” this evaluation offers a rational reality check.

    5. Practical Lessons for Developers and Technical Teams

    The value of this evaluation methodology extends well beyond academia. For developers deploying large models in enterprises, at least four takeaways apply:

    5.1 Adopt a “Verifiability-First” Evaluation Mindset

    When assessing models for your business scenarios, prioritize task formats whose outputs can be programmatically validated. For example, run test cases against generated code, perform static checks on generated configurations, or validate constraints on generated data—rather than relying on manual spot checks and subjective judgment.

    5.2 Distinguish “Retrieval and Reproduction” from “Exploration and Generation”

    Be explicit in system design about whether the requirement is a “deterministic task with a standard answer” (format conversion, common-pattern code generation) or an “exploratory task without a standard answer” (architecture design, troubleshooting hard bugs). The former can be trusted to a large degree when paired with automated validation; the latter must keep a human expert in the loop as the final arbiter.

    5.3 Beware “Professional-Looking Errors”

    The more fluent and domain-appropriate a model’s output, the harder its hidden errors are to spot. In production, key outputs should be paired with a validation layer—rule engines, formal verification tools, or cross-checking with a second model—forming a complete “generate–verify–fallback” pipeline.

    5.4 Favor Research-Oriented Evaluations Over Leaderboard Scores

    When selecting models, look beyond public leaderboards to evaluations that include unseen problems without standard answers. These better reflect a model’s robustness in the long tail of real business scenarios. Enterprises can also build small-scale “internal exams” from their own business data to continuously track genuine changes in model capability.

    Conclusion

    What a 100-question exam measures is not just a handful of model scores, but a clear capability boundary: large language models are already powerful tools for knowledge integration and reasoning assistance, yet “independently exploring the unknown” remains the territory of human researchers.

    For technology practitioners, what matters is not debating whether AI will replace anyone, but learning to drive efficiency boldly in the deterministic zones where models excel, stay cautious in the exploratory zones where models are unreliable, and use verifiable engineering mechanisms to harden that boundary into system design. That may be the most practical lesson this evaluation offers the industry.