How to Evaluate the Research Capabilities of Large Language Models in Theoretical Computer Science

    As large language models achieve increasingly strong results on mathematical competitions, it is important to distinguish solving known problems from conducting original research. This article examines a research-oriented evaluation framework for AI in theoretical computer science, centered on automatically verifiable tasks, open conjectures, proof construction, counterexamples, and problem solving. The results show that current models remain unable to independently solve established open problems, while already offering practical value as research assistants.

    1. Background: Why “AI Solving Problems” Is Not the Same as “AI Conducting Research”

    Over the past two years, large language models have achieved increasingly strong results on mathematical competition problems. These achievements have repeatedly triggered public debate, including speculation about whether AI is already capable of solving major mathematical research problems at scale.

    Behind this excitement lies a fundamental conceptual confusion: problem-solving ability is not the same as research ability.

    The difference can be understood across several dimensions:

    • Whether the procedure is known in advance: Solving a problem is like following a recipe. The process is usually structured, the requirements are clear, and there is often a unique answer. Research, by contrast, has no predefined path. Researchers must explore an unknown solution space and may first need to determine whether a problem is solvable at all.
    • Whether the result is already known: Problems in a test set generally have standard answers. In open research, even the person who posed the question may not know the conclusion.
    • How correctness is verified: The answer to a conventional problem can often be checked quickly. A research proof requires rigorous, peer-review-level validation, and verification can be extremely costly.

    Theoretical computer science (TCS) lies at the intersection of mathematics and computer science. It studies computation as a fundamental object, including algorithmic optimality, computability, and lower bounds in computational complexity. Its primary research method is rigorous mathematical proof.

    Because many TCS results can be expressed and formally verified, the field provides an ideal environment for evaluating whether AI systems possess genuine research capabilities.

    2. Why Existing Benchmarks Do Not Measure Research Ability

    The number of benchmarks for large language models has grown rapidly, and evaluation scores are frequently refreshed. Yet most of these leaderboards share the same methodological limitation: they measure the ability to reproduce solutions to problems with known answers, not the ability to explore problems whose answers are unknown.

    Traditional evaluations face three major limitations.

    First, data contamination. Many benchmark questions come from public problem sets or historical competitions and may already be present in model training data. A high score may therefore reflect memorization rather than genuine reasoning.

    Second, inadequate verification. If a model claims to have proved a conjecture that has remained open for decades, who can determine whether the proof is correct? A flawed proof may run for dozens of pages, making manual verification extremely expensive. The industry has already seen cases in which model developers released hundreds of manuscripts claiming to resolve famous conjectures, only for most of the proofs to be found defective. This exposes the large gap between an AI claim and a correct AI result.

    Third, a mismatch between benchmark tasks and research work. Core research activities—proposing conjectures, constructing counterexamples, designing proof strategies, and revising ideas after failure—are largely absent from existing evaluations.

    The field therefore needs a new evaluation approach with objective scoring, automatable verification, and tasks that more closely resemble real research.

    3. Evaluation Design: Use Problems That Machines Can Grade

    The central design principle of this evaluation is simple: every task must be capable of being judged automatically and rigorously by a program.

    This may appear conservative, but it directly addresses the most difficult problem in claims that AI has solved a major open question: verification.

    The evaluation team used several deliberate design choices.

    First, it focused on open conjectures. The assessment included well-known unresolved problems in theoretical computer science. These problems have unknown conclusions, recognized difficulty, and potentially significant value if solved. At the same time, their solutions—whether proofs or counterexamples—can be specified in a form that permits strict verification.

    Second, it covered four representative research activities. The evaluation contained 100 questions in four categories, corresponding to major activities in the daily work of theoretical computer scientists:

    1. Conceptual understanding and theorem application — testing knowledge of fundamental concepts, classical theorems, and the conditions under which they apply;
    2. Proof construction — requiring the model to independently produce a rigorous mathematical proof for a given statement;
    3. Counterexamples and constructions — asking the model to construct an object satisfying specified constraints or provide a counterexample showing that a claim is false;
    4. Open-problem solving — directly challenging unresolved conjectures and testing whether the model can produce original research results from scratch.

    These categories form a capability gradient. They progress from knowing established facts, to reasoning rigorously, to creating new constructions, and finally to addressing unknown problems.

    4. Results: A Necessary Reality Check

    The evaluation tested a group of leading large language models available at the time. Its findings should be interpreted with caution.

    None of the evaluated models independently produced a correct solution to the open conjectures that remain unresolved today. This result stands in sharp contrast to the public expectation that AI is about to solve major mathematical problems in large numbers.

    The result is not surprising when the underlying structural challenges are considered:

    • Research requires long-horizon creative reasoning. Solving an open conjecture may require weeks or years of sustained work, involving repeated experimentation, hypothesis formation, and self-correction. This is fundamentally different from the limited-step reasoning available within a model’s context window.
    • Hallucinations are especially costly in mathematical proofs. A generated proof may be fluent and use technically appropriate terminology while quietly introducing an unsupported assertion at a critical step. In mathematics, such errors are far more damaging than ordinary mistakes in general question answering.
    • Training data is biased toward known results. Open conjectures remain open precisely because successful solutions are absent from the available literature. As a result, a model’s pattern-learning mechanism has little successful material to imitate.

    The evaluation also showed a positive side. Models performed relatively well on conceptual understanding, restating theorems, and reconstructing proofs of known results. This suggests that they already have practical value as research assistants for tasks such as retrieving information, organizing ideas, and checking details.

    The more reasonable current characterization is that AI is a copilot for mathematicians, not a replacement for them.

    5. A Rational Reflection on the AI Mathematics Hype

    This evaluation attracted attention because it provided quantitative evidence to cool down an overheated narrative about AI and mathematical research.

    A common industry assumption is that strong performance on competition problems can be directly extrapolated into the claim that a model will soon solve problems on the scale of the Riemann Hypothesis. That inference does not hold.

    Competition problems are essentially search and pattern matching within a closed answer space. Research on open problems is exploration and creation within an open hypothesis space. The gap between these activities cannot be expected to disappear simply by adding compute or increasing model size.

    The ability to solve established problems can also create a harmful evaluation cycle:

    1. Vendors optimize models for leaderboard performance;
    2. Models become better at solving benchmark questions;
    3. The public assumes that research capability has improved at the same rate;
    4. Expectations continue to rise.

    Research-oriented evaluations driven by verifiability can help break this cycle. They anchor assessment to a more demanding standard: the production of genuine, verifiable research results.

    6. The Next Steps for Evaluating Research Capability

    This evaluation provides a reusable methodological framework and points to several directions for future work.

    • Build dynamic problem sets. As model capabilities evolve, evaluation tasks must be continuously updated. Otherwise, they may eventually enter training corpora and lose their value. An automated “generation–solving–verification” pipeline could help expand the pool of open problems over time.
    • Evaluate processes, not only outcomes. Future assessments could examine the model’s research trajectory: What hypotheses did it propose? How did it respond to failure? Such data could improve understanding of model reasoning mechanisms.
    • Evaluate human–AI collaboration. Instead of focusing only on whether AI can independently solve a conjecture, a more practical target may be the productivity gains achieved by mathematicians working with AI. This would measure AI’s role as research infrastructure and leverage.
    • Integrate formal verification more deeply. Proof assistants and other machine-verification tools could be closely coupled with model generation. If an AI-generated claim is checked immediately by a machine, the verification gap could be substantially reduced.

    7. Conclusion

    This evaluation, developed by a collaboration involving a Turing Award winner and leading large-model researchers, was designed to test not how quickly models solve textbook-style problems, but how close they are to functioning as genuine research workers.

    The conclusion is clear and pragmatic: on the most demanding open problems, AI is currently unable to solve them independently. However, it already provides meaningful value as an assistance tool.

    For enterprise researchers and AI practitioners, the central lesson is straightforward: distinguishing “can solve problems” from “can conduct research” is the first step toward avoiding technology misjudgments and misallocated resources.

    Evaluation of research capability is only beginning. In this process, verifiability will be one of the most important guideposts.