Can LLMs Really Do Research? A Theoretical Computer Science Benchmark Provides an Answer

    Large language models now produce polished mathematical manuscripts, but is that research? This article examines a new benchmark built around theoretical computer science, where proofs can be checked automatically. The result is sobering: none of the evaluated models solved an open conjecture on its own. It also highlights how machine-verifiable tasks can separate problem-solving ability from original research.

    Background: When AI Starts Writing Proofs

    Large language models have made rapid progress in mathematics. Some frontier labs have publicly released hundreds of AI-written mathematical manuscripts, covering number theory, geometry, and mathematical physics. That has triggered widespread discussion about whether AI will replace mathematicians.

    The timing of these releases is significant. In almost every knowledge domain, LLMs have a critical weakness: they make errors that are hard to verify. Their answers are fluent and confident, but the hallucinations they contain often need domain experts to check line by line.

    Mathematics is the exception, especially mathematical proofs. Correctness can be checked automatically by formal systems. Producing a proof is hard, but checking whether a proof is valid is relatively easy. The situation is like Sudoku: solving is demanding, verifying a completed grid takes a moment. This makes AI progress in mathematics particularly measurable and less controversial.

    So the real question is: are those AI-generated “proofs” genuine research, or just research-shaped text?

    Existing Benchmarks Miss the Key Distinction

    LLM evaluation leaderboards are refreshed almost every two months, but most benchmarks measure problem-solving, not research. The difference is fundamental:

    • Solving problems is like cooking from a recipe. The task type, steps, and grading criteria are fixed, and the training data probably contains similar patterns.
    • Research has no prescribed path. Even the person who defines the problem does not know the answer in advance, and success cannot be predicted.

    This creates two evaluation difficulties:

    1. Designing research-level questions is hard, because the closer a question is to real research, the harder it is to define objective grading criteria.
    2. Grading is even harder. A model can produce hundreds of pages of “proofs,” and human review is unrealistic. If a model claims to have solved an open conjecture, who verifies it?

    That is why reports of AI-generated mathematical manuscripts attract attention but still lack a credible quantitative conclusion.

    A Workable Solution: Choose a Field Where Machines Can Grade

    A new evaluation effort offers a clever approach: only ask questions that machines can judge. Theoretical computer science is an ideal testbed.

    What Is Theoretical Computer Science?

    TCS is not about fixing computers or writing application code. It studies the limits of computation itself. A useful analogy: programmers are like drivers, focused on the engine and interior; TCS is concerned with traffic itself — how fast can we get from point A to point B, and are there roads that can never be traveled?

    It sits at the intersection of mathematics and computer science. Its questions revolve around computation, and its primary method is rigorous proof. Crucially, many of its conclusions can be verified automatically, which avoids the grading problem.

    Core Result

    When the strongest current LLMs were tested with such an exam, the conclusion was sobering: none of the evaluated models could solve any still-open conjecture on its own.

    In other words, high scores on standard benchmarks reflect pattern matching and problem-solving fluency. Research-level creation — working on open questions with no known answer — still remains human territory. For researchers worried about AI “wiping out” mathematics, this is reassuring.

    The Exam: Four Task Types

    The benchmark contains roughly 100 problems, divided into four categories that correspond to core activities in a theoretical computer scientist’s daily work:

    1. Knowledge reproduction: tests mastery of existing theorems, definitions, and classic results — the “basic skills” of research.
    2. Derivation and computation: gives a concrete object, such as a graph or a complexity expression, and asks for step-by-step derivation or proof. This tests the rigor of formal reasoning.
    3. Literature comprehension: supplies a paper context and checks whether the model can correctly understand, restate, and extend existing research results.
    4. Open exploration: uses questions that resemble real research, measuring how a model behaves when no standard answer exists.

    Because every problem’s answer is designed to be machine-verifiable — through automatically checkable proof steps or programmatically checkable constructions — the evaluation can run at scale, at low cost, and without disputes over grading.

    Implications for Developers and Researchers

    Although this work focuses on mathematics, it offers several broader lessons:

    • Read leaderboards by dimension. A high score means the model has learned many generalizable problem-solving patterns, not that it can do original research. Choose models according to the nature of your task.
    • Verifiability is key. To assess AI capability objectively in any domain, design machine-checkable tasks rather than relying on human scoring.
    • Human-AI collaboration still has a clear division of labor. AI can help with literature review, proof drafts, and result checking. Breakthroughs on open conjectures will continue to depend on human intuition and creativity in the short term.

    Conclusion

    The real value of this theoretical computer science “exam” is not ranking models. It is establishing a scientific measurement method: decompose “research ability” into verifiable subtasks, and for the first time clearly measure what AI can actually do. The conclusion is clear-eyed: AI is still young in mathematical research, and a good evaluation system is exactly what it needs to keep improving.