Can LLMs Do Theoretical Research? Insights from a Comprehensive TCS Benchmark

    A recent benchmark evaluating top large language models (LLMs) against 100 questions across Theoretical Computer Science (TCS) reveals that no model can independently solve unsolved open conjectures. This article explores the distinction between AI test-taking abilities and actual research capabilities, the unique value of TCS in AI evaluation due to its machine-verifiable nature, and the key takeaways for developers and decision-makers regarding the true reasoning limits of AI.

    1. A Unique Benchmark Sparks Attention

    Recently, a new evaluation paper sparked widespread discussion in academia. The research constructed a large-scale benchmark covering the entire field of Theoretical Computer Science (TCS), using 100 carefully designed questions to test current top-tier large language models (LLMs). The research team, comprising experts from top universities and research institutions, reached a conclusion that pours cold water on the optimism that “AI is about to replace mathematicians”: No tested model can independently solve any currently unsolved open conjectures.

    This sigh of relief from mathematicians precisely exposes a critical blind spot in current AI capability evaluation—we have been masquerading “test-taking ability” as “research ability.”

    2. What is TCS: The Discipline Studying the Essence of “Computation”

    To understand the value of this benchmark, one must first grasp the unique nature of TCS.

    Theoretical Computer Science does not care about how to repair computers or write business code; it studies the essential boundaries of “computation.” This can be understood through an analogy:

    • Engineering development is like driving a car, focusing on engine performance and interior comfort—namely, specific implementation details.
    • TCS is like traffic planning, focusing on how fast one can theoretically get from point A to point B, and whether certain routes are inherently impassable.

    TCS sits at the intersection of mathematics and computer science. Its core working method is proof: given a proposition, proving it true or false through rigorous logical reasoning. All problems revolve around “computation,” including algorithmic complexity lower bounds, the capability boundaries of computational models, and cryptographic foundations.

    3. Why Existing Leaderboards Fall Short: Test-Taking ≠ Research

    LLM benchmark scores are refreshed almost every two months, with various leaderboards dazzling the eye. However, these leaderboards suffer from two fundamental flaws.

    3.1 An Essential Difference Between Test-Taking and Research

    Test-taking is essentially “cooking from a recipe”: the steps are ready-made, the answers are standardized, and the question setter knows the correct solution. Real research, however, faces problems without standard answers—the solution path must be explored independently, success cannot be predicted in advance, and some questions have plagued academia for decades. This open-ended exploration cannot be measured by fixed grading rubrics.

    3.2 Grading Itself is a Challenge

    Even if a model claims to have solved a problem, verification is extremely difficult. A “proof” can span hundreds of pages, making manual line-by-line review unrealistically expensive. If a model claims to have cracked a famous conjecture, who confirms it is correct? Previously, research institutions publicly released batches of AI-generated mathematical manuscripts, sparking huge controversy in the community, which exposed exactly this issue: generation is easy, verification is hard.

    The mainstream view, echoed by Terence Tao, points out that AI has a fatal flaw in almost all fields—it makes unverifiable errors. Mathematical proof is the sole exception, as its correctness can be automatically checked by formal verification machines, much like Sudoku: solving it is hard, but verifying the correctness takes only one pass.

    4. Benchmark Design: Enabling Machine Grading

    The brilliance of this benchmark lies in only asking questions that machines can grade. The closer a question is to real research, the harder it is to score, but TCS happens to be an exception: most of its conclusions have machine-verifiable structures, such as the inclusion relations of complexity classes, reductions of decision problems, and constructions of combinatorial structures. Their correctness can be programmatically checked.

    The 100 questions are categorized into four working modes of theoretical computer scientists’ daily routines, covering a complete difficulty gradient from basic concept understanding to frontier open problems. This design makes the evaluation resemble an “exam” while maximally approximating the form of “real research.”

    5. Core Conclusions and Insights

    The evaluation conclusions can be summarized in three points:

    1. Total failure on open problems: No frontier model can independently solve any unsolved conjecture. This defines the current ceiling of AI capabilities.
    2. “Test-taking” ability is highly misleading: Models that score high on standardized exams will experience a drastic decline in performance when facing real research-oriented problems.
    3. Verifiability is the cornerstone of research evaluation: Only domains where conclusions can be machine-verified offer “uncontroversial” AI research contributions. This also explains why mathematics and theoretical computer science are the preferred testing grounds for evaluating AI research capabilities.

    6. Implications for the Industry

    For developers and technical decision-makers, this research sends clear signals:

    • When evaluating AI coding assistants, distinguish between “template code generation” and tasks requiring reasoning like “algorithm design/complexity analysis.” High scores in the former cannot be extrapolated to the latter.
    • When building AI research assistance tools, prioritize integrating formally verifiable steps (e.g., theorem provers, test-driven verification) so that human-machine collaboration is built on a verifiable foundation.
    • When viewing AI capability marketing, be wary of using benchmark scores as a substitute for real capability evidence, especially in scenarios involving open-ended and creative tasks.

    AI is still a long way from independently conducting real mathematical research, and designing “auto-gradable research questions” is itself an important step towards an AGI evaluation system.