Why Frontier LLMs Still Can't Crack Open Conjectures: What a TCS-Focused Research Benchmark Tells Us

    A new evaluation led by a Turing Award laureate tests frontier large language models on 100 research-style problems across theoretical computer science. The results show that while LLMs excel at solving problems with known answers, none can independently resolve a single open conjecture—highlighting the gap between benchmark scores and genuine research ability, and the importance of machine-verifiable proof checking in AI-assisted science.

    Why Frontier LLMs Still Can’t Crack Open Conjectures: What a TCS-Focused Research Benchmark Tells Us

    1. Background: When AI-Generated Math Starts Causing Anxiety

    Recently, a leading foundation model company published hundreds of AI-generated mathematical manuscripts spanning number theory, theoretical computer science, geometry, and mathematical physics, organized by relevance into hundreds of “result families.” The question “Will AI make mathematical research obsolete?” briefly became a hot topic on social media.

    The core tension in this debate is straightforward: on one hand, large language models keep posting higher scores on competition problems and textbook exercises; on the other, no one can say with confidence whether these models are capable of original research. Against this backdrop, a new evaluation effort—led by a Turing Award laureate and carried out jointly by academic research institutions and a major domestic LLM team—deserves attention. The researchers designed an exam covering the full breadth of theoretical computer science (TCS) and used it to systematically probe the research-level capabilities of today’s strongest frontier models.

    2. Why Theoretical Computer Science Is the Testing Ground

    2.1 What TCS actually studies

    Theoretical computer science is not about assembling hardware or writing production code. It concerns the abstract concept of computation itself: which problems can be solved within reasonable time, which problems are provably unsolvable, and where the complexity lower bounds of algorithms lie. An analogy: programmers are like drivers, focused on vehicle performance and driving experience; TCS researchers are like the planners of the transportation system, asking how fast one could theoretically travel from point A to point B, and which routes are provably impassable.

    2.2 The unique advantage: machine-verifiable conclusions

    Evaluating research ability faces a fundamental problem: grading. If a model claims to have resolved a conjecture that has been open for decades, its “proof” could run to hundreds of pages—line-by-line human review is practically impossible. TCS (and mathematical proof in general) is the exception: the correctness of a proof can be formalized and mechanically checked by automated theorem-verification tools. It works like Sudoku: solving is hard, but checking a completed grid is easy. This breaks the vicious cycle of “the more it looks like real research, the harder it is to grade,” providing rare ground for objective evaluation.

    2.3 An authoritative view from the field

    Renowned mathematician Terence Tao told Nature that AI has a fatal flaw in almost every domain: it makes errors that cannot be verified. Mathematical proof is the sole exception—it can be checked automatically by machines. This explains why AI companies can most easily produce indisputable results in mathematics, and it underscores the weight of this evaluation’s conclusions.

    3. The Core Design of the Evaluation

    3.1 A fundamental difference from existing benchmarks

    Most existing leaderboards test “problem solving”: the questions have standard answers and established solution patterns—essentially cooking from a recipe. Real research has no prescribed steps; the results are unknown in advance, and even the problem setters don’t know the answers. This evaluation aims to measure exactly that: putting AI to genuinely research-grade work and seeing how far it can go.

    3.2 Exam structure

    According to publicly available information, the exam contains 100 problems, organized around the four types of research activities that theoretical computer scientists routinely engage in, covering the field’s major directions. The common design principle is to stay as close as possible to real research scenarios while ensuring that conclusions can be graded automatically by machines. This strikes a careful balance between authenticity and evaluability.

    4. Findings and Implications

    The headline conclusion is a cold shower: among the open conjectures that remain unresolved to this day, not a single one was solved by any of the frontier models tested using their own capabilities. In other words, models can excel at recombining and restating existing knowledge, but they remain out of their depth when facing truly unknown problems that demand original chains of reasoning.

    This carries several implications for practitioners and decision-makers:

    • Benchmark scores do not equal research ability. High scores on drill-style tests reflect pattern matching and knowledge retrieval, and cannot be extrapolated to open-ended research tasks.
    • Verifiability should be a standard component of the AI research toolchain. Any workflow that lets AI participate in proof generation should integrate automated formal verification, keeping unverifiable errors out.
    • Human-AI collaboration remains the realistic path. In the near term, AI is better suited as a research assistant—retrieving literature, verifying special cases, generating candidate ideas—while conjecture-level breakthroughs still require human researchers in the lead.

    5. Conclusion

    The value of this exam is not in ranking models, but in drawing a clear line: between problem solving and research, a gap remains that has yet to be crossed. For mathematicians, this may come as a relief; for AI researchers, it points to the next direction truly worth conquering.