Benchmarking Frontier LLMs on Math and TCS: From Solving Problems to Doing Research
A recent large-scale evaluation tested frontier large language models against high-quality problems spanning theoretical computer science (TCS). The results show that while today's models excel at standard exercises, none can independently prove open conjectures. This article explains why mathematics is an ideal benchmark, what distinguishes problem-solving from research ability, and what this means for AI evaluation and real-world deployment.
Why Math Became the Litmus Test for LLMs
Among all domains used to evaluate AI capability, mathematics—especially theoretical computer science (TCS)—has long been considered the “cleanest” testing ground. The reason is verifiability: once a proof is written, whether each reasoning step holds can be mechanically confirmed through formal tools or line-by-line human review. It resembles Sudoku—finding the solution is hard, but checking it takes only one pass. This property makes AI results in mathematics the least controversial, leaving little room for the fuzzy zone of “looks correct but cannot be verified.”
Recently, a team led by a Turing Award laureate, bringing together researchers from academia and industry, released a systematic evaluation. They built a high-quality problem set covering the full research landscape of theoretical computer science and ran a unified assessment on a group of today’s strongest large language models. The conclusion was sobering: when faced with open conjectures that remain unsolved to this day, not a single tested model could independently produce a valid proof.
What Theoretical Computer Science Actually Tests
To understand the significance of this evaluation, it helps to clarify what TCS is. It is not about fixing computers or writing code—it studies the boundaries and limits of “computation” itself.
A common analogy: programmers are like drivers, concerned with engines and interiors; TCS researchers care about the transportation system itself—what is the fastest possible way to get from A to B? Are there routes that can be proven permanently impassable? TCS sits at the intersection of mathematics and computer science. Every problem revolves around computation, and its core working method is rigorous proof.
The evaluation problem set covered multiple subfields, including algorithm design and complexity analysis, computational complexity theory, cryptographic foundations, and formal verification. The problems were not competition-style exercises but questions closer to real research scenarios.
The Gap Between “Solving Problems” and “Doing Research”
The evaluation surfaced a critical distinction: high scores on standardized problems do not equate to research ability. The difference shows up on three levels.
1. Certainty of steps. Solving exercises is like cooking from a recipe—the steps are established and the standards are clear. Research has no predetermined path; success cannot be judged in advance, and for many open problems, even the people who posed them do not know the answer.
2. Different evaluation systems. Exercises have standard answers to compare against. But if a model claims to have “cracked a conjecture open for decades,” grading itself becomes a hard problem—who verifies that the proof is correct? This is exactly one of the pain points the evaluation work aims to address: building benchmarks with clearly stated problems, well-graded difficulty levels, and objectively checkable answers.
3. Different sources of capability. Test-taking ability largely comes from reproducing and recombining solution patterns already present in training data. Research breakthroughs require proposing new ideas and new structures—an ability that current model architectures have not been proven to possess.
Methodological Lessons from This Evaluation
From an engineering perspective, this work offers three general lessons for building AI evaluation systems:
- Evaluations should mirror the real research distribution. Common benchmarks focus on competition or textbook problems, which only prove that a model “can solve exercises.” A problem set spanning the full field with graded difficulty levels is needed to characterize a model’s true capability on research tasks.
- Verifiability is the core design principle. Mathematical proofs, being mechanically checkable, are the least gameable evaluation scenario—and therefore a suitable window into the upper limit of model reasoning.
- Performance on open problems should be held to a strict standard. A model may produce attempts that come close to known solutions, but in mathematics the gap between “close” and “correct” is enormous. Evaluations need to expose this gap clearly.
A Rational View of Current Boundaries
The core conclusion of this evaluation can be summarized as follows: frontier LLMs have reached a very high level in mastering existing knowledge and executing established reasoning procedures, but on open research problems requiring original breakthroughs, they remain fundamentally behind top human mathematicians.
For developers and technical teams, the practical implication of this positioning is clear: AI today is best used as an “accelerator” in research and engineering—assisting with literature review, verifying existing ideas, generating candidate approaches, and automating tedious derivation checks. Posing problems, conceiving key ideas, and adjudicating final conclusions must still be led by humans. Understanding this capability boundary helps us integrate LLMs into real workflows far more effectively than chasing leaderboard scores.
