Evaluating LLM Mathematical Ability: From Problem-Solving Competitions to Research Capability Assessment
Large language models keep setting new records on math benchmarks, but most of these benchmarks measure problem-solving, not research. This article explains the difference, introduces theoretical computer science (TCS) as a naturally machine-verifiable testbed for research-level evaluation, breaks down a benchmark design covering four types of real research activities, and discusses what the sobering results mean for researchers and engineering teams.
From “Solving Problems” to “Doing Research”: Evaluating LLM Capability in Theoretical Computer Science
1. Why We Need to Rethink the “Mathematical Ability” of LLMs
Over the past two years, large language models have repeatedly posted high scores on math benchmarks, and the question of whether “AI has replaced mathematical researchers” has become a public debate. But a closer look at these results reveals a widely overlooked issue: the vast majority of leaderboards measure problem-solving ability, not research ability.
There is a fundamental difference between the two:
- Problem-solving: The problem is known to have a solution, a standard answer, and a clear grading rubric. The model only needs to find matching patterns in its massive training distribution and produce a correct reasoning chain.
- Research: The problem itself may be open— even the person posing it doesn’t know the answer. There are no prescribed steps, no reference path, and often no way to know in advance whether the direction is even feasible. The value of research lies in creating new knowledge, not reproducing known solutions.
Using high leaderboard scores to claim a model “can do research” is like using a driving test score to infer that someone can design a transportation system: the former tests proficiency within fixed rules, while the latter requires deep understanding and original breakthroughs.
An even trickier issue is evaluation credibility. The closer a task gets to real research, the harder it is to grade automatically: verifying whether a proof is correct or whether a construction is optimal often requires hours or even days of expert review. This creates a dilemma: an evaluation must be hard enough to be discriminative, but tasks that hard are difficult to grade automatically.
2. Theoretical Computer Science: A Naturally “Verifiable Research Testbed”
Theoretical computer science (TCS) happens to offer a way out of this dilemma.
If everyday software development is about “how to build the engine and assemble the interior,” TCS is about “the theoretical fastest route from point A to point B, and which roads are provably impassable.” It sits at the intersection of mathematics and computer science, with core questions centered on computation itself: upper and lower bounds on computational complexity, the limits of algorithms, the feasibility foundations of cryptography, randomness and derandomization, and more.
What makes TCS ideal as an evaluation vehicle is a unique property: most of its research conclusions can be verified automatically by machines. It resembles Sudoku—filling in the entire grid is hard, but verifying a completed grid only requires checking the rules row by row and column by column. In concrete TCS scenarios:
- Every step of a complexity lower-bound proof can be formally checked;
- An algorithmic construction can be implemented and run to verify its correctness and complexity claims;
- A reduction can be mechanically verified for whether its mapping holds.
This asymmetry—“hard to solve, easy to verify”—is the classic signature of NP-class problems in computational complexity theory. TCS-based evaluation turns this theoretical property into an engineering advantage: it preserves research-level difficulty while enabling automated, reproducible grading.
3. Benchmark Design: ~100 Tasks Covering Four Types of Real Research Activity
Recently, a team including a Turing Award laureate, scholars from the Institute for Interdisciplinary Information Sciences at Tsinghua University, and a Chinese LLM research team released a work (TCSExam) that builds an evaluation system along exactly these lines. Its core design: around the daily work of theoretical computer scientists, roughly 100 tasks were created, covering four typical research activities.
1. Understanding and Applying Concepts and Definitions
Tests whether the model truly understands core TCS concepts (such as complexity classes, reductions, and measures of randomness) and can correctly apply definitions to reason in new contexts, rather than reciting textbook statements.
2. Proof Tasks
Requires the model to complete theorem proofs or key steps of proofs. This is the main body of TCS research and the most demanding test of logical-chain integrity—any vague step invalidates the whole proof. There is no such thing as “approximately correct.”
3. Construction and Algorithm Design
Requires the model to produce concrete algorithms, protocols, or constructions, and to argue that they meet specific complexity or correctness requirements. These tasks test creativity and rigor simultaneously.
4. Open Problem Exploration
Directly selects still-unsolved open conjectures or frontier problems and observes whether the model can make substantive progress. This is the category closest to real research, and the litmus test distinguishing a “problem-solving machine” from a “research assistant.”
The four task types form a progression: from understanding existing knowledge, to rigorous derivation, to creative construction, and finally to breaking into the unknown. This mirrors the growth path of a researcher.
4. Results: A Cold Shower for the “AI Replaces Math” Narrative
The evaluation results are sobering—arguably a cold shower: for genuinely unsolved open conjectures, none of the frontier models evaluated could solve them on their own.
Models performed respectably on concept understanding and routine proof tasks, in some cases approaching or reaching the level of a human graduate student. But once the tasks moved into open-problem territory requiring original breakthroughs, model outputs typically degenerated into several patterns:
- Restating partial results already known in the literature, packaged as “progress”;
- Presenting seemingly rigorous proofs that gloss over key steps or argue in circles;
- Reducing the problem to a known result while omitting the conditions necessary for that reduction to hold.
These failure modes are themselves highly valuable for research: they expose the current cognitive boundary of LLMs—models excel at efficient retrieval and recombination within existing knowledge systems, but they do not yet possess the ability to genuinely “propose and verify original mathematical insights.”
For mathematicians and theoretical computer scientists, this is, in a sense, a sigh of relief: for the foreseeable future, deep research work still requires human intelligence in the lead.
5. Implications for Developers and Engineering Teams
The value of this kind of evaluation goes far beyond academic discussion; it also offers practical guidance for engineering:
1. Set tiered capability expectations. When bringing LLMs into R&D workflows, classify tasks into three layers: “retrieval/application — reasoning/recombination — original breakthroughs.” The first two categories can safely be delegated to models for efficiency gains; the third must retain human review.
2. Prioritize “automatically verifiable” scenarios for adoption. Similar to TCS, engineering contains many tasks that are “hard to generate, easy to verify”: run tests after code generation, run validations after configuration changes, execute and compare after SQL generation. Embedding models into a “generate–verify” loop is the best practice for controlling risk.
3. Beware of “leaderboard illusion.” When procuring or selecting models, general leaderboard scores should only serve as an initial filter. A more reliable approach is to build an evaluation set from difficult samples drawn from your own business, construct an evaluation pipeline with automated checks, and continuously track model performance on your real task distribution. With general-purpose cloud infrastructure such as object storage, log services, and monitoring/alerting, such a continuous evaluation system can be built at low cost.
6. Conclusion
The move from “solving problems” to “doing research” is the concept most easily conflated when evaluating LLM capabilities. Measured against a machine-verifiable hard discipline like TCS, the picture becomes clearer and more pragmatic: models are powerful tools for knowledge integration and reasoning assistance, but they remain a clear distance away from independently pushing the boundaries of human knowledge. For researchers, this means their professional value remains secure; for engineers, it means the right posture is not “letting AI think for you,” but building human–AI collaborative R&D workflows with verification as the safety net. A rational understanding of capability is the prerequisite for maximizing the technology dividend.
