From Solving Problems to Doing Research: Where Are the Real Limits of LLMs in Mathematics?
This article examines a recent systematic benchmark built to evaluate frontier large language models across the full spectrum of theoretical computer science (TCS). It explains why strong performance on competition-style math problems does not translate into genuine research ability, outlines the design principles behind research-oriented evaluation—full-domain coverage, difficulty gradients aligned with real research, and verifiable reasoning—and summarizes the key capability boundaries revealed: deduction outperforms construction, long proof chains are unstable, models lack metacognition on open problems, and cross-domain transfer is weak. It concludes with implications for model development, including iterative propose-verify-correct loops, evolving benchmarks, and capability profiles instead of single scores.
1. A Noteworthy Evaluation Signal
A recent joint effort by academic research institutions and AI research teams has drawn considerable attention: the researchers built a systematic benchmark covering the entire field of theoretical computer science (TCS) and used it to assess the true mathematical and theoretical research capabilities of today’s most advanced large language models.
Theoretical computer science sits at the intersection of mathematics and computer science. It does not deal with engineering questions such as fixing computers or writing code; instead, it asks about the nature of “computation” itself: which problems can be solved computationally, which problems are in principle impossible to solve efficiently, and where the lower bounds on the cost of solving them lie. If engineers are like drivers, focused on the engine and the handling experience, then theoretical computer scientists study the entire transportation system—theoretically, how fast can one get from point A to point B, and is there any route that is doomed to be impassable?
The core working method of this field is mathematical proof. That makes it an excellent touchstone for testing a model’s “research ability”: the problems span a wide range, the difficulty gradient is naturally clear, and many frontier problems have no known answer—even to the people who pose them.
2. Why Solving Problems Is Not the Same as Doing Research
In recent times, headlines about “AI cracking hard math problems” have repeatedly made news. But a closer look reveals that most of these results come from competition problems or standardized test items, which differ fundamentally from genuine mathematical research.
2.1 Problem-solving is a closed task; research is an open one
Solving problems is essentially executing established methods: problem types follow patterns, solutions follow templates, and answers have a single standard. It is much like cooking from a fixed recipe—the steps are given, and the quality of the result has a reference. Research, by contrast, confronts problems that have never been solved. There is no ready-made path, and one may not even know whether the problem is solvable at all or in which direction the solution lies. A model’s pattern-matching strength thrives on closed problems, but that advantage decays rapidly once it enters the domain of open questions.
2.2 The asymmetry of verification difficulty
Sudoku-style problems share a property: filling in a solution is hard, but verifying it is easy. Most conjectures in mathematics are the opposite—not only is solving them difficult, verifying the correctness of a candidate answer is equally difficult. When a model claims to have “proved a conjecture that has been open for years,” reviewers must invest effort comparable to original research to judge whether the claim is genuine. As a result, AI results on open mathematical problems inherently lack a trustworthy referee.
2.3 Hidden contamination in training data
Standardized problems exist in massive quantities on the internet, and models have very likely “seen” both the problems and their answers during pretraining. Even when evaluators apply decontamination measures, a high score may still reflect memorization rather than reasoning. By contrast, frontier proof tasks in theoretical computer science have far lower coverage of complete derivations in public corpora, which strips away memorization effects to a much greater degree.
3. Key Principles of Evaluation Design
For an evaluation to genuinely characterize “research ability” rather than “test-taking ability,” the benchmark design needs to follow several principles—principles that this work also embodies.
3.1 Full-domain coverage instead of point sampling
Mathematical ability is not a single dimension. In theoretical computer science alone, there are highly abstract directions such as computability theory and complexity theory, as well as more constructive directions such as algorithm design and analysis, cryptographic foundations, and combinatorial optimization. Testing only one category of problems makes it easy to mistake a model’s local strength for overall ability. Covering all major subfields is the only way to draw a “topographic map” of capability.
3.2 A difficulty gradient aligned with real research
Evaluation items should progress layer by layer—from textbook level, to survey level, to paper level, and finally to open-problem level. Frontier models can almost all pass undergraduate-level questions on definitions and theorems. The real dividing line appears in problems that require multi-step constructions, the introduction of new lemmas, or combined reasoning across multiple areas of knowledge. A graded design can pinpoint exactly where a model’s capability “cliff” lies.
3.3 Verifiability first
For proof-based tasks, evaluation should not look only at the final conclusion; it must examine whether every step of the reasoning chain strictly holds. Feasible approaches include: using formal verification tools to machine-check parts of the proofs, having domain experts review proof skeletons section by section, and designing closed problems with known answers alongside open problems with unknown answers for comparison. Only when the verification process itself is rigorous enough do the evaluation conclusions carry conviction.
4. What Boundaries Does This Kind of Evaluation Reveal?
Drawing together the general observations from such systematic evaluations, several boundaries of current LLM mathematical capability can be summarized.
First, deduction outperforms construction. Models perform relatively well at checking whether a given piece of reasoning is valid and at completing standard derivations. But on tasks that require actively constructing counterexamples, designing new algorithms, or proposing new lemmas, the success rate drops significantly.
Second, long-chain reasoning is unstable. Mathematical proofs often require dozens of strictly connected steps. Models have low error rates on short chains, but as the chain grows, an early step that is merely “plausible-sounding” rather than rigorous gets amplified by subsequent steps, and the probability that the entire proof collapses rises rapidly.
Third, models lack metacognition on unsolved problems. A genuine researcher facing an open problem can form judgments about whether a solution exists and how confident they are. Models, in contrast, tend to confidently produce a complete but flawed “proof.” This overconfidence is precisely the most dangerous trait in research settings.
Fourth, cross-domain combination is weak. Breakthroughs in modern mathematics often come from transferring the tools of one field into another. Models already possess considerable knowledge within individual domains, but their ability to spontaneously transfer tools and combine them innovatively remains very limited.
5. Implications for Future Development
Reasoning backward from the evaluation perspective, several directions for model improvement are worth attention.
First, combining reasoning processes with verification mechanisms. Rather than having a model output a complete proof in one shot, it is better to build an iterative “propose–verify–correct” loop: the model produces derivation steps, external verifiers (including formal proof systems and automated checkers) validate them step by step, and errors are fed back and corrected immediately. This is analogous to the role continuous integration plays in safeguarding code quality in engineering.
Second, evaluation benchmarks need to keep evolving. Static benchmarks get “caught up” by training data, leading to inflated scores. Evaluation sets should retain a certain proportion of private items and newly generated open problems, and should be updated regularly, so that scores always reflect a model’s generalization and reasoning ability rather than corpus coverage.
Third, moving from “scores” to “capability profiles.” A single aggregate score is of limited help in understanding a model’s research potential. Far more valuable is characterizing how the model performs across different subfields, different reasoning depths, and different levels of verification strictness—so users know exactly which stages the model can handle and which stages must be covered by human experts.
6. Conclusion
The significance of this systematic evaluation in theoretical computer science lies not in ranking models against each other, but in providing a measuring stick that is much closer to “real research.” It reminds us that rapid progress in problem-solving ability does not automatically amount to a breakthrough in research ability. The true boundary of AI in mathematics lies in whether it can move from “executing known methods” to “inventing unknown ones”—and whether we can build a credible verification system for that capability. How we answer this question will determine whether AI can grow from a powerful problem-solving tool into a genuine research collaborator.
