How Far Can Large Language Models Go in Mathematical Research? A View from Theoretical Computer Science Evaluation
This article explains why theoretical computer science (TCS) works well for testing the research ability of large language models, how a research-level evaluation differs from standard benchmarks, and what its results show about open mathematical problems. It also covers practical lessons for developers who evaluate and use model reasoning, including layered testing, verification loops, and caution toward claims of mathematical breakthroughs.
1. Why Use Theoretical Computer Science to Test Large Language Models
Theoretical Computer Science (TCS) sits at the intersection of mathematics and computer science. It does not focus on writing code or deploying systems. Instead, it studies computation itself: Is a problem solvable in principle? What is the lower bound on the optimal complexity of a solution? Which approaches are structurally doomed to fail?
An analogy helps. Application developers are like drivers. They care about the engine, the interior, and the driving experience. TCS researchers care about the transportation network itself: how fast one can theoretically travel from point A to point B, and which congestion is structurally impossible to resolve. The main output of TCS is not code. It is proof.
Because proof sits at the center of TCS, the field offers a natural testing ground for the research ability of large language models. Proofs have a distinctive property: verifying them is far easier than constructing them. Consider Sudoku. Filling in a solution is hard, but checking whether a completed grid is correct is straightforward. If a model claims to have proved a theorem, its reasoning chain can, in principle, be checked step by step with formal verification tools. This is difficult to do in other disciplines, such as physics or medicine.
2. Evaluation Design: From Solving Problems to Doing Research
The evaluation was produced jointly by leading academics and AI practitioners. It covers several core branches of TCS, including algorithmic complexity, cryptography foundations, computational complexity theory, and combinatorial optimization. Several frontier large language models were tested.
The design differs fundamentally from conventional leaderboards:
- Standard benchmarks measure problem-solving ability. Questions have fixed answers, and the solution paths are relatively predictable. The task resembles following a recipe: the steps are given, and the result has a known standard.
- Research-level evaluation measures exploratory ability. No solution steps are specified, and the question setters may not know the answer. Whether the work succeeds cannot be guaranteed in advance.
This distinction addresses a core problem in how large language models are currently evaluated. Leaderboard scores are updated roughly every one or two months. A high score shows that a model can solve problems, but it does not show how well the model performs real research. A model that earns gold-medal-level scores on competition problem sets may perform very differently when faced with an open conjecture that has resisted proof for decades.
3. Key Finding: Open Problems Remain Beyond AI’s Reach
The results were sobering for the industry. On open conjectures that remain unresolved, none of the evaluated models could complete a proof independently.
The results point to several key facts about the current limits of large language models.
1. A capability gap exists between known and unknown problems.
Models perform well on problem types covered by their training data. This performance reflects strong pattern matching and the ability to retrieve and recombine existing knowledge. Open conjectures remain open because the reasoning paths humans have already tried all fail. Solving them requires constructing new mathematical objects or new proof techniques. This kind of creation from nothing still exceeds current model capabilities.
2. Reliability in long reasoning chains remains fragile.
Research-level proofs often require hundreds of tightly linked steps. A model may have high accuracy on each individual step, but its overall reliability drops sharply as the steps compound. A more serious problem is that models produce “proofs” with subtle flaws and state them with high confidence. Non-experts find these errors especially hard to detect.
3. Verifying the output has become a new challenge.
When a model generates a proof running to hundreds of pages, manual checking becomes extremely costly. If a model claims to have solved a conjecture, who verifies that the proof is correct? This raises an important technical direction: machine-assisted formal verification. Translating natural-language proofs into formal languages that proof assistants can check mechanically is a key link between AI generation capability and mathematical rigor.
4. Implications for Developers: Evaluating and Using Model Reasoning Correctly
The findings also matter for developers and technical decision-makers.
First, adopt layered evaluation. When selecting or building a model, do not rely on a single leaderboard score. Test separately across three types of tasks: routine questions with standard answers, complex questions that require multi-step reasoning, and exploratory problems without established solution methods. Performance across these three types of tasks is not necessarily correlated.
Second, build verification loops around AI output. In domains such as mathematics, code, and logical deduction, use automated checks wherever possible. Examples include unit tests for generated code, constraint solvers for logical conclusions, and static analysis tools for flaws in reasoning. A “generate and verify” architecture is far more reliable than trusting generated results directly.
Third, use general-purpose cloud infrastructure to support evaluation workflows. Large-scale evaluations need elastic compute, object storage for problem sets and results, logging services to trace reasoning processes, and monitoring and alerting to track resource consumption. These components can be assembled from standard cloud computing services without depending on a specific vendor’s solution.
Fourth, approach claims of AI solving hard math problems with caution. When a model is said to have achieved a research-level breakthrough, ask: Has the proof been formally verified? Have domain experts independently reviewed it? Can the result be reproduced?
5. Outlook: Where AI Mathematical Ability Goes Next
Overall, current large language models show an uneven profile in mathematics. Their breadth is impressive, but their depth is limited. They can quickly absorb the existing body of human knowledge and apply it skillfully. However, they have not yet moved from solving problems to doing research. Future progress may come from stronger consistency in long reasoning chains, deeper integration with formal verification tools, and greater creativity in generating hypotheses and constructing counterexamples.
For the industry, rigorous research-level evaluation is valuable. It shows that inflated benchmark scores do not equal genuine progress in intelligence. Real progress will require measuring models on questions where even the question setters do not know the answer.
