LLM Math Evaluation: The Real Gap Between Solving Problems and Doing Research
A joint evaluation of frontier large language models in theoretical computer science (TCS) offers a sobering answer to a popular question: can AI truly do mathematics? While models keep setting new records on math benchmarks, this study shows that strong problem-solving scores do not equal research capability—no tested model could independently crack the open conjectures. This article explains why problem-solving and research are fundamentally different skills, why verification is harder than generation, how grading hundreds of pages of proofs becomes a bottleneck, and what this means for applying LLMs to real math and TCS research, along with practical engineering recommendations.
1. Starting from a Trending Topic: Has AI Really “Killed” Math?
Every so often, a topic like “AI has finished off mathematics” trends on social media. The reasoning behind this sentiment seems solid: compared with writing, coding, or creative art, mathematics—especially mathematical proofs—has an advantage that is hard to match in other fields: whether an answer is correct can be verified automatically by a machine. In theory, whether an argument is sound can be handed to a formal proof checker for step-by-step verification, without relying on human experts to scrutinize it repeatedly. It is like Sudoku: filling in a hard grid is extremely difficult, but checking whether it follows the rules once it is filled takes only a glance.
Precisely because of this “verifiability,” mathematics is widely regarded as the least controversial highlight arena for large models. AI teams are eager to publicize their scores on math competition problems and olympiad questions, because a machine’s victory in math can hardly be questioned. But do these results really mean the model “can do mathematical research”? A dedicated theoretical computer science evaluation, jointly completed by scholars from top universities and an LLM development team, gives a rather sober answer.
2. Theoretical Computer Science: Where Math Meets Computing
To understand this evaluation, you first need to understand the field it assesses—Theoretical Computer Science (TCS).
If you think of TCS as the “Olympiad” of the mathematical world, what it studies is not how to repair a computer or how to write a piece of runnable code, but the very nature of “computation” itself. A fitting analogy: a programmer is like a driver who cares about the engine, the interior, and the driving experience, whereas theoretical computer science is more like the study of traffic itself—from point A to point B, how fast is theoretically possible? Are there some destinations that, no matter how you plan the route, are simply impossible to reach?
TCS sits at the intersection of mathematics and computer science, and all of its questions revolve around “computation”: whether a problem is solvable, how many resources solving it requires, and how low an algorithm’s lower bound can be pushed. Unlike engineering implementation at the front end, the core tool of TCS research is proof, not running and testing.
3. What Is This “Exam” Actually Testing?
The goal of this joint evaluation is direct: to test how strong the “research” ability of the most advanced current LLMs really is within the complete field of theoretical computer science—not merely their “problem-solving” ability.
The special nature of the research field needs emphasis. Many problems in TCS are open conjectures—their answers are still unknown to anyone, and even the person posing the question does not know the correct answer. These problems are fundamentally different from ordinary exam questions: a standard problem has a standard answer and an established solving procedure, so the model only needs to reproduce the reasoning path; an open conjecture has no known answer, and if a model could truly provide a correct proof, its contribution would equal an original research breakthrough.
4. Core Conclusion: Solving Problems Is Not the Same as Doing Research
The results of the evaluation are a cautionary tale: for those open conjectures that remain unresolved to this day, none of the tested reference models was able to crack any of them on its own.
This conclusion needs to be understood in the right context. Scores of large models on math benchmarks are refreshed almost every two months, leaderboards appear endlessly, and the numbers keep getting more impressive. But what these leaderboards mainly measure is “problem-solving” ability: the problem types have been trained on repeatedly, the solving paradigms are traceable, and the scoring criteria are fixed. High scores on a leaderboard can prove that a model is a qualified “exam taker,” but they cannot prove that it has the research ability to independently open up new results.
5. Why Problem-Solving and Research Are Two Different Capabilities
The gap between problem-solving and research can be analyzed on three levels.
1. Certainty of steps
Problem-solving is essentially cooking by the recipe: ingredients, heat, and steps are all ready-made, and the evaluation criteria are clear—no one would accept a “creative” dish that adds chocolate to sweet-and-sour pork. Research has no ready-made recipe; along the way you must judge for yourself which direction to go, many attempts are doomed to fail, and no one even knows whether “this dish should be made this way.”
2. Structure of the exploration space
From a computational complexity perspective, solving and verifying are often asymmetric. For many problems, the solution space is enormous and blind enumeration is infeasible; but once a candidate answer is given, verifying whether it is correct can be quite easy. The following illustrative pseudocode captures this asymmetry:
solve(problem):# Search space grows exponentially with size; no known efficient general strategyfor candidate in search_space(problem): # size O(2^n)if satisfies(candidate, problem):return candidatereturn unknownverify(problem, candidate):# Given a candidate, check constraints one by one; usually polynomial timefor constraint in problem.constraints: # size O(n)if not check(constraint, candidate):return falsereturn true
Precisely because verification is easy, a large model’s “correct” results are easily trusted; but precisely because solving is hard, a model’s “generation” ability does not equal a “discovery” ability—it can efficiently produce seemingly reasonable reasoning, yet it struggles to actually find that previously-unreached solution within an exponential search space.
3. The ability to define goals
Research is not only about solving problems; it also requires judging which problems are worth solving and how to formalize a vague problem into a tractable mathematical object. This ability to “pose problems” and “define problems” is exactly what current models, trained with an answer-focused objective, most lack.
6. The Grading Challenge: When a Model Writes Hundreds of Pages of Proof
A more practical challenge than “solving or not solving” is “how to grade.”
A large model can generate hundreds of pages of “proof” in one go. Who will review it page by page? If a model claims to have cracked a conjecture that went unsolved for decades, who has the ability and the energy to confirm, line by line, whether it actually got it right?
This exposes the bottleneck of the entire AI math research pipeline: generation cost is extremely low, verification cost is extremely high. When models can produce large amounts of lengthy, seemingly rigorous arguments in bulk, the reviewing capacity of human experts instead becomes the weak link in the whole process. Formal proofs and automatic machine verification can theoretically ease this contradiction—translating a proof into a form that a proof checker can verify step by step, letting the machine take on the line-by-line checking. But the barrier to formalization itself is not low either; how to convert natural-language reasoning into verifiable form without loss is still an open problem. It can be roughly understood as the following process:
claim = model.generate_proof(problem) # Low generation cost; may contain hidden errorsformal = formalize(claim) # Formalization may distort or failif verifier.check(formal): # Machine verifies step by stepreport(claim, "verified")else:report(claim, "rejected_or_inconclusive")
In this chain, the significance of evaluation lies not only in producing scores, but more in exposing the structural contradiction of “easy to generate, hard to verify.”
7. Implications for the Boundaries of AI Math Research
Taken together, this evaluation outlines the current capability boundaries of large models in mathematical research:
- Strong at reproduction, weak at pioneering. Models can reproduce known reasoning paradigms with high quality, but on truly open problems with no known answer, they have made no breakthroughs.
- Leaderboards are not research power. Refreshing benchmark scores mainly reflects “exam-taking ability,” which is not the same dimension as the research ability to independently produce new theorems.
- Verifiability is a double-edged sword. It makes math results easier to trust, but it also masks the enormous gap between “verifying a candidate solution” and “discovering a solution from scratch.”
- Grading is becoming the new bottleneck. As generation ability improves rapidly, verification and formalization become the key factors constraining the practical deployment of AI math research.
It needs to be clarified that “no model cracked the open conjectures” does not negate the mathematical value of large models. In scenarios such as assisting with literature review, generating candidate ideas, completing formalization drafts, and taking on mechanical derivations, models are already powerful efficiency tools for mathematical researchers. The boundary drawn by the evaluation is the highest standard line of “fully autonomous original research,” not the entire value of the model.
8. Practical Recommendations for Engineering
For teams that want to apply large models to math and theoretical computer science research scenarios, a few practical directions are worth considering:
Build a dual-track “generation + verification” system. Do not treat a model’s output directly as a conclusion. Formal verification, symbolic computation, unit-style checks, and similar steps should be incorporated into the workflow, letting the machine take on the checking work it is good at, and letting human energy focus on problem definition and direction judgment.
Prioritize high-value, verifiable subtasks. Introduce models first in verifiable, traceable steps such as literature search and review, idea enumeration, derivation drafts, and proof-formalization assistance, then gradually move toward harder steps.
Replace single leaderboards with a trustworthy evaluation system. A single leaderboard is easily optimized for through “exam-oriented” tuning. A more robust approach is to introduce comprehensive evaluations with multiple task types, open problems included, and strict verification of results, so as to approximate real research ability.
Rely on mature cloud and engineering infrastructure. Supporting the above workflow usually requires reliable compute scheduling, data storage, logging, and monitoring and alerting—general cloud capabilities as a foundation—so that large-scale, long-cycle evaluation and inference tasks can be managed stably with cost control.
Overall, the “exam paper” of theoretical computer science reflects not a failure of AI in mathematics, but the real gap that exists between “exam-taking” and “research.” Only by seeing this boundary clearly can large models be placed in the truly appropriate position in mathematical research.
