Evaluating LLMs' Math Research Capabilities: Problem Solving, Verification, and Boundaries
A joint assessment by leading researchers found that today's most advanced large language models still cannot independently crack open mathematical conjectures. This article explains why theoretical computer science serves as a meaningful yardstick for this test, why "solving problems" and "doing research" are fundamentally different abilities, how verification costs shape evaluation design, and what engineering teams should take away when choosing or assessing models.
1. The Viral “AI Math Myth” and a Cold Splash of Reality
For a while now, headlines about “AI cracking hard math problems” have kept making the rounds. These discussions rest on a nearly universal assumption: mathematics is the field where AI is most likely to produce “undisputed results.” The reasoning is intuitive — a coding bug may only surface in production, and a translation error may go unnoticed, but a math proof can be checked automatically by a machine. It’s like filling in a Sudoku: run it through once and you know whether it’s valid.
That’s precisely why a study co-signed by several top researchers carries so much weight. After evaluating a batch of the most advanced large language models, the authors concluded that not one of the models tested could independently solve any of the open conjectures that remain unresolved to this day.
This conclusion deserves attention because it puts a long-conflated question on the table: what capability do LLMs’ ever-rising benchmark scores actually measure? To answer that, we first need to understand the disciplinary background this evaluation relies on.
2. Why Use Theoretical Computer Science as the Yardstick
Theoretical Computer Science (TCS) is often likened to the “Olympics” of mathematics. But it doesn’t study how to fix a computer or how to write good code — it studies the nature and limits of “computation” itself.
Think of it this way: a programmer is like a driver, focused on engine performance and interior features. TCS, by contrast, studies the entire transportation system — how fast you can theoretically get from point A to point B, and which routes are impossible in principle. It sits at the intersection of mathematics and computer science, with every question revolving around “computation” and every method relying primarily on rigorous mathematical proof.
Choosing such a field as the yardstick is deliberate:
- Correctness is decidable: Whether a proof is right has an objective standard, reducing the noise of subjective scoring.
- Difficulty has a gradient: It includes both routine textbook problems and open conjectures that have resisted attacks for decades.
- It separates levels of capability: Problem-solving ability and research ability get cleanly distinguished in this domain.
3. “Problem Solving” vs. “Research”: Two Fundamentally Different Things
To understand the evaluation’s conclusion, we first need to clarify a distinction that’s often overlooked.
Solving problems is essentially cooking from a recipe. The steps are already laid out, and the evaluation criteria are clear. Just as no one adds chocolate to a sweet-and-sour pork dish, standard problems typically have well-defined solution patterns — the model simply retrieves and recombines the matching pattern from its training data. This capability improves rapidly as models grow larger and training data expands, which explains why benchmark scores climb almost every two months.
Research, on the other hand, is pioneering cooking with no recipe. There are no prescribed steps, and no one can say in advance whether it will work out. Worse, for many open problems even the person posing the question doesn’t know the answer — if no ground truth exists, how do you verify a model’s so-called “solution”?
The gap between the two can be summarized across three dimensions:
| Dimension | Problem Solving | Research |
|---|---|---|
| Steps | Ready-made, pattern-based | No established path |
| Answer | Known ground truth | Often unknown |
| Goal | Hit a known pattern | Explore unknown frontiers |
The evaluation’s sober conclusion confirms exactly this: current models are already strong at “solving problems,” but a fundamental gap remains in “doing research.” Benchmark scores demonstrate a combination of pattern matching and reasoning — not the original research capability of posing new questions and building new proof frameworks.
4. The Underestimated Challenge: Who Grades the Papers?
Beyond capability itself, the evaluation exposed a practical engineering dilemma — verification cost.
When a model outputs hundreds of pages of “proof” in one go, claiming to have cracked a conjecture that went unsolved for decades, the question immediately turns sharp: who has time to read it page by page? And who is capable of judging whether it’s actually correct?
This gives rise to two parallel verification approaches:
- Formal automated checking: Express the proof in a form that a formal system can recognize, then let a proof checker verify its logical validity step by step. This corresponds to the optimistic view that “machine-checkable” correctness can be delegated to machines — but it requires the model’s output to be formalizable and machine-checkable in the first place.
- Manual expert review: For the parts that can’t be automatically decided (especially open-ended, exploratory problems), you have no choice but to rely on long-term review by domain experts — costly and slow.
A workable evaluation pipeline roughly looks like this (illustrative pseudocode):
def evaluate_model(model, benchmark):results = []for problem in benchmark.problems:# 1. Model generates a candidate solution / proofcandidate = model.generate(problem.statement)# 2. Attempt automated formal verificationverdict = formal_checker.verify(candidate, problem.spec)# 3. Route: record auto-decidable cases, send the rest to humansrecord = {"id": problem.id,"difficulty": problem.difficulty, # routine problem / open conjecture"category": problem.area, # subfield it belongs to"auto_verified": verdict, # True / False / None"needs_human_review": verdict is None,}results.append(record)return summarize(pass_rate=...,auto_check_ratio=...,human_review_load=...,)
This pipeline deliberately treats “automated verification coverage” and “manual review load” as first-class metrics — because in a serious mathematical evaluation system, the ability to decide correctness efficiently and reliably matters just as much as how well the model answers.
5. Key Tradeoffs in Evaluation Design
Doing this well means the benchmark’s design itself is more demanding than it first appears. Overall, a credible evaluation of math and research capability usually has to balance several tradeoffs:
- Breadth of domain coverage: You can’t cherry-pick only subfields where problems are easy to write, or you’ll systematically overestimate model capability. This is exactly why covering the “entire field” of theoretical computer science matters.
- Tiered difficulty: Including both routine problems and open conjectures is what reveals the capability ceiling, rather than keeping the model in its comfort zone.
- Guarding against data contamination: Benchmark questions and answers must be strictly isolated from public corpora, otherwise a model may have “memorized the answer” rather than genuinely solved the problem.
- A closed verification loop: Pair each category of problem with an executable means of judgment (automated or manual), and explicitly record the cost and confidence of that judgment.
6. Implications for Developers and Engineering Practice
Setting aside the headline narrative, this evaluation offers concrete, pragmatic lessons for the engineering side.
First, don’t infer research capability directly from benchmark scores. When selecting or assessing a model, be clear about whether you’re testing “problem solving” or “research.” If your goal is to assist with day-to-day development (code completion, standard algorithm problems, structured reasoning), current models’ problem-solving ability is already quite usable; if your goal is frontier exploration, you must soberly recognize the current capability boundary.
Second, treat “verifiability” as a first-class design concern. When building AI-assisted math or rigorous engineering scenarios, prioritize task forms whose results can be automatically verified, and constrain unreliable free-form generation within checkable bounds. On the engineering side, you can use container platforms to isolate execution environments, use logging services and monitoring/alerting to track the verification process, and use object storage to persist evaluation datasets and results — forming a traceable evaluation loop.
Third, budget for manual review. When automated verification can’t provide full coverage, expert review is an unavoidable step. The load should be quantified when designing the evaluation system, rather than pretending it can be fully automated away.
7. Conclusion
What makes this evaluation truly valuable isn’t the score it gave any particular model, but a line it clearly drew: LLMs are already strong at “solving problems,” but “research” is another matter entirely. Being able to cook a standard dish from a recipe and being able to invent a dish no one has ever made are separated by exactly the boundary AI has yet to cross.
For engineering practitioners, rather than chasing ever-rising benchmark scores, it’s better to focus on more fundamental questions: “Is the task verifiable?”, “Does the evaluation cover real difficulty?”, and “Is the verification cost manageable?” Acknowledging the boundary is precisely the precondition for using capability effectively.
