Evaluating LLM Math Ability: The Fundamental Gap Between Problem-Solving and Research

    This article examines what it means to evaluate large language models on mathematical and theoretical computer science tasks. It explains why math has become a preferred benchmark, what theoretical computer science (TCS) actually investigates, and why the ability to solve well-defined problems does not equate to the ability to conduct original research. Drawing on an evaluation spanning the full TCS domain, it shows that while frontier models excel on closed problems, none could independently resolve the open conjectures tested. The article then addresses the practical challenge of verifying machine-generated proofs through formal verification, outlines an evaluation workflow, and offers actionable recommendations for building more layered, credible AI assessment systems.

    1. Why Math Became the Litmus Test for AI

    Among the many dimensions used to evaluate large language models, math holds a special place. Not because it is somehow “higher” than writing or coding, but because it has one rare property: answers can be verified automatically and objectively by a machine.

    Once a proof is written, whether it is correct does not depend on a grader’s subjective preference. The chain of logic either holds together or breaks somewhere along the way. It works much like a Sudoku puzzle: filling in a grid can be extremely difficult, yet once it is filled, right and wrong are obvious at a glance. This asymmetry, where verification costs far less than solving, makes math one of the cleanest measuring sticks for a model’s reasoning ability.

    For this reason, vendors also tend to showcase math and reasoning scores first when releasing a model, since these results are the least contested and the easiest for the public to understand as proof of capability.

    2. What Theoretical Computer Science Actually Studies

    To understand a high-quality capability evaluation, we first need to be clear about what it examines. The core field here is Theoretical Computer Science (TCS). It does not study how to assemble hardware or write runnable code; it studies the boundaries and laws of “computation” itself.

    An analogy helps draw the distinction:

    • A programmer is like a driver, focused on engine performance and interior comfort, that is, “how to drive this particular route well.”
    • Theoretical computer science is like someone who studies the entire transportation system, asking “what is the theoretical fastest way to get from point A to point B?” and “is there a route that is doomed to be impassable no matter how much you optimize it?”

    TCS sits at the intersection of mathematics and computer science. Every problem revolves around “computation,” and its output is presented almost entirely in the form of mathematical proofs. This means that evaluating a model’s research ability in this field is essentially evaluating its ability to construct mathematical proofs.

    3. The Fundamental Gap Between Problem-Solving and Research

    Benchmark scores for current large models seem to be refreshed every two months, and the sheer number of leaderboards is dizzying. But we need to stay clear-eyed: leaderboards can prove that a model can “solve problems,” but they cannot reflect its “level of research.”

    The difference between the two is structural.

    Solving a problem is essentially cooking by following a recipe. The steps are ready-made, the patterns have been seen in training data, the standards are clear, and the final answer is either right or wrong. You will almost never see someone add chocolate to a sweet-and-sour pork dish, because “innovation” that deviates from the standard answer is simply marked wrong. A model converges to a known answer within a limited number of steps by relying on pattern matching and recombining chains of reasoning.

    Research, by contrast, is original cooking without a recipe. It has three features that problem-solving does not:

    1. There are no predefined steps; the researcher must explore the path themselves.
    2. The result cannot be predicted in advance; whether it will succeed and how long it will take can be guaranteed by no one.
    3. Even the person who poses the question may not know the answer; genuinely open conjectures have remained unresolved in the mathematical community for decades or even over a century.

    Therefore, a model that scores near the top on a standard problem set does not necessarily possess the ability to propose new paths, tolerate prolonged uncertainty, or make progress on problems that have no standard answer.

    4. Frontier Models’ Scorecard: Passing, but Not Breaking New Ground

    The conclusion of an evaluation built around the full domain of theoretical computer science is sobering: for a batch of open conjectures that remain unresolved to this day, not one of the reference models under evaluation could solve them independently.

    This result does not diminish the value of the models. On closed problems (those with known answers and standard solutions), frontier models perform impressively, handling multi-step logical reasoning, combinatorial optimization, algebraic manipulation, and other tasks. In some specialized subfields, they even approach or exceed the level of human contestants.

    But once you move into the realm of open problems, that is, directions where the problem is defined yet no complete proof has been given by any human, every model stops at “generating reasoning that looks plausible but contains flaws,” unable to truly complete a proof from scratch. This line marks the real boundary of current AI math research capability:

    • Models are good at reasoning within a known framework, that is, “walking faster along a path someone else has already walked.”
    • Models are not yet good at opening up the framework itself, that is, proposing entirely new proof ideas, defining effective auxiliary constructions, and recognizing which paths are worth pursuing.

    5. The Harder Problem: How to Grade a “Proof”

    Even if a model claims to have cracked a conjecture that went unsolved for decades, a practical question immediately arises: who verifies that it got the proof right?

    A model can output hundreds of pages of “proof” in a single run, packed with symbolic derivations and logical leaps. Reviewing it page by page is prohibitively expensive for human experts, and the experts themselves can make mistakes. This creates the core engineering challenge for putting evaluation into practice, the scalability of verification.

    The mainstream approach the industry uses to address this challenge is formal verification: translating natural-language proofs into a formal language that a machine can rigorously check, and having a proof checker verify the legitimacy of every reasoning step. As long as the formal statement is accepted by the checker, its correctness no longer depends on human judgment. In essence, this transforms “hard to verify” into “hard to formalize,” using a one-time, reusable toolchain to spread the cost of all subsequent verification thin.

    A framework for an evaluation workflow that distinguishes “problem-solving” from “research” can be illustrated as follows:

    1. def evaluate_model(model, task):
    2. if task.type == "closed_problem":
    3. # Problem-solving: has a standard answer for direct comparison; low verification cost
    4. answer = model.solve(task.prompt)
    5. return score_against_answer(answer, task.ground_truth)
    6. elif task.type == "open_conjecture":
    7. # Research: no known answer; requires both formal verification and expert review
    8. proof = model.generate_proof(task.prompt)
    9. if not formal_checker.verify(proof): # The formal checker validates each step
    10. return "proof_invalid" # The logic chain breaks; reject directly
    11. return expert_review(proof) # After passing machine checks, hand off to expert review
    12. # For either task type, all logs and intermediate artifacts should be persisted throughout
    13. # to enable traceability, reproducibility, and later auditing

    In this workflow, the evaluation system itself also requires solid engineering support. Batch scheduling of tasks and models can rely on container platforms for orchestration. Persistence of massive amounts of proof text and intermediate artifacts can be handled by object storage. Asynchronous verification tasks can be smoothed and decoupled through a message queue. And key metrics such as elapsed time, failure rates, and scores across dimensions during evaluation should be connected to logging and monitoring/alerting services, so that trends in model capability can be continuously observed.

    6. Implications and Practical Recommendations for Evaluation Systems

    The greatest value of this evaluation is not the score given to any single model, but the reminder to the entire industry: capability assessment must distinguish between levels. Based on this, here are a few actionable recommendations:

    1. Separate “problem-solving ability” and “research ability” into two independent evaluations. The former measures pattern reuse and multi-step reasoning; the latter measures originality and the ability to advance open problems. Conflating the two will seriously overestimate a model’s research potential.

    2. Introduce open problems as probes for research ability. Select problems that humans have not yet solved or have only partially progressed on, and examine whether the model can produce new contributions that withstand formal verification, rather than merely reproducing known conclusions.

    3. Prioritize building a formal verification pipeline. Upgrading verification from “manual page-by-page review” to “automatic machine checking plus spot-check review by experts” is the prerequisite for evaluation to become scalable and trustworthy.

    4. Be wary of the misleading nature of leaderboard numbers. A rising score reflects “progress in training and problem-solving techniques,” which cannot be simply equated with “a breakthrough in research ability.” A genuine research breakthrough should be defined by verifiable new proofs, not by an attractive percentile.

    In summary, frontier large models have already traveled quite far down the road of “problem-solving,” but the distance from “being able to solve problems” to “being able to do research” is not a gap that can be crossed simply by piling on parameters. It is a deep divide rooted in originality and certainty of verification. Viewing this boundary rationally, neither blindly optimistic nor quick to declare it impossible, is the correct starting point for building a reliable AI evaluation system.