LLM Math Benchmarks: Why Solving Problems Isn't the Same as Doing Research

    A full-field benchmark in theoretical computer science (TCS) shows that today's frontier LLMs handle competition problems, textbook exercises, and formal derivations well, yet none could independently resolve any long-standing open conjecture. This article explains what TCS actually studies, why its closed-versus-open problem split separates pattern reproduction from genuine research, why grading a hundreds-page proof is harder than generating one, and what developers should take away for model selection and system design.

    1. A Full-Scope Exam in Theoretical Computer Science

    A recently released benchmark covering the entire field of theoretical computer science (TCS) has drawn wide discussion. It is co-signed by several leading researchers and heads of frontier model teams, and it covers the most representative set of large language models available at the time. The methodology is straightforward: build an exam that spans the subfields of TCS, have the strongest models of the moment answer every question, then measure how many they got right and how well.

    TCS is frequently misunderstood. It is not about “fixing a machine,” nor about “writing code that runs.” Its actual subject is computation itself: how many steps does a problem require in the worst case? Is there a lower bound that makes some class of problems provably infeasible to solve efficiently? Can one problem be reduced to another?

    An analogy helps. If a programmer is a driver focused on the engine, the interior, and fuel economy, TCS studies traffic itself — what the theoretically fastest trip between two points is, and whether certain routes are permanently impassable. It sits at the intersection of mathematics and computer science. Every question revolves around computation, and the core research method is rigorous mathematical proof rather than a program that happens to run.

    2. Why This Benchmark Differs From Typical Leaderboards

    Claims that “AI is conquering mathematics” have been circulating for a while. Math became the battleground AI labs most like to showcase for a practical reason: the correctness of a mathematical proof can be checked by a machine relatively objectively. It resembles solving Sudoku — filling the grid from scratch is hard, but verifying a completed grid cell by cell is easy. This asymmetry between generation and verification is what lets math be the area where large models most easily produce uncontested results.

    That is exactly why two very different capabilities must be separated.

    One is problem-solving. It amounts to cooking from a recipe: the steps are largely already given, the answer key is fixed, the rubric is clear. During training, models see large volumes of similar problem types and solution patterns; when they meet a new problem, what they do is pattern matching plus compositional generalization. It is why you never see anyone add chocolate to a sweet-and-sour pork dish — the training distribution has implicitly constrained the space of “reasonable answers.”

    The other is research. There is no pre-set recipe, no prescribed sequence of steps, and nobody knows in advance whether the route will work out. Worse, for many problems even the person posing them does not know the answer: they are either long-standing open conjectures or problems that researchers have failed to crack after years of attempts.

    TCS was chosen as the yardstick precisely because it separates these two capabilities cleanly. Mature closed problems test pattern reproduction; open conjectures test genuine exploration from zero.

    3. Findings: No Model Solved the Hard Problems, and That Says a Lot

    The benchmark’s central conclusion is restrained: across the open conjectures that remain unresolved today, not one of the participating models managed to solve any of them on its own.

    The result sounds like a cold shower, but it carries three pieces of useful information.

    First, the strength of current models lies in solving problems within known frameworks. Whether the task is a hard competition problem, a textbook exercise, or a derivation with an established solution pattern, models show considerable reasoning and generation ability. Their benchmark scores are refreshed roughly every month or two, at a pace that draws attention.

    Second, there is a real gap between leaderboard scores and research capability. A high score only proves the model can do problems; it says nothing about whether it can do research. This is like acing an exam — it does not imply the ability to independently discover new theorems. The former is optimization within fixed boundaries; the latter means expanding, or even redrawing, those boundaries.

    Third, what is missing on the way from solving to research is the ability to pose problems autonomously and explore unknown routes. Research requires a model to do more than follow known paths: it must keep searching, self-correct, and eventually deliver a verifiable new conclusion under conditions where information is incomplete, direction is unclear, and most attempts are wrong. This is currently the weakest link in large models.

    4. The Harder Part: Grading

    More troublesome than “cannot solve” is “cannot grade accurately.”

    Imagine a model outputs hundreds of pages of “proof” claiming to have cracked a conjecture that stood open for decades. Who grades it? Having a person check it page by page is extremely costly and highly error-prone; a machine can check the formal consistency of the derivation chain, but it cannot guarantee with certainty that the conclusion is correct.

    A short piece of illustrative code makes the difference in difficulty between the two problem classes more visible:

    1. def evaluate(model, problem):
    2. # The model produces an answer or a proof
    3. answer = model.solve(problem)
    4. if problem.answer_is_known:
    5. # Closed problem: a reference answer exists, so scoring is
    6. # automatic and low-cost
    7. return compare(answer, problem.gold_answer)
    8. else:
    9. # Open conjecture: no reference answer exists, so review depends
    10. # on humans or semi-automated pipelines.
    11. # At most, a machine can check the formal consistency of the
    12. # derivation; it cannot guarantee the conclusion is correct.
    13. return check_proof_consistency(answer) # far from a complete verdict

    The pseudocode exposes the essence of the problem. For closed problems, verification is a mechanical comparison at polynomial cost. For open conjectures, there is no “reference answer” to compare against at all. What a machine can do is check whether each step is internally consistent and obeys the rules — but “every step is self-consistent” does not imply “the overall conclusion holds.” This asymmetry between verification and generation is the most fundamental technical obstacle in assessing frontier models’ mathematical ability, and a key problem this direction still has to solve.

    5. What This Means for Developers

    Setting aside the grand narrative of “can AI do math,” the benchmark offers a few practical takeaways for engineering practitioners.

    • Choose models by scenario, not by a single leaderboard. If the task has clear boundaries and a reference solution — structured reasoning, formatted proof completion, standard algorithm problems — frontier models already provide meaningful help. If the task is essentially open-ended research exploration, expectations should be kept cautious.
    • Make verification a first-class citizen in system design. When applying large models to math or logical reasoning, instead of expecting the model to return the final answer directly, design a two-stage “generate + verify” pipeline: let the model handle exploration and generation, then use an independent formal tool or rule engine to check the output, reducing the risk of results that look right but are wrong.
    • Value traceability and reproducibility. Keeping inspectable intermediate artifacts at every derivation step makes it possible to locate an error quickly, rather than facing a large block of black-box output with no way to proceed.

    It is also worth stressing that training these frontier models and running evaluations at scale place high demands on infrastructure — storing and retrieving massive datasets, scheduling compute for distributed training and inference, and observing and monitoring the experiment process. In practice, teams typically rely on general-purpose cloud capabilities to carry these pieces: object storage to manage datasets and model artifacts, container platforms to schedule training and inference jobs elastically, and monitoring and alerting services to track experiment status and resource consumption. Platforms such as Baidu AI Cloud provide exactly this kind of general infrastructure, letting teams concentrate on the models and algorithms themselves rather than low-level operations.

    6. Conclusion: The Boundary Marks the Direction

    The most memorable point of this benchmark is not whether “AI solved it” or “AI failed to solve it” for some specific problem. It is that the evaluation draws a clear line around the current capability boundary of large models: efficient solving within established frameworks, persistent weakness in front of open unknowns; strong generation, weak independent verification and weak production of new conclusions.

    For mathematical research, that is both a sober reminder and a clear direction. Genuinely valuable research ability lies not in maxing out scores on familiar problem types, but in whether one can open a new path where nobody points the way — and make the correctness of that path acceptable to both machines and humans. Whoever crosses the gap between generation and trustworthy verification first comes closest to the threshold of AI participating in real mathematical research.