Benchmarking LLM Math Research: Where Problem-Solving Ability Ends and Open Research Begins

    LLM scores keep improving, but high marks on math benchmarks mostly prove that models can solve problems, not that they can do research. This article explains why theoretical computer science (TCS) is a useful stress test for mathematical reasoning, why machine-checkable proofs give AI an easier place to score, and what one comprehensive TCS benchmark found when leading frontier models were set against genuinely open conjectures. You will learn the limits of natural-language proofs, how a layered "generate → formalize → verify → expert review" pipeline works, which four capabilities define the current research boundary and how to measure them, and what a reproducible evaluation setup needs to include.

    1. Theoretical Computer Science: A Stress Test for LLM Reasoning

    Theoretical computer science (TCS) sits at the intersection of mathematics and computer science. If a working programmer is like a driver who cares about the engine and the interior, TCS cares about the entire transportation system: what the theoretical fastest route between two points is, and whether some paths can never be traversed at all. It studies neither how to repair a machine nor how to write code that runs. It studies computation itself, and its primary research tool is proof.

    Collaborative research involving leading academic groups and AI teams has recently produced a comprehensive benchmark that covers the full breadth of TCS, and several frontier LLMs have been put through it.

    2. The Gap Between Solving Problems and Doing Research

    LLM leaderboards are refreshed every month or two, but most of them demonstrate that a model can solve problems. They say little about its research ability.

    • Solving problems is much like cooking from a recipe: the steps are ready-made, there is a reference for grading, and as long as the model has seen enough similar problems in its training data, it can reproduce the standard solution with high probability.
    • Doing research has no prescribed moves. Nobody can say in advance which path to take or whether it will lead anywhere, and many problems do not even have known answers to their authors. The task is to construct a new proof from scratch.

    Using benchmark scores to paper over this difference leads outside observers to overestimate a model’s research potential. A benchmark with real value has to target problems whose answers are unknown.

    3. Machine-Checkable Math: Where AI Scores Relatively Easily

    The achievements AI companies argue about least are usually the mathematical ones, and the key factor is verifiability. A mathematical proof can be checked by machine using formal verification tools such as automated theorem provers and interactive proof assistants. One point deserves emphasis: a natural-language proof normally has to be formalized first, by a human or a semi-automated tool, before a machine can check it. It cannot be directly “translated” into a machine-checkable form.

    Common reasons formalization fails include ambiguous or skipped steps in the natural-language text, references to undefined concepts, notation or premises that do not line up, creative constructions that are hard to express formally, and a formalized goal that is not equivalent to the original proposition.

    This asymmetry, where generation is hard and verification is easy, is what makes mathematics a relatively objective ground for testing a model’s reasoning.

    An important counterpoint: passing formal verification does not mean the work has research value. Even a machine-verified proof has to be judged against the difficulty of the conjecture itself, the length of the proof, whether it introduces new concepts, and expert review, before its real contribution can be assessed.

    4. The Central Finding: No Model Independently Solved an Open Conjecture

    Frontier models look strong on standardized problems. On genuinely open research problems, however, the benchmark delivered a sobering result: across the open conjectures listed in this evaluation, none of the participating models independently produced a solution.

    This conclusion holds only within the problem set and measurement criteria defined by the evaluation (the specific benchmark name, the number of open conjectures, the model versions tested, and the exact criteria should be taken from the original evaluation report). It should not be generalized into a universal judgment about all mathematical problems.

    Nor does it negate the value of these models. They show clear gains at reproducing known proofs, assembling existing lemmas, and supporting draft derivations. When it comes to proposing entirely new constructions or making original breakthroughs across conceptual gaps, however, a distinct ceiling remains. Today’s LLMs are closer to diligent, well-read research assistants than to researchers capable of opening up a new direction on their own.

    5. The Hard Part: Grading a Several-Hundred-Page Proof

    The thornier stage is grading. A model may produce several hundred pages of “proof” in one pass. Who has the time to read it page by page?

    The key premise: a formal verifier checks only the “formalized goal” (formalized_proof), not the original natural-language text (proof). Downstream verification therefore has to point at the formalized artifact. A workable approach is layered verification:

    1. # Evaluation pipeline sketch: generate -> formalize -> automatic check -> human review
    2. def evaluate_on_tcs(model, problem):
    3. natural_proof = model.generate_proof(problem.statement)
    4. # 1) Formalizable cases: formalize first, then have the verifier check the
    5. # formalized goal step by step
    6. if problem.is_formalizable:
    7. formalized_proof = formalizer.translate(natural_proof, problem.statement)
    8. return {"verdict": "verified", "ok": formal_verifier.check(formalized_proof)}
    9. # 2) Open conjectures or cases that cannot be decided automatically:
    10. # send them to the expert review queue
    11. if problem.is_open_conjecture:
    12. return {"verdict": "pending", "task": expert_review.submit(natural_proof)}
    13. # 3) Everything else: sampling plus consistency cross-checks
    14. return {"verdict": "sampled", "ok": cross_check(natural_proof)}

    The core principle is straightforward: automate whatever can be machine-checked, fall back on experts for what cannot, and use sampling and cross-validation for the cases in between, to control cost without losing rigor. No single automated stage is sufficient to support a high-stakes claim such as “an open problem has been solved.”

    6. Turning “Boundaries” Into Measurable Metrics

    A comprehensive benchmark can sketch the current boundaries of LLM performance in mathematical research. The more useful step is to express those boundaries as measurable metrics:

    Boundary What it means Measurable metric
    Original construction Strong at recombination and analogy, weak at inventing new objects or strategies from nothing Original construction success rate
    Long-horizon reasoning Error accumulation and logical drift become more likely as the number of steps grows Long-horizon reasoning error rate
    Self-verification Lacks a reliable self-check on whether a proof is correct Self-verification consistency
    Problem posing Research begins with asking good questions; models mostly answer questions that already exist Problem-posing quality score

    7. Engineering the Evaluation: A Minimum Reproducibility Checklist and Compliance

    To make the evaluation runnable, reproducible, and extensible, start by fixing a minimum reproducibility checklist:

    • Problem set version: the version number of the problem collection and the criteria used to select items.
    • Proof file format: the storage format for both natural-language and formalized proofs.
    • Verifier version: the automated theorem prover or interactive assistant used, including its kernel version.
    • Container image: a pinned execution environment, with different models isolated from one another.
    • Expert review sheet: review items, scoring criteria, and a recording template.
    • Failure sample archive: retained failure traces, so failure modes can be located.
    • Measurement criteria: how the denominator is defined, how success is determined, and how ties are resolved.

    These pieces can be assembled on general-purpose infrastructure such as object storage, container platforms, message queues, and monitoring with alerting, combining machine-checkable verification and human review into one stable, reproducible automated pipeline. (The cloud services mentioned here are general architecture examples only; they imply no particular product capability or guarantee of results. Actual selections depend on the deployment.)

    Risk and compliance: proof drafts, expert opinions, and model outputs may contain sensitive or non-public information. Access control, redaction, and retention and destruction policies should be in place so that evaluation data stays compliant throughout its lifecycle.

    8. FAQ

    • Can LLMs complete mathematical proofs independently? Across the open conjectures listed in this evaluation, no. Reproducing known proofs and combining lemmas are their strengths.
    • How is a TCS evaluation run? Through a layered pipeline: generate → formalize → automatic verification → expert review.
    • Can formal verification fully replace experts? No. It checks only whether the formalized goal is valid, not whether the work has research value.
    • What are common evaluation mistakes? Extrapolating research ability from problem-solving benchmarks, treating “passes verification” as “has research value,” and ignoring measurement criteria and model versions.

    9. Conclusion

    A comprehensive TCS-oriented evaluation sends a clear signal: LLMs have become fluent at solving problems, yet they linger at the doorway of doing research. In an evaluation context, a benchmark score is not research ability, and reproduction is not innovation. Acknowledging and quantifying these boundaries is precisely where the evaluation earns its value. Treating models as tools that amplify research productivity, rather than as replacements for researchers, is both the more pragmatic and the closer-to-truth position today.