Evaluating LLMs' Math and Research Capability: Blind Spots and Breakthroughs in Existing Benchmarks

    Benchmark scores for large language models keep rising, yet they reveal little about whether a model can actually do research rather than just solve problems. Using theoretical computer science (TCS) as the evaluation domain, this article examines why problem-solving and research are fundamentally different capabilities, why automatically verifying hundreds of pages of proofs is a hard engineering problem, and why math has become AI's most controllable yet most misleading testing ground. It shares the sobering finding that no current frontier model can independently crack open conjectures, identifies three capability gaps—long-chain reasoning robustness, proposing effective conjectures, and self-correction—and offers five practical principles for building a trustworthy, research-grade evaluation system.

    Introduction: Evaluation Anxiety Behind the Boom in Benchmark Scores

    Over the past two years, leaderboards for evaluating large language models have been refreshed every month or two. Scores across math, coding, and logical reasoning keep climbing, and the constant churn of rankings is dizzying. Yet a pointed question persists: what do these scores actually measure? They can show that a model “can solve problems,” but they say almost nothing about whether it is capable of “doing research.” When a group of researchers turned their attention to theoretical computer science (TCS)—the interdisciplinary field where mathematics meets computer science—and used a comprehensive, domain-wide evaluation suite to test the strongest models available today, the findings were telling: the models’ problem-solving ability is already impressive, yet none of them could independently crack a genuinely open problem.

    This article steps beyond a single event to examine a broader, more engineering-relevant question: how do we evaluate the mathematical and research capabilities of large models, where are the blind spots in existing benchmarks, and how can we build a more trustworthy evaluation system?

    Problem-Solving Is Not Research: A Fundamental Divide Between Two Capabilities

    To understand the limitations of evaluation, we first need to distinguish two kinds of ability.

    Problem-solving is essentially “cooking by following a recipe.” A problem has clearly stated conditions, a relatively fixed solution path, a single accepted answer, and steps that can be traced one by one. Models perform well on competition math and standardized exam questions precisely because they have learned these “recipes” from massive corpora and can quickly match them to the appropriate “cooking procedure.”

    Research is entirely different. There are no clearly defined problem boundaries, no ready-made steps to follow, and it is not even known in advance whether a final answer exists. An open conjecture may resist attack for decades, and even the person who posed it may not know the correct answer. Researchers must propose hypotheses, design paths, endure extensive failure, and search through countless trials and errors for the one route that barely exists.

    The gap between these two is exactly what makes evaluation design difficult: measuring research with a problem-solving ruler systematically overestimates a model’s ability. When benchmark problems all have close templates in the training data, high scores reflect memorization and pattern matching more than genuine creative reasoning.

    Theoretical Computer Science: A Question About “The Limits of Computation”

    Why choose theoretical computer science as the evaluation target? Because the discipline sits precisely at the intersection of mathematics and computer science, every problem revolves around “computation” itself, and it depends heavily on rigorous mathematical proofs.

    If computer engineering is like driving, then ordinary programmers care about engine performance and interior features, while theoretical computer science cares about traffic itself: what is the theoretically fastest way to get from point A to point B? Does some route turn out to be mathematically impossible? Answers to these questions do not depend on any specific machine’s implementation; they are abstract characterizations of the nature of computation.

    This “proof as the final deliverable” property makes TCS an ideal evaluation ground: it has clear correctness criteria, yet is abstract and open enough to genuinely distinguish “problem-solving” from “research.” Such evaluations typically cover multiple subfields—complexity theory, algorithm design, computational geometry, cryptography fundamentals, combinatorial optimization, and more—with difficulty ranging from standard textbook exercises all the way to unsolved open problems.

    The Grading Challenge: How to Automatically Verify Hundreds of Pages of Proofs

    Evaluating TCS ability inevitably runs into an engineering problem: how do we determine whether a proof is correct?

    A model can generate hundreds of pages of derivation in a single pass, and manually reviewing them page by page is prohibitively expensive. Errors in proofs are also extremely subtle—a small deviation in a symbol definition or a missed boundary condition can invalidate an entire line of argument. This imposes two key requirements on evaluation design.

    First, tiered answer verifiability. The evaluation system must annotate each problem with its verification method according to the nature of the problem: problems with standard answers can be checked automatically; problems requiring proof rely on formal verification or expert review; and open problems can only be recorded as “solved or not,” with no intermediate score.

    Second, prioritize formalizable proofs. Math is often regarded as AI’s most “controllable” battleground precisely because it has an advantage scarce in other fields—correctness can be checked automatically by machines. This is like a Sudoku puzzle: filling it in is hard, but once it is filled, verifying whether it is correct is easy. Formal proof tools can transcribe a mathematical argument into a form that programs can verify step by step, turning the expensive, subjective process of “judging whether a proof is true” into an objective, reproducible automated pipeline.

    The following pseudocode summarizes the skeleton of a verifiable evaluation pipeline:

    1. def evaluate_proof(problem, model_answer, difficulty):
    2. # 1. Tiering: first determine the problem type
    3. if problem.has_standard_answer:
    4. return exact_match(model_answer, problem.gold)
    5. # 2. Attempt formal verification (the highest tier of verifiability)
    6. formal_proof = try_formalize(model_answer)
    7. if formal_proof and proof_checker.verify(formal_proof):
    8. return {"status": "verified", "score": 1.0}
    9. # 3. Fall back to expert / multi-model cross-review (high cost, strong subjectivity)
    10. if problem.type == "open_problem":
    11. # Open problems cannot receive an intermediate score; we can only record whether they were truly solved
    12. return {"status": "unsolved", "score": None}
    13. reviews = [expert_or_ensemble_review(model_answer) for _ in range(k)]
    14. return aggregate(reviews) # Consistency aggregation to reduce single-point misjudgment

    The value of this pipeline lies not in the specific implementation but in a core principle of evaluation design it reveals: the higher the verifiability, the more trustworthy the evaluation conclusion; the more open the problem, the harder it is to give a meaningful intermediate score.

    Why Math Is the Most “Controllable” Battleground for AI

    Before asking whether AI has “replaced” a field, it is worth understanding its capability boundaries.

    AI shares a common weakness across many domains: it makes errors that cannot be verified. In text generation, image generation, and open-ended Q&A, a fluent-looking answer may harbor a fundamental error, and verifying it often costs as much as redoing the work. These “high non-verifiability” tasks are inherently unfavorable for objective evaluation.

    Math is the opposite. Proof problems in particular have correctness that can be automatically judged by machines, with errors precisely located. This verifiability makes math the source of the achievements large models can attain most easily and least controversially. Evaluation researchers focus on this field precisely because it is the “pure sample” for testing the depth of a model’s reasoning—few confounding factors, objective criteria, and reproducible conclusions.

    But this also creates a cognitive trap: verifiability does not equal solvability. A high score in a field may only indicate that the model is good at the subset that can be automatically graded, not that it truly understands the essence of the problems.

    A Reality Check: Why No One Can Crack Open Conjectures

    Let us return to the original question. When frontier models are placed in a full-domain TCS evaluation and face the open conjectures that remain unresolved to this day, the conclusion is sobering: no tested model was able to genuinely solve these problems on its own.

    This does not mean the models are incapable. On the contrary, they perform quite well on standard-difficulty problems with clear solution paths. The real gap shows up in three areas:

    • Robustness of long-chain reasoning: A proof of an open problem may require dozens or even hundreds of consecutive derivation steps, and models tend to produce links that “look plausible but are actually broken” over long chains.
    • The ability to propose effective conjectures: The first step of research is often “guessing the right direction,” but models tend to combine within known patterns and lack the ability to genuinely step outside the framework and propose new hypotheses.
    • Self-correction and metacognition: Human researchers, when stuck, reflect on whether they have taken the wrong path and switch strategies; models tend to keep pushing “confidently” in the wrong direction.

    Together, these three points lead to one conclusion: what current large models are good at is pattern recognition and pattern combination, not the original hypotheses and deep reflection that research requires. This also explains the disconnect between high problem-solving scores and genuine research breakthroughs.

    How to Build a Trustworthy, Research-Grade Evaluation System

    If you want to design an evaluation scheme oriented toward research capability for your own model or team, you can start from the following directions:

    1. Design problems in tiers by verifiability. Divide the evaluation set into four tiers—“standard-answer problems / formalizable-proof problems / expert-review problems / open problems”—and apply different scoring strategies to each, avoiding a single uniform standard that conflates different kinds of ability.

    2. Introduce a formal verification step. For proofs that can be formalized, transcribe them into a machine-verifiable form as much as possible, turning subjective scoring into objective automated checking. This not only improves evaluation credibility but also significantly reduces the manual cost of large-scale evaluation.

    3. Use cross-review instead of single-point judgment. For problems that cannot be verified automatically, use multi-model cross-evaluation or expert sampling review, and reduce misjudgment through consistency aggregation while retaining disputed samples for later manual adjudication.

    4. Distinguish “scores” from “solution status.” Open problems should not receive an intermediate score but should be recorded as an independent dimension—“whether truly cracked”—to prevent a single aggregate score from masking a model’s weaknesses in creative research.

    5. Focus on the process, not just the result. For proof tasks, assessing the coherence of the reasoning chain, the correctness of key steps, and where errors occur reflects true capability better than merely checking whether the final answer matches.

    In practice, an evaluation pipeline often needs to work with infrastructure such as compute scheduling, log retention, and result monitoring to remain stable and reproducible at scale. With containerized job orchestration, message queues for asynchronous grading, and a unified logging and alerting system, a research-grade evaluation can evolve from “runs once” into an engineering capability that is “sustainable, observable, and traceable.”

    Conclusion

    Large-model evaluation is shifting from “who can solve problems” to “who can do research.” Theoretical computer science, with its distinctive property of “proof as the deliverable and verifiability as the feature,” has become an ideal ruler for measuring the depth of a model’s reasoning. The real value of this evaluation is not how many points a given model scored, but that it drew a clear boundary: current models are skilled at combining and solving within existing frameworks, yet still lack the ability to independently propose and crack original, novel problems. For developers, understanding this boundary—and designing more rigorous evaluation and training strategies accordingly—is the most worthwhile takeaway this round of evaluation buzz leaves for the technical community.