LLM Math Evaluation: From Problem Solving to Proof

    This article examines what a recent evaluation of large language models on theoretical computer science tasks can and cannot reveal about their mathematical abilities. It covers evaluation design, three capability levels, proof verification, practical use, and the limits of the conclusions.

    Large language model performance in mathematics has become an important reference for assessing intelligence. Mathematical ability is not a single dimension: there is a wide gap between basic arithmetic and frontier theorem proving. A recent evaluation covered several subfields of theoretical computer science and systematically tested mainstream LLMs. This article reviews what the study shows across four areas—evaluation design, capability levels, proof verification, and practical application—and which conclusions still require caution.

    Why Use Theoretical Computer Science for Evaluation?

    Theoretical computer science studies the laws and limits of computation itself. For example, it asks whether a problem has an efficient algorithm and where the capability boundaries of different computational models lie. Answers to such questions usually take the form of rigorous mathematical proofs.

    Using this field as an evaluation setting has two practical advantages. First, task difficulty spans a wide range, from algorithm analysis problems in undergraduate courses to unresolved research questions. This makes it possible to distinguish which capability level a model has reached. Second, whether a proof is correct can in principle be checked rigorously, so the scoring criteria are relatively clear.

    However, the specific subfields covered, the number of tasks, the model versions, and the sample size determine how far the conclusions can be generalized. Before citing the conclusions, readers should return to the original report and check these parameters. This article does not repeat unverified numbers.

    Three Levels of Capability

    Level 1: Basic Computation and Routine Proofs

    The first level involves basic calculation and routine proofs. Standard algebraic simplification, formula derivation, and proof problems with fixed solution methods are areas where frontier models perform relatively well. These tasks have clear solution paths and are closer to solving problems by following established steps. Models have seen many similar patterns in training data, so their performance is relatively stable.

    Level 2: Multi-Step Reasoning and Integrated Problem Solving

    The second level involves multi-step reasoning and integrated problem solving. When a problem requires the consecutive use of multiple theorems or connections between different branches of mathematics, accuracy drops noticeably. Errors in a long reasoning chain accumulate, and models have more difficulty adjusting their overall strategy midway in the way human researchers do. Performance on combinatorics and complex algorithm analysis tasks is especially unstable at this level.

    Level 3: Open Problems and Creative Argumentation

    The third level involves open problems and creative argumentation. For long-standing open problems, the models in the evaluation did not produce recognized breakthroughs. Such problems have no ready-made path, and researchers themselves are often uncertain about the direction of an answer. Under the evaluation settings and samples used, this result suggests that the models’ ability to produce original proofs is limited. It does not directly support the conclusion that models will never be able to do research, but it is enough to show that there is still a clear distance between current capabilities and research-level problems.

    Solving Exercises vs. Doing Research

    Answers to exam-style problems already exist, and the solution process is closer to an advanced form of knowledge retrieval and pattern matching. Research work is different. Researchers need to formulate their own questions, construct hypotheses, design argumentation paths, and screen many failed directions for feasible ones.

    Based on the evaluation results, current mainstream LLMs are good at searching for better solutions within known paths, while their ability to open new paths has not yet shown reliable progress. This distinction explains why the same model can perform well on exercises but tend to stall on research problems.

    The Challenge of Proof Verification

    When a model claims to have produced a proof of an important conjecture, verification becomes the key step. The traditional approach is peer review. However, proofs can run to hundreds of pages with complex logical chains, and the cost of manual line-by-line checking is high.

    Formal proof assistants can machine-check proofs. But translating a natural-language proof into a formal language is itself labor-intensive and usually requires specialized training and considerable time.

    More concerning is the “hallucinated proof.” A model may write a proof that is structurally complete and confident in tone while containing subtle logical errors. Such errors are harder to detect than obvious calculation mistakes because they do not look wrong on the surface. Therefore, model-generated proofs should not be accepted directly before formal review or expert scrutiny.

    How to Use These Models in Practical Work

    For tasks with clear standard answers—such as exercise explanations, formula transformations, and mathematical calculations in code—models can noticeably improve efficiency, but results still need spot checks.

    For research-oriented tasks, models are better suited to an assistant role. They can help test whether a conjecture holds on small-scale instances, list special cases, or suggest possible ideas. The final judgment remains the responsibility of the researcher.

    When developing applications, the following practices can reduce risk:

    1. Ask the model to write out derivations step by step and mark the assumptions each step depends on.
    2. Have the model self-check key conclusions and verify them with runnable tools, such as symbolic computation or formal checking.
    3. Set up human review for proofs and citations in the output, especially for content that affects decisions.

    How can you verify a proof generated by a model? First check whether it lists all assumptions, then confirm whether each lemma has a clear source, and finally review key steps with formal tools or domain experts.

    What metrics are commonly used in TCS evaluations? Common ones include stratified accuracy, performance at different difficulty levels, error type distribution, and whether intermediate conclusions can be independently checked. Specific metrics should be based on the original report.

    Conclusion

    The evaluation’s main judgments are threefold. Large models are already quite useful for routine mathematical tasks. They have clear weaknesses in multi-step reasoning. On open problems, there has not yet been a substantive breakthrough. All three points should be understood within the evaluation conditions and should not be directly extrapolated to all scenarios.

    Three questions are worth watching next: whether models can maintain consistency in long reasoning, whether they can work reliably with formal tools, and whether they can produce genuinely new results in branches of mathematics that can be verified. Answering these questions will require more public, reproducible research.