Evaluating Frontier LLMs on Math and TCS: Research Boundaries and the Grading Problem

    This article explains why theoretical computer science (TCS) makes an effective testbed for measuring the research-level math ability of frontier large language models (LLMs). It covers the four dimensions of a high-quality benchmark (coverage, difficulty, verifiability, and anti-contamination) and the capability boundaries current evaluations tend to reveal. It then examines grading challenges and outlines practical recommendations for researchers and enterprise users, centered on a "verifiability first" principle.

    Why Use Theoretical Computer Science to Test LLMs

    If theoretical computer science (TCS) is the crown jewel of mathematics, it is not about “how to fix a computer.” It studies the fundamental laws of computation itself. A typical programmer is more like a driver who cares about the engine, the transmission, and the interior. A TCS researcher is more like a city traffic planner who asks: what is the theoretically fastest path from point A to point B? And have any routes been proven to be permanently closed?

    TCS sits at the intersection of mathematics and computer science, and every problem revolves around computation. These include algorithmic complexity, computability, approximation hardness, communication complexity, and circuit lower bounds. Its conclusions are established mainly through rigorous mathematical proofs, not experiments or engineering implementations. For that reason, it is an excellent touchstone for testing whether a large model can “really do research.”

    Why Math and TCS Are a Low-Controversy Battleground for AI Evaluation

    On math tasks, large models often earn the least disputed results, for two reasons. First, a problem usually has a single correct answer. Second, although filling in the answer can be hard, checking whether it is right is relatively clear-cut. It is a bit like Sudoku: solving the puzzle can be mind-bending, but once it is complete, its validity is easy to check at a glance.

    This gives evaluation a unique advantage. The boundaries of a correct answer are clear, scoring is objective, and results are less likely to be distorted by the subjectivity of human grading. Compared with open-ended writing or code generation, where evaluation criteria are fuzzy, math and theoretical computing offer a more reliable yardstick for model capability.

    How to Design a High-Quality Benchmark

    To truly measure research-level math ability, a rigorous benchmark needs to focus on at least four dimensions:

    1. Coverage breadth: “Full-domain” coverage means not just choosing easy questions. It means spanning multiple subfields, such as complexity theory, algorithms, cryptography, coding theory, combinatorial optimization, and game theory and mechanism design. This prevents a model from earning a high score by “farming” one type of question.
    2. Difficulty gradient: Questions should span multiple difficulty tiers, from textbook examples to frontier research problems. Only then can a benchmark tell “good at solving problems” apart from “capable of research.” The most discriminating questions are those with no ready-made solution pattern that demand multi-step reasoning or constructive proofs.
    3. Answer verifiability: Ideally, questions should have standard answers that can be verified automatically or semi-automatically. Solving a problem is somewhat like cooking from a recipe: the steps are ready-made, the result has a standard, and you rarely see chocolate added to sweet-and-sour pork. Problem setters should therefore prioritize questions whose conclusions can be clearly judged true or false.
    4. Anti-contamination and anti-leakage: A benchmark must guard against training data contamination. That is, it must check whether a question and its solution have already appeared in the model’s pretraining corpus. Otherwise, a “high score” may reflect memorization rather than reasoning.

    Where AI Actually Stands in Math and Theoretical Computing

    Evaluation results usually reveal a few clear boundaries:

    • Strong on standardized problems, few breakthroughs on open conjectures. On questions with clear steps and established patterns, models have approached or even surpassed human competitors. However, when facing open conjectures that have resisted attack for decades, they can hardly offer original, verifiable, substantive progress. The former is “following the recipe”; the latter is real research.
    • Insufficient stability in long-chain reasoning. In multi-step logical derivations, models tend to make mid-way jumps that seem reasonable but are actually wrong. These errors are often subtle and hard for the model to detect on its own.
    • Missing verifiability of proofs. A model can generate large amounts of text that looks rigorous. Whether that text can be confirmed by formal systems or peer experts, however, is another matter.

    The Hardest Part: Grading

    Grading is a core challenge that this type of evaluation cannot avoid. When a model claims to have “proved a conjecture that has remained unsolved for years,” who verifies whether it has written correctly?

    This involves two levels:

    • Fact level: Did the model actually complete the logical closure, or did it use specious arguments to get away with a superficially plausible answer?
    • Form level: Can the conclusion be checked step by step by formal verification tools, such as interactive proof assistants, turning “looks correct” into “machine-confirmable”?

    One promising direction is “verifiability first.” It means prioritizing questions and answers that can be checked by formal systems, turning subjective grading into objective verification. This also explains why formal mathematics, automated theorem proving, and related fields have been gaining momentum in recent years. They provide a machine-checkable path for putting AI mathematical reasoning into practice.

    Practical Advice for Researchers and Enterprise Users

    • Don’t rely on the total score alone. Look at a model’s score distribution across subfields, and watch out for the illusion of high scores caused by uneven performance (“being good at some subjects and weak at others”).
    • Distinguish memory from reasoning. Use anti-contamination designs and tests with new questions to determine whether a model’s capability comes from retrieval and memorization or from genuine derivation.
    • Introduce formal verification. In high-value, high-risk scenarios, such as critical algorithm proofs or cryptographic parameter justifications, prioritize workflows that can be verified automatically. Do not rely solely on a model’s own claims.
    • Build an internal evaluation loop. Teams can accumulate a set of hard problems and standard answers specific to their own domains. Combined with general-purpose infrastructure such as logging services and monitoring and alerting, they can build a continuous regression evaluation pipeline. This makes evaluation results traceable and reproducible.

    Conclusion

    Evaluating frontier LLMs on math and TCS is essentially about answering a deeper question: how far is AI from “really doing research”? Current evidence suggests that AI is already an extremely strong problem solver, yet it remains a novice researcher. Making verifiability a core design principle provides a credible yardstick for capability. It also points the way for AI to move from “solving problems” to “doing science.”