Theoretical Computer Science: A Verifiable Benchmark for Evaluating LLM Research Capability
Large language model benchmark scores keep rising, but a strong benchmark result is not the same as research capability. This article explains the difference between answering known problems and conducting original research, why grading is the hardest part of evaluating large language models, and why mathematics—especially theoretical computer science—provides a machine-verifiable testbed. It also outlines a tiered benchmark design, from basic concepts to open conjectures, and discusses what verifiable results mean for enterprise deployment.
1. Higher Benchmark Scores Do Not Mean Higher Research Ability
Large language model evaluation scores are refreshed almost every few weeks, and leaderboards can be overwhelming. But an important distinction must be kept clear: a benchmark measures whether a model can answer a given question. That is not the same as whether it can do research.
Answering a question is closer to cooking a dish from a recipe. The steps are already defined, the expected outcome is known, and the result is usually predictable. Research is different. There is no fixed recipe. Whether a method will work is uncertain, and for some hard problems even the person who poses the question may not know the answer.
There is also an easy-to-miss trap: data contamination. If the standard solution to a problem is already widely present in training data, a model may answer correctly not because it can reason, but because it is reproducing memorized content. When the problem itself exists in the model’s “memory,” a high score does not prove that the model can explore a new problem from scratch.
2. Grading Is the Hardest Part of Evaluation
Suppose a model can generate hundreds of pages of “proof” in one pass. A new problem immediately appears: who has the time to check it page by page? If it claims to have solved a conjecture that has remained open for decades, who can quickly confirm whether the work is correct?
This leads to two practical difficulties. First, manual review can become prohibitively expensive. Second, hallucinated errors—results that look rigorous but contain a subtle invalid step—are very hard to detect.
Mathematics, and proof in particular, is special because verification is often easier than solving. It is like Sudoku: filling the grid is difficult, but checking whether the filled grid is correct can be done at a glance.
Formal proofs can be checked automatically by machines. Given a sequence of derivation steps, a verifier only needs to confirm whether each step is valid and whether the correct premises are used. It does not need to understand the entire solution in advance. For this reason, mathematics and theoretical computer science—closely related to mathematics—become some of the least controversial domains for evaluating large language models, because correctness can be decided by a machine rather than by subjective human judgment.
3. Why Choose Theoretical Computer Science
Theoretical Computer Science (TCS) studies computation itself. It does not focus on repairing computers or writing application code. Instead, it asks more fundamental questions: Can a problem be computed at all? If it can, what is the fastest possible way to compute it? Are there paths that are impossible in principle?
A useful analogy is this: programmers are like drivers, focusing on engines and interiors, while theoretical computer science studies the entire transportation system. It asks what the theoretical shortest travel time is between two points and which routes are fundamentally impossible. TCS sits at the intersection of mathematics and computer science. Its questions are centered on computation, and its results are usually expressed as theorems and proofs.
The main reason to choose TCS as an evaluation ground is simple: most of its conclusions can be verified by machines. This directly avoids the critical weakness of “grading difficulty.”
4. How Should Such a Benchmark Be Designed
A benchmark truly aimed at research capability should resemble real research settings while still allowing automatic grading. A reasonable design divides problems into several levels, from easier to harder, roughly reflecting the daily work of researchers:
- Concept and definition problems: test mastery of basic concepts, theorems, and definitions. These cover fundamental skills.
- Algorithm and complexity reasoning problems: problems with clear answers that test whether the model’s reasoning chain is coherent and rigorous.
- Paper-level proof reconstruction problems: give the model known but difficult results and observe whether it can independently reconstruct their proofs.
- Open conjecture problems: select problems that remain unsolved and have no standard answer, in order to probe the upper limit of the model’s ability.
Even if the first three categories are answered correctly, the model may simply be reproducing known conclusions from training data. Only the fourth category—truly open problems—can distinguish “reorganizing known knowledge” from “creating unknown knowledge.”
The grading process can be designed as a closed loop of generation and verification:
def evaluate(model, problem, verifier):# 1. Generate a candidate proof based on the problem descriptioncandidate = model.solve(problem)# 2. Machine verification: check whether each derivation step is validis_valid, reason = verifier.check(candidate)# 3. Accept only results that pass formal verification; reject unverifiable “beautiful answers”return is_valid, reason
The key point is that the model’s self-claimed correctness is not accepted. Only the verifier’s decision counts. An answer that cannot be verified is not scored, no matter how complete it appears.
5. Conclusion: A Reality Check on the Idea That AI Is Dominating Mathematics
The benchmark results offer a sobering conclusion: when facing truly open conjectures, none of the evaluated models could solve them solely through its own ability.
Several judgments can be drawn from this:
- Current frontier models are good at recombining and reproducing known structures, but they are not yet capable of original innovation that creates something from nothing.
- The focus of evaluation should shift from “how many questions can be answered correctly” to “whether verifiable new results can be produced.”
- Verifiable domains are the best testbed. Using unverifiable open problems as a measuring stick can better approach and test real research ability.
For enterprises and developers, this also points to a practical direction: deploy large language model capabilities first in scenarios where results can be automatically verified, such as code generation with formal verification, mathematical reasoning, and logical constraint checking. At the same time, general-purpose evaluation services, logging services, and monitoring and alerting capabilities can help turn the “generation–verification” process into a reproducible and sustainable evaluation pipeline. Baidu AI Cloud’s container platform, object storage, logging, and monitoring and alerting capabilities can support such large-scale, measurable evaluation tasks and make the question “is the model actually capable?” move from subjective impression to objective evidence.
