Can Large Language Models Do Real Research? A Theory of Computer Science Benchmark Gives the Answer
A joint team of leading academics and engineers from frontier large model developers has published an evaluation targeting state-of-the-art large language models, and its results have sparked debate in both academic and engineering circles. The exam contains 100 problems spanning algorithms and upper bounds, lower bounds and impossibility, reductions, and constructions — task formats whose conclusions can be checked automatically by a machine rather than by human reviewers. The headline finding is restrained: among the open conjectures that remain unresolved today, no reference model was able to solve any of them on its own. This article explains what theoretical computer science actually studies, why proofs make mathematics the strictest testing ground for AI, how the benchmark cleanly separates "solving exercises" from "doing research," and what engineering teams can take from it when designing their own evaluation pipelines.
Introduction: When AI Starts Writing Proofs
Not long ago, a joint research team made up of leading academics and engineers from frontier large model teams published an evaluation of frontier large language models, and it has stirred up considerable discussion in both academic and engineering communities.
The background is not a comfortable one. A while earlier, an AI research organization released a large batch of AI-generated mathematical manuscripts at once, grouped by topic into hundreds of “result families” spanning more than a dozen directions, including number theory, theoretical computer science, geometry, and mathematical physics. The claim that “AI has finished mathematics” spread widely on social media, with anxiety and excitement existing side by side.
The conclusion this research team reached, however, was like a bucket of cold water: faced with the open conjectures that remain unresolved today, not a single model included in the reference set was able to genuinely solve one on its own.
To understand the weight of that conclusion, two things need to be clear first: what theoretical computer science actually studies, and why “mathematics” has become the most distinctive — and least forgiving — test of gold for AI.
Theoretical Computer Science: The Study of Computation Itself
If software development were driving, programmers would care about the engine, the transmission, and the interior — which language, which framework, whether the system runs fast. Theoretical computer science (TCS) is not concerned with any single car; it is concerned with the traffic of the whole city: what is the fastest possible way to get from point A to point B? What determines that lower bound? Is there a road that, no matter how it is built, is destined to be impassable?
TCS sits at the intersection of mathematics and computer science, and everything revolves around the word “computation.” Its core output is not runnable code but proofs. It answers universal questions about the nature of computation: what is the theoretical lower bound on the cost of solving a problem, whether a certain class of structures must exist, and whether one problem can be reduced to another. Once established, these results do not depend on any specific machine, language, or implementation.
Precisely because proofs are at its core, TCS shares a natural advantage with pure mathematics — correctness can be strictly determined. That is also why it has become especially important in the era of large models.
High Benchmark Scores Still Don’t Measure Research Capability
Leaderboards for large models iterate roughly every two months, and the scores keep climbing until they are hard to take in. But one question is unavoidable: what these leaderboards mostly demonstrate is that “models can solve exercises,” not that “models can do research.”
The difference between the two is fundamental:
- Doing exercises is like cooking from a recipe: the steps are given, the grading criteria are standard, and nobody puts chocolate in a sweet-and-sour pork dish. The problems come from the training distribution, the patterns are highly reproducible, and models can score well through pattern matching alone.
- Doing research has no recipe: there are no prescribed steps, nobody can be sure in advance whether it will work out, and in many cases even the person posing the problem does not know the correct answer. Research means feeling your way along paths no one has walked, while accepting the possibility of failure.
The bigger problem is grading. If a model claims to have cracked a conjecture that no one has solved in decades, who verifies that it wrote it correctly? Those AI mathematical manuscripts are a ready example: astonishing in number, but how many of them are genuinely rigorous, reproducible results is very hard for outside observers to confirm one by one. Once the evaluation criteria themselves are unreliable, even a high score loses its meaning as a reference point.
So for an evaluation to genuinely reach “research capability,” it has to find a class of problems whose conclusions can be verified automatically by a machine while the task format is still close enough to real research. TCS happens to be a rare intersection of the two.
Why Mathematics and TCS Are Suited to Testing AI
A well-known mathematician once pointed out to a leading academic journal that AI has one fatal flaw in nearly every field: it makes errors that cannot be verified. Text, code, and experimental conclusions usually rely on people to judge right from wrong, and people get tired, miss things, and can be persuaded. Mathematics is the exception, proofs in particular: whether a proof is correct can be handed to a machine and checked step by step.
It is a bit like Sudoku. Filling in a Sudoku grid is hard, but once it is filled, a quick pass tells you whether it is right. Hard to generate, easy to verify — this asymmetry is the distinctive value of mathematics (and TCS) as an AI test of gold:
- Verification cost is controllable: the correctness of a proof can be determined through formalization or structural checking, without relying on subjective review.
- Results are unambiguous: a conjecture is either proved or it is not; there is no intermediate state of “roughly correct.”
- Hard to bluff: because verification is strict, a fabricated proof that merely “looks right” thanks to fluent language will be identified quickly.
For exactly this reason, AI’s least controversial achievements have tended to appear in mathematics; and for exactly this reason, a genuinely rigorous, machine-judgeable TCS evaluation can cleanly separate “solving exercises” from “doing research.”
How the Exam Is Designed: 100 Problems, Four Task Types
The core methodology of this evaluation can be summarized in one sentence: only ask problems a machine can grade.
Behind that is a clear trade-off. The more a problem resembles real research, the harder it usually is to grade automatically; TCS is a rare exception, since most of its conclusions can be verified by a machine, which is why “high research content” and “automatic grading” became compatible here for the first time at scale.
The exam contains 100 problems, organized by category, roughly corresponding to the kinds of work theoretical computer science researchers deal with most often. In general terms, these task formats typically include:
- Algorithms and upper bounds: designing a better algorithm under given constraints and proving its correctness and efficiency (that is, giving an achievable performance upper bound).
- Lower bounds and impossibility: proving “at least how much cost” a problem requires, or proving that a certain class of efficient algorithms cannot exist in principle (that is, showing that a particular road is doomed to be impassable).
- Reductions: transforming a new problem into a known one, thereby establishing a relative difficulty relationship between the two.
- Constructions and existence: constructing a combinatorial structure that satisfies certain conditions, or using rigorous argument to prove that a certain object must exist.
These tasks are no longer “plug into a formula and get an answer”; they require models to start from the problem itself and perform genuine reasoning and proof. Grading does not rely on an army of human reviewers but on machine verification of proof conclusions, which both makes scale possible and compresses to the maximum the room for anything that merely “looks right.”
The minimal illustration below gives a feel for what “machine-gradable” means — the evaluation cares about whether the conclusion can be verified, not how nicely the model puts things:
# Illustration: the value of a "machine-gradable" answer is that its conclusion can be checkedclaim = model.prove(conjecture) # The model returns: statement + proof stepsif verifier.check(claim): # The verifier checks the logic step by stepscore += 1 # Pass -> scoreelse:score += 0 # Fail -> no score, no matter how fluent the wording
The point of this pseudocode is not the syntax but the idea behind it: the score is determined solely by “whether the proof holds,” completely independent of language style, length, or tone. That is what allows it to cut through the “benchmark score bubble.”
Key Finding: Open Conjectures Are Still Untouched Territory
Once the evaluation ran, the conclusion was quite restrained: among the open conjectures that remain unresolved today, no reference model was able to genuinely solve any of them on its own.
Two layers of meaning in this result are worth chewing on repeatedly.
First, it pours cold water on an overheated topic. Between the “numerical prosperity” of large volumes of AI-generated mathematical text and “substantive breakthroughs” stands a threshold built out of strict verification. In the parts that can be verified automatically, models have indeed improved quickly; but the genuinely open problems that require original insight remain untouched territory.
Second, it puts a floor under rational expectations. The point of the evaluation is not to declare that AI “can’t do it,” but to draw the boundary of capability clearly: what models are good at, which kinds of tasks they are reliable on, and where they make errors that cannot be verified. For people making technology choices, this “sense of boundaries” is more valuable than any single attractive score.
Takeaways for R&D and Engineering Practice
Although this evaluation is academic work, it is equally instructive for teams doing engineering implementation. A few actionable principles can be drawn from it:
- Make evaluation close to real tasks, not exercise-solving patterns. If your acceptance criteria are only “can it reproduce the standard answer,” what you measure is most likely memorization rather than capability. When designing evaluations, introduce tasks that have no ready-made recipe and require genuine reasoning wherever possible.
- Prefer tasks that are automatically verifiable in order to scale assessment. Only when things can be graded and rechecked can evaluation be upgraded from manual spot checks into a pipeline. This is in line with the approach widely adopted in industry of “objective questions + automatic grading + regression tests.”
- Treat “verification cost” as a first-class design concern. The asymmetry between generation and verification is the foundation of evaluation credibility. Being able to establish cheap and reliable checks on outputs is often more important than pursuing more impressive generation.
- Capability evaluation needs engineered supporting infrastructure. To run evaluations at scale, across many tasks, and reproducibly, a whole set of infrastructure usually has to work together: a container platform to isolate and reproduce runtime environments, object storage to accumulate datasets and results, a message queue to schedule massive numbers of evaluation tasks, and monitoring, alerting, and logging services to track the status and regressions of every run. On cloud platforms such as Baidu AI Cloud, most of these capabilities are offered as general-purpose product categories and can be used to build a stable, observable evaluation pipeline.
In the end, what this TCS evaluation provides is not a simple answer to “can AI replace researchers,” but a mirror that reflects the boundary of capability: where things can be verified, models have made astonishing progress; where original insight is needed for open problems, people still have to open the way. For every developer and technical decision-maker, rather than chasing ever-refreshing benchmark scores, the better question to settle first is this: what exactly do we want — “solving exercises,” or “doing research”?
