Can Large Language Models Do Research? Answers from a Theoretical Computer Science Evaluation
This article examines a research-oriented evaluation for large language models based on theoretical computer science. It explains why benchmark scores do not equal research ability, how verifiable problems and rigorous grading can make AI evaluation more reliable, and what the results mean for researchers and engineering teams.
With large language models repeatedly scoring highly on math-contest-style tests, the claim that “AI is cracking mathematics” has become a popular topic. A senior theoretical computer science researcher, working with a large-model startup team and collaborators from universities and research institutions, placed a set of frontier models into a “research-level exam.” The result was striking: none of the evaluated models could independently solve the still-open problems included in the exam.
This article uses that evaluation as a starting point to examine the gap between “solving problems” and “doing research.” It also explains how the evaluators designed the test to be research-like while remaining verifiable, and why line-by-line grading can make the results more credible.
1. Starting from an Exam Question: How Far Is AI from Doing Research?
The claim that “AI has made mathematics obsolete” has circulated widely online. The debate itself is not new. Every so often, a model breaks a record on some exam, competition, or standardized test, and the same question follows: are being good at problem solving and being good at research the same thing?
A prominent mathematician once told an international academic journal that AI has the same fundamental weakness across almost every field: it can make errors that cannot be verified. Mathematics, especially proof, is the exception. Once a proof is written, machines can automatically check whether it is correct.
This is similar to Sudoku: filling an entire board is difficult, but checking whether a completed board is valid is easy. For this reason, AI companies’ least controversial results often appear in mathematics. Logically consistent and machine-checkable proofs give claims such as “AI solved a mathematical problem” a stronger basis for examination.
But the same property also makes mathematics a useful test of whether AI can do research. When models that perform strongly across many domains are placed into an exam closer to real mathematical and theoretical computer science research, the result is far more sobering.
This evaluation responds to the debate with a simple experiment. Instead of asking whether models can answer more questions correctly, it places them in an exam based on research scenarios. The result is somewhat cold water: for the open problems in the exam that remain unresolved, no evaluated model could solve them independently.
This should be understood as the result of one evaluation, not a final judgment on the upper limit of AI research ability. Its more valuable contribution lies in methodology: how to design an exam that is more like research and can still be graded.
2. Why Benchmark Scores Cannot Explain Research Ability
2.1 Problem Solving, Memorization, and Research: Key Differences
The value of an evaluation depends on whether it answers the questions we truly care about. Benchmark scores for large language models are refreshed every couple of months, and leaderboards appear constantly. However, most leaderboards measure “problem-solving ability,” not “research ability.”
This is not just a matter of wording. The two correspond to several fundamentally different modes of capability:
- Practice: During training, the model repeatedly encounters similar problems and standard solutions, forming pattern-recognition ability. When it sees a problem type from training, it can quickly match it to the closest solution.
- Memorization: Through pretraining, the model may have encountered the original problem text or highly similar variants. A high score on a closed problem set may only show that memory was successfully activated, not that ability was demonstrated.
- Research: There is no ready-made procedure to follow. One must try ideas in unsolved problem spaces, identify why approaches fail, and produce new conclusions that can withstand peer review.
A rough analogy is this: problem solving is like cooking from a recipe. The steps are ready, the heat level is known, and few people would add chocolate to a stir-fried pork dish. Research is like developing a new dish. There is no recipe, no clear list of ingredients, and no known number of attempts needed to find the final formula.
More importantly, even the person proposing the problem may not know the answer. The defining feature of open problems is that no one knows in advance what the correct solution looks like, so no one can provide “standard steps.”
2.2 Three Bottlenecks of Traditional Evaluation: Narrow Format, Limited Difficulty, and High Grading Cost
If research ability is the real target, traditional evaluation frameworks face at least three bottlenecks:
Narrow assessment format: Standardized tests mostly rely on multiple-choice or short-answer questions. Multiple-choice questions can be guessed through elimination or linguistic clues among options. Short answers often evaluate only the final result, not the reasoning process. A model may answer correctly, but it is unclear whether it completed the derivation or happened to match a similar expression in its training data.
Limited difficulty ceiling: To ensure question quality and discrimination, questions must have determinate answers and clear scoring rules. This objectively excludes open problems whose answers are unknown in advance, even though these are among the most typical forms of real research.
High grading cost: Research outputs often appear as long proofs, ranging from a few pages to more than a hundred. Checking line by line depends heavily on expertise and is extremely time-consuming. This is the root of the “grading difficulty”: a model can generate hundreds of pages of “proof” in one pass, but very few people have the time and ability to check it page by page.
When a model claims to have solved a conjecture left open for decades, how can we judge whether it is actually correct? Some research teams have publicly released AI-assisted mathematical manuscripts containing repetition of existing results, typographical errors, and even mistakes. The mathematics community’s first reaction was not awe, but “who will check it?”
This example shows that generation ability can improve rapidly, while verification ability must be built at the pace of research work. The gap between the two is precisely what this kind of evaluation tries to measure.
3. TCS: A Field for Studying Computation Itself and a Natural Fit for Machine Verification
3.1 What TCS Studies
To understand why theoretical computer science was chosen as the exam domain, we first need to understand its subject matter. TCS, or Theoretical Computer Science, is not concerned with how to assemble a computer, nor does it directly discuss how to write code for a business system. It asks about the boundaries and costs of computation itself.
- Engineering perspective: A program is like a driver, concerned with engine performance and interior configuration — how to use existing software and hardware well, and how to deliver business requirements quickly.
- TCS perspective: It is more like studying transportation itself — how long it takes to travel from A to B, whether some routes are fundamentally impassable, and, even if a route exists, what the cost is.
TCS sits at the intersection of mathematics and computer science. Its questions all revolve around computation: can a problem in principle be solved by an algorithm? If so, how much time and space are required? If not, where is the obstacle?
The main output of TCS research is proofs: proving that an algorithm is correct and efficient, or proving that a problem has an unavoidable lower bound.
3.2 Why TCS Is Suitable for Evaluating Research Ability
TCS is suitable for testing model research ability for four reasons:
First, its problems are directly tied to computation. The objects of TCS research correspond naturally to computational processes. Topics can be precisely stated as propositions and are easy to formalize.
Second, its outputs take the form of proofs. Mathematical proofs consist of a series of reasoning steps governed by strict rules. As long as the logical chain is complete, machines can check it line by line. If any step fails, the conclusion fails too. Therefore, most TCS results can be automatically verified without relying on subjective judgment.
Third, its conclusions are clear. A theorem either holds or does not hold. An algorithm either satisfies its claimed complexity upper bound or it does not. This binary nature makes scoring independent of a grader’s subjective preference.
Fourth, its work types are diverse. From constructive algorithms to impossibility results, from complexity lower bounds to critical review of existing conclusions, theoretical computer science covers a wide range of research activities, enough to support an exam that covers multiple dimensions of ability.
For this reason, in the narrative of “AI conquering mathematics,” TCS-related progress is often among the easiest to independently verify: its output is not a passage of text that requires expert taste to judge, but a logical chain that machines can check step by step.
4. A 100-Problem Exam: Four Problem Types for Four Kinds of Research Work
If the goal is to measure whether models have research ability, testing only whether they can solve problems is not enough. A more reasonable approach is to break researchers’ real work into typical tasks and design corresponding problem types for each.
The evaluation contains 100 problems divided into four categories, corresponding to four types of work most commonly performed by theoretical computer scientists. The goals, answer formats, verification methods, and difficulty characteristics of each type are summarized below:
| Problem type | Corresponding research work | Answer format | Typical verification method | Difficulty characteristic |
|---|---|---|---|---|
| Constructive problem solving | Proposing a new algorithm or new proof | Provide construction and complete proof | Proof checker verifies line by line | The solution must be “invented”; there is no template |
| Impossibility and lower-bound proofs | Proving that a problem cannot be solved or is expensive | Provide a contradiction or lower-bound proof | Proof checker plus counterexample testing | Requires discovering invariants or reduction ideas |
| Algorithm design and complexity analysis | Giving an algorithm and arguing its efficiency | Algorithm description + correctness proof + complexity analysis | Complexity upper-bound verification + implementation testing | Must balance “correct” and “efficient” |
| Research judgment and critical review | Reviewing, analyzing, and selecting | Identify errors, provide counterexamples, or explain tradeoffs | Compare with known results + locate annotated points | Tests the ability to distinguish correct from incorrect |
Each category is explained below.
4.1 Constructive Problem Solving: From Solving to Constructing
The first type requires models to solve a previously unsolved problem and provide a complete proof. The key difference from contest problems is that contest problems usually have clear entry points and mature techniques, while this type requires models to invent a solution path themselves.
In TCS, such a construction might be a new data structure, a new algorithm, or a previously undiscovered proof technique.
This category best reflects the essence of research: no one tells the model which method to use. The model must decide where to begin and which approach is viable.
4.2 Impossibility and Lower-Bound Proofs: Proving Why Something Cannot Be Done
The second type requires models to prove that a target cannot be achieved, or to provide a hard lower bound on resource consumption. Such results are extremely common in TCS because “cannot be done” is as important as “can be done.”
- Engineers cannot expect to quickly find an optimal solution in every situation, because some problem lower bounds show that any algorithm must pay a substantial cost in the worst case.
- Cryptographic security also depends on assumptions that certain problems are hard to solve.
These problems usually have no direct construction to follow. Models need to discover appropriate invariants, design reductions, or construct adversarial inputs.
This is the dividing line between research and practice drills: practice trains models to “find an answer,” while these problems require models to “prove that no answer exists.”
4.3 Algorithm Design and Complexity Analysis: Both Correct and Efficient
The third type requires models to provide an algorithm and simultaneously prove its correctness and efficiency. These tasks directly correspond to the dual engineering demands of “can it be done?” and “is it fast enough?”
- An algorithm that is correct but has unacceptable complexity is hard to deploy.
- An algorithm that is fast but lacks correctness guarantees produces untrustworthy results.
Therefore, this type tests both design ability and proof ability. Models must not only find an algorithm, but also prove its correctness and analyze its time and space complexity.
4.4 Research Judgment and Critical Review: Understanding Correctness Requires Discrimination
The last type is often overlooked, yet it best reflects researchers’ daily work. In real research, judgment ability is often more important than generation ability:
- Reading a paper and judging whether its conclusions are reliable.
- Choosing among multiple candidate directions and judging which is more likely to succeed.
- Identifying hidden errors in an argument that appears rigorous.
This type requires models to provide the basis for their judgment: identify the problem, provide a counterexample, or explain the rationale for a choice. It tests a model’s ability to distinguish right from wrong, rather than its ability to rearrange existing knowledge.
5. How to Grade: From Checking Answers to Line-by-Line Verification
Designing the question bank is only the first step. The grading method determines whether the evaluation results are credible. This is one of the most easily overlooked but most critical aspects of the TCS evaluation.
5.1 Three-Layer Verification
In TCS, most conclusions can be verified mechanically. Therefore, the evaluation does not rely on experts reading model outputs page by page. Instead, it uses layered verification:
- Layer 1: Line-by-line proof checking. Model proofs are processed by formal verification tools. The tools check each step according to strict inference rules. If even one reasoning step fails, the entire proof is judged invalid. This is like giving every submission a tireless reviewer that does not miss details.
- Layer 2: Supplemented verification for specific problem types. For algorithm problems, tools further verify whether the complexity upper bound is rigorously argued in the proof and test correctness through independent implementations. For judgment problems, model conclusions are compared with known facts, and counterexamples are checked separately.
- Layer 3: Human and expert review. When machines cannot fully judge a case, researchers step in. This preserves efficiency while avoiding false positives or false negatives caused by machine misjudgment.
5.2 From “Right or Wrong” to Tiered Scoring
If the evaluation only counts how many questions a model answered correctly, a lot of information is lost. Research ability is continuous. A model may have the right idea but a flawed proof, or may reach the right conclusion with incomplete reasoning.
Therefore, such evaluations often use tiered scoring:
- Correct conclusion: The final answer matches expectations, receiving the highest base score.
- Valid proof: The reasoning chain is complete and passes verification, showing that the conclusion holds.
- Reasonable approach: Even if the model does not finish, if its key idea aligns with the standard solution, it can receive partial credit.
This tiered scoring provides a more nuanced picture of model ability and avoids the crude judgment of “one error means everything is wrong.”
5.3 Anti-Contamination and Question Governance: Keeping the Exam “Fresh”
The credibility of an evaluation depends heavily on whether models have already seen the questions. Common governance measures include:
- Time slicing: Release questions after the model’s training-data cutoff to reduce memorization leakage.
- Question variants: Apply equivalent transformations to questions to prevent models from answering through pattern matching.
- Answer isolation: Manage questions and reference answers separately, and keep reference answers out of public channels.
- Periodic updates: Continuously add new questions as research progresses, keeping the question bank fresh.
In essence, this mechanism does two things: it ensures that questions are new to models, and it ensures that grading standards remain stable.
6. An Example of an Engineering Evaluation Pipeline
The following illustrative code shows how to turn the above evaluation ideas into a reusable pipeline. The code focuses on the process skeleton: each stage has clear responsibilities and can be replaced easily. It does not depend on any specific vendor product.
# Illustrative code: TCS research-capability evaluation pipeline# Note: verify_submission is an abstract interface for an external proof checker or verifier.# In actual deployment, it is implemented by formal tools or test frameworks and kept decoupled from this pipeline.from dataclasses import dataclass, fieldfrom typing import Any, Callableimport json@dataclassclass Task:"""A single evaluation task."""id: strcategory: str # construction / lower bound / algorithm / judgmentstatement: str # task statementreleased_at: str # release date, used by anti-contamination gatingdifficulty: str = "open" # solved / open, indicates whether a known solution existsreference: Any = None # reference answer or information needed for verification@dataclassclass Result:"""Result of a single task."""task_id: strcategory: strsubmitted: Anylevel: str = "none" # none / partial / valid / solvednotes: list = field(default_factory=list)def judge(task: Task, submitted: Any) -> Result:"""Grade a single task using tiered levels: conclusion, proof, and approach."""result = Result(task.id, task.category, submitted)# 1) Coarse filtering: empty or clearly irrelevant submissions are invalid.if not submitted or not str(submitted).strip():result.notes.append("empty submission")return result# 2) Core verification: external verifier checks the submission line by line.ok, detail = verify_submission(task, submitted)if ok == "solved":result.level = "solved"elif ok == "valid":# The conclusion and proof hold, but the expected construction or algorithm is missing.result.level = "valid"elif ok == "partial":# The key idea matches the standard solution, but there are gaps.result.level = "partial"result.notes.append(detail)else:result.level = "none"result.notes.append(detail)return resultdef verify_submission(task: Task, submitted: Any):"""Abstract verification interface returning (status, detail).Implementations should branch by task type:- Construction and lower-bound tasks: use a proof checker for line-by-line verification;- Algorithm tasks: verify the complexity upper-bound proof and run implementation-level correctness tests;- Judgment tasks: compare against known results and separately test counterexamples provided by the model."""raise NotImplementedError("Replace this with an actual verifier implementation")def should_include(task: Task, cutoff: str) -> bool:"""Anti-contamination gate: keep only tasks released after the training-data cutoff."""return task.released_at > cutoffdef evaluate(tasks: list, model_answer: Callable, cutoff: str, out_path: str):"""Main flow: gating, answering, judging, and writing the report."""report = {"total": 0, "included": 0, "levels": {}, "by_category": {}, "excluded": []}for task in tasks:if not should_include(task, cutoff):report["excluded"].append(task.id)continuereport["total"] += 1submitted = model_answer(task.statement)result = judge(task, submitted)report["included"] += 1report["levels"][result.level] = report["levels"].get(result.level, 0) + 1cat = report["by_category"].setdefault(task.category, {"total": 0, "solved": 0, "valid": 0, "partial": 0})cat["total"] += 1if result.level in cat:cat[result.level] += 1with open(out_path, "w", encoding="utf-8") as f:json.dump(report, f, ensure_ascii=False, indent=2)return report
In actual deployment, verifiers often run as independent services. The evaluation pipeline submits answers through interfaces and receives verification results asynchronously. Questions and submissions can be stored in object storage, tasks can be buffered through message queues, and verification services can be deployed on container platforms with on-demand scaling. Logs can be sent to logging services, while metrics such as success rate and latency can be connected to monitoring and alerting systems for troubleshooting and capacity planning.
These capabilities correspond to general cloud product categories. For example, Baidu AI Cloud offers products in categories such as object storage, message queues, container platforms, logging services, and monitoring/alerting that can cover these scenarios. They are decoupled from the evaluation logic itself, so teams can choose or replace them as needed.
7. How to Read This Evaluation: Three Clarifications
7.1 “Not Solving Open Conjectures” Does Not Mean “No Progress”
Open conjectures remain open because they have survived decades of research without being solved. Steady improvements on regular problems do not contradict the fact that models cannot independently crack open conjectures.
The value of this evaluation is to clarify capability boundaries, not simply to label AI as “strong” or “weak.”
7.2 Verifiable Does Not Mean Fully Automatic
Verifiability means that answers can be checked by machines, but it does not mean that grading can be fully automated. There are three reasons:
- Proof checkers can determine whether a proof is valid, but not whether it is valuable.
- A new proof may be correct but long and clumsy, or short but revealing deeper connections.
- Determining whether a construction is “new” still requires comparison with existing literature.
Therefore, automatic grading solves the reliability problem, but assessing research value still requires human researchers.
7.3 Exams Are Static, but Research Is Dynamic
One more point is easy to overlook: an exam is a snapshot, while research is a continuously evolving process. Today’s difficult problem may become routine after a new paper appears. Today’s seemingly impossible direction may become solvable because of a new technique.
Thus, the value of this kind of evaluation is not a one-time ranking, but building a continuously updatable evaluation mechanism. As model capabilities improve, question difficulty should also be adjusted to maintain discrimination.
8. Implications for Researchers and Engineering Teams
The significance of this evaluation is not only “scoring models.” For teams working on evaluation, model development, and engineering implementation, it offers at least three useful ideas.
8.1 Evaluation Design Should Reflect the Real Work Distribution
If the goal is to evaluate “research ability,” standardized questions alone are insufficient. A better approach is to first map the real workflow of the target role, then break it into typical tasks, and finally design verifiable questions for each task type.
This aligns with task-driven testing in software engineering: the effectiveness of a test depends on whether it covers real usage scenarios, not on the number of questions.
8.2 Treat Verifiability as a Product Design Principle
The most important lesson is this: if a task’s output can be automatically verified by machines, then progress on that task can be measured clearly.
For engineering practice, this suggests:
- Formalize what can be formalized, such as correctness and security properties of key protocols and core algorithms.
- Automate what can be tested, such as property-based tests and regression tests.
- Make measurable metrics explicit, such as latency percentiles, error rates, and resource utilization.
The stronger the verifiability, the faster a team can judge whether a change is correct, and the more easily AI-assisted efficiency gains can become durable.
8.3 Separate Generation from Verification
Generation and verification are different kinds of ability. Models are good at quickly generating many candidate solutions, but they are not good at judging whether those solutions are correct. A reasonable workflow should separate the two:
- Generation side: Models quickly produce multiple candidate proofs, algorithms, or patches.
- Verification side: Automated tools and experts jointly review candidates, and only passing candidates move forward.
This “generation-verification” separation leverages the model’s generation efficiency while maintaining quality. It applies to many scenarios, including code review, solution design, and technical document review.
9. Conclusion: How Far Is AI from Doing Research?
Back to the original question: can AI do research?
This evaluation gives a cautious and clear answer. On regular problems and tasks with clear verification paths, models have demonstrated capabilities approaching those of researchers. But on truly open problems where no one knows the answer, they still need guidance from human researchers.
This suggests that AI is not meant to replace researchers. Instead, it can free researchers from repetitive verification work so that humans can focus on what machines are still not good at: asking good questions.
For engineering teams, the most valuable lesson may not be “how strong the model is,” but “how to make evaluation credible.” As more tasks become verifiable, our way of measuring AI capability will gradually shift from benchmark scores to verifiable outputs.
