LLM Research Capability Evaluation: A New Answer from TCS

    A new 100-question benchmark in theoretical computer science tests whether frontier large language models can handle research-level problems. None of the reference models solved the benchmark's open conjectures. Unlike conventional leaderboards, the evaluation is designed to be machine-gradeable, using problems constructed from recent literature and checked through formal proofs, proof checkers, solvers, or executable tests. The article explains why TCS is a suitable setting for verifiable research evaluation, how to interpret model scores, what the results imply for AI and mathematics, and how developers can apply the same principles to build reliable LLM evaluation pipelines.

    A new 100-question benchmark for theoretical computer science (TCS) research has appeared, and none of the reference models solved its open conjectures. This comes as the question “Can AI replace mathematicians?” has repeatedly become a hot topic, with hundreds of AI-generated mathematical manuscripts made public across more than a dozen fields. Amid that debate, a group of researchers released a benchmark built from TCS questions to test then-frontier large language models. The result was a cold shower for overheated expectations.

    The paper is already public on a preprint platform. It has five authors from universities, research institutions, and a large-model company. One is Turing Award winner Andrew Yao, dean of the Institute for Interdisciplinary Information Sciences at Tsinghua University. Another is Yang Zhilin, technical lead behind a leading AI assistant.

    Why does this paper matter? In short, it turns the vague question “Can AI do research?” into an engineering problem: whether a result can be graded by a machine.

    1. What TCS studies: not coding, but computation itself

    TCS stands for Theoretical Computer Science. It does not focus on installing systems, debugging frameworks, or making APIs work. Instead, it asks a single question: for a class of problems, what can computation itself achieve, and what can it not achieve?

    An analogy helps:

    • An ordinary programmer is like a driver, concerned with engines, transmissions, and interiors.
    • An algorithm engineer is like a race-car tuner, concerned with how fast a car can complete a lap.
    • TCS studies traffic itself: the theoretical minimum travel time between two points, the lower bound on the cost required to prove a target result, and which routes are mathematically impossible to connect.

    TCS overlaps deeply with mathematics, and its objects of study are almost entirely about computation: algorithms, computational complexity, computability, foundations of cryptography, coding and pseudorandomness, interactive proofs, and more. Its main tool is proof, not experimentation or engineering implementation.

    Dimension Conventional engineering practice TCS research
    Core question Can it be implemented and shipped? Can it be solved efficiently under a given model, or are there unavoidable lower bounds?
    Main output A runnable system Theorems, lemmas, constructions, and proofs
    Correctness check Tests pass; online metrics meet targets Logically rigorous and verifiable
    Lifetime of conclusion Changes with versions and iterations Once rigorously proved, it holds over the long term

    Because TCS results exist in the form of rigorous proofs, the field has a rare property in engineering: correctness can be checked by machines.

    2. Why higher leaderboard scores still do not prove research capability

    Over the past two years, LLM benchmarks have been refreshed almost every two months. But benchmarks test problem-solving, while research is exploration. The two are separated by a wide gap.

    1. Answers are already known. Conventional problems have standard answers, and they may already appear in training data together with explanations. Scores therefore reflect memorization and pattern matching more than original research.
    2. The process is constrained. Solving exam problems is like cooking from a recipe: steps are given and there is a standard taste. Research has no fixed steps, and even the problem itself must first be framed—what exactly should be proved? For some questions, even the person who set them does not know the answer.
    3. Grading is often impractical. If a model claims to solve a conjecture that has been open for decades, it is difficult for outsiders to automatically verify the proof in most fields.

    To measure research capability, grading must meet three conditions:

    • The conclusion can be checked by a machine.
    • The problem is newly constructed after the training-data cutoff, or is still unsolved by anyone.
    • The solution requires real reasoning and construction, not retrieval and reuse.

    TCS happens to satisfy these conditions. Most results can be formalized as propositions and then checked step by step using formal proof systems or specialized checkers.

    The paper therefore uses only questions that can be graded by a machine. The full set contains 100 questions in four categories, corresponding to four common activities in theoretical computer science research:

    1. Prove lower bounds or impossibility results, answering “why can’t it be done?”
    2. Provide verifiable constructive results, answering “how can it be done?”
    3. Rigorously disprove an existing conjecture or provide a counterexample.
    4. Directly attack an open problem.

    All questions were newly constructed from recent literature. Progress on open problems will continue to be made public, making it easier to track which model solves them first.

    In the author’s view, the clever part of this design is that it turns subjective evaluation into objective verification:

    • Grading no longer depends on human review or comparison with reference answers. It relies on proof checkers, solvers, and test scripts.
    • The grading process can be fully reproduced.
    • The evaluation standard does not drift with model versions.

    From an engineering perspective, this is essentially a research benchmark that comes with its own unit tests.

    3. Four reminders when reading the scores

    This type of score should be distinguished from ordinary benchmarks. It is closer to a “paper-level” difficulty scale rather than a pass rate on a 100-point scale. Some discussions have noted that participating models score highly on ordinary TCS problems but drop noticeably on this benchmark.

    When reading specific numbers, it is worth paying attention to:

    • Sampling count. Questions are scored with pass@1, meaning one generation without relying on resampling. In research, one attempt may be only an idea; the same model may produce noticeably different results with more samples.
    • Prompts and tools. Whether models can call symbolic computation tools or proof checkers, and how prompts are designed, can significantly affect scores.
    • Problem scope. Difficult questions may be replaced as progress is made public, so scores from different times and versions should not be compared directly.
    • Single-point results. A score is a snapshot at one moment. The conclusion matters more than the number.

    4. What it means that no model solved the open conjectures

    The paper gives mathematicians some breathing room. Its core conclusion is simple: among the reference models, none could independently solve the open conjectures in the benchmark.

    This does not mean “AI cannot do mathematical research.” A more accurate interpretation has four points:

    1. Autonomously proposing and solving new problems remains the hardest part of research. Models can perform long-chain reasoning given a goal, but deciding which problem is worth doing and which route is feasible among countless possibilities still relies mainly on humans.
    2. There is a gap between solving problems and proposing them. Exam questions are given by others; the first threshold in real research is finding the right formulation of the problem.
    3. The verification advantage is real. Where results can be machine-checked, correctness is determined by proof checkers and solvers rather than human impression, so claims such as “AI has solved problem X” can be rigorously tested.
    4. Interactive proofs are lowering the cost of rigor. Increasingly mature workflows combining formal proofs with expert review make model outputs that are difficult to distinguish between “rigorous” and “plausible” easier to check.

    When OpenAI’s o1 scored 92.4 on the 2024 American Invitational Mathematics Examination (AIME), Terence Tao raised a similar concern: once competition-level problems such as those in Mathematical Olympiad settings are solved, benchmarks can saturate quickly. For AI mathematics, the real difficulty has never been merely “producing an answer”; it is producing a correct conclusion that can be rigorously proved.

    Over the past year, several models have made progress on International Mathematical Olympiad (IMO)-level problems, with related teams claiming gold-medal-level performance. This supports his view: competition-style questions are saturating quickly. The TCS research benchmark points to a clearer direction: the next stage of comparison should focus on problems that can be rigorously verified and whose answers are not known in advance.

    5. Lessons for developers: building a trustworthy LLM evaluation system

    The methodology of the TCS benchmark also applies to enterprise evaluation of AI systems. Whether the use case is a coding assistant, knowledge-base question answering, or mathematics and scientific computing, the key is to make judging criteria as executable and reproducible as possible.

    1. Prefer verifiability. Prioritize tasks with automatic judges, such as code that passes unit tests, SQL that returns expected results, structured output that passes schema validation, or calculations that can be checked by identities.
    2. Prevent data contamination. Build evaluation sets from private data, real business tickets, and newly written questions. Keep at least one set private.
    3. Layer capabilities. Separate memorization and format compliance, standard-scenario reasoning, and open-ended research. Avoid using a single score to hide weaknesses.
    4. Sample multiple times. For hard tasks, report both pass@1 and pass@k, and observe score variance. A single success may be luck.
    5. Analyze failures. Record reasoning traces, failure types (calculation errors, misuse of premises, out-of-scope assumptions), and token consumption, instead of looking only at one score.

    In practice, a trustworthy evaluation usually requires:

    • Containerized isolated environments for safely executing model-generated code or formal checks.
    • Object storage for saving questions, model outputs, and historical results.
    • Message queues for decoupling “generation” and “grading” to improve throughput.
    • Logging and monitoring with alerts for tracking score trends and abnormal fluctuations.

    These capabilities can be combined on demand using cloud services, such as those offered by Baidu AI Cloud, subject to regional and service availability. After each model version or inference-parameter change, run a regression evaluation and archive the result. This turns the evaluation system from a one-time score run into a continuous quality gate.

    A simplified grading scheduler can be written as follows:

    1. # Illustrative code: run grading in an executable way and keep evidence for audit
    2. def run_eval(model, dataset, grader, sandbox, storage):
    3. report = []
    4. for item in dataset:
    5. # Generate multiple samples per item and report both pass@1 and pass@k
    6. samples = [model.generate(item.prompt) for _ in range(item.k)]
    7. results = []
    8. for idx, ans in enumerate(samples):
    9. # Run the automatic grader in an isolated environment to avoid model output affecting the grading environment
    10. ok, detail = sandbox.run(grader, item, ans)
    11. # Save the question, output, grading details, and result for review and failure analysis
    12. storage.save(item.id, idx, ans, ok, detail)
    13. results.append(ok)
    14. report.append({
    15. "id": item.id,
    16. "pass@1": sum(results) / len(results),
    17. "pass@k": int(any(results)),
    18. })
    19. return report

    6. Conclusion: redefining “AI can do research” through verifiability

    The value of the TCS benchmark is not just a scoreboard. With a rigorous experimental design, it shows that evaluating the research capability of large models must be built on machine-gradeable foundations. It also clearly marks the current capability boundary: models are strong at reasoning under a given goal, but remain far from independently proposing and solving open problems.

    For developers and enterprise users, the most useful lesson is this: instead of chasing leaderboard rankings, build an evaluation system for business scenarios that is verifiable, reproducible, and continuously runnable. Let machines grade answers, and let humans judge whether a problem is worth solving. This may be the most realistic division of labor for human-machine collaboration in doing research.