Evaluating LLM Research Capabilities: From Benchmark Problems to Real Research
A recent public evaluation used theoretical computer science as a test set for frontier large language models and found that the models still cannot independently produce complete, verifiable solutions to open conjectures. This article explains why leaderboard scores are not the same as research ability, why TCS is a useful proving ground, how to design layered and verifiable evaluations, and what developers and enterprises should consider when assessing models for research assistance.
Evaluating LLM Research Capabilities: From Benchmark Problems to Real Research
A recent public evaluation study used theoretical computer science (TCS) as an exam paper to assess a set of frontier large language models. The result was unsurprising but valuable: on open conjectures that remain unresolved, the reference models cannot yet independently provide complete, verifiable solutions. This conclusion should not be read simply as “the models fail.” It should be treated as a necessary calibration—a reminder that rising leaderboard scores and improving research capability are not the same thing.
1. Behind Leaderboard Inflation: Solving Problems Is Not Doing Research
Large-model benchmark scores are refreshed almost every few months, and new leaderboards keep appearing. But most leaderboards answer a relatively narrow question: can a model produce output close to a standard answer on an existing problem? They are good at measuring pattern matching, knowledge recall, and short-chain reasoning, but they are poor at measuring genuine research capability.
The biggest difference between problem solving and research is whether the goal is predefined. Problem solving is more like cooking from a mature recipe: the steps are largely given, the taste has a standard, and deviations are easy to spot. Research is more like designing a new dish in an unknown cuisine: there are no fixed steps, and a standard answer may not exist. For many open problems, even the person who posed the problem does not know the final answer. The research process often involves trial and error, counterexample search, hypothesis revision, and failure identification.
Therefore, a research-oriented evaluation cannot only check whether the final answer “looks right.” It must also check whether the intermediate argument can be reviewed, whether the conclusion can be verified, and whether failure is honestly recognized. Otherwise, a model may score highly through language fluency or answer memorization without demonstrating transferable research ability.
2. Why TCS Is a Good Touchstone for Research Capability
TCS is special because many of its conclusions can be verified by machines. Whether a proof is correct, whether a counterexample holds, and whether an algorithm complexity derivation is self-consistent can usually be broken down into checkable steps. This makes TCS one of the few “gradeable” areas for research-oriented evaluation.
However, this does not mean that every TCS problem can be scored automatically. Open conjectures have no known answer, so a machine cannot directly judge whether a model has truly solved them. A more reasonable approach is layered evaluation:
- Verifiable problems: The model must provide a complete proof or counterexample, which is checked by proof checkers and test cases.
- Semi-open problems: The model is not required to solve the final conjecture. Instead, the evaluation assesses whether it can propose testable intermediate propositions, construct small-scale counterexamples, or provide proof sketches.
- Open exploration problems: The focus is on problem formalization, hypothesis generation, identification of failure boundaries, and follow-up verification paths, rather than claims of “solving” the problem.
This layered design helps avoid a common misconception: mistaking “the model produced a seemingly complete proof” for “the model solved the problem.” On open problems, one of the most valuable capabilities is precisely to acknowledge uncertainty and provide intermediate results that can be further verified.
3. Evaluation Design: Abstracting Scoring Dimensions from Four Types of Research Work
The exam paper in the public study contains 100 problems and corresponds to four types of work that theoretical computer scientists do daily. The specific problems should not be copied directly, but an evaluation framework can be abstracted from them: problem definition and formalization, proof construction and verification, algorithm design and complexity analysis, and open-problem exploration and hypothesis generation. Each type requires different scoring priorities.
Problem definition and formalization: This examines whether the model can translate a natural-language problem into a precise mathematical object, clarifying inputs, outputs, constraints, and boundary conditions. Scoring should focus on whether the definition is unambiguous, whether key conditions are missing, and whether checkable examples can be provided.
Proof construction and verification: This examines whether the model can generate a proof that can be checked step by step. Scoring should not only look at the conclusion, but also at whether lemmas hold, whether the reasoning chain is closed, and whether there are circular arguments or hidden assumptions.
Algorithm design and complexity analysis: This examines whether the model can propose an algorithm, analyze its correctness and complexity, and identify lower bounds or counterexamples. Scoring should include small-scale tests, randomized differential testing, and checks of complexity derivations.
Open-problem exploration: This examines whether the model can propose testable intermediate propositions, construct counterexamples, provide proof sketches, or clearly state reasons for failure. Such problems should not be scored against a single standard answer; instead, multiple verifiers and human spot checks should be used.
In addition, contamination-resistant design is essential. Private held-out sets, dynamic variant problems, process records, and cross-checking with multiple verifiers can help prevent models from relying on public answers in training data. For open problems, the model’s assumptions, attempted paths, and reasons for giving up should be recorded, and “verifiable intermediate progress” should be included in scoring.
4. Example Scoring Pipeline
The following is a generic pseudocode for an evaluation pipeline. The key point is verifiability and traceability, rather than reliance on a specific tool:
function evaluate(problem, model_output):parsed = parse_structure(model_output)if parsed.invalid:return score_failure("Unparseable structure")formal_score = check_formalization(problem, parsed)proof_score = run_proof_checker(problem, parsed.proof)test_score = run_small_tests(problem, parsed.algorithm)boundary_score = check_failure_acknowledgement(parsed)score = weighted_sum(formal_score,proof_score,test_score,boundary_score)record_trace(problem.id, model_output, score, checker_logs)if is_open_problem(problem):score = human_review_sample(problem, model_output, score)return score
The key to this pipeline is not “producing a total score,” but preserving checker logs, failure reasons, and human review records. For research-oriented evaluation, traceability is often more important than a single score.
5. Implications for Developers and Enterprises
First, do not use general leaderboards directly as the basis for model selection. If the business involves complex reasoning, code generation, or research assistance, build a domain evaluation set close to the actual scenario and add verifiable checkers.
Second, prioritize “verifiable reasoning” tasks. Proofs, test cases, constraint satisfaction, and counterexample search that can be checked by machines are more stable than pure text scoring and easier to turn into continuous regression tests.
Third, position the model as a research assistant rather than a replacement. The fact that open conjectures have not yet been independently solved by the reference models shows that current models are better suited to hypothesis generation, proof sketching, counterexample search, and literature clue organization. Final judgment still requires human experts.
Fourth, engineering implementation can use general cloud capabilities to orchestrate the evaluation pipeline. For example, object storage can manage problems and results, container platforms can isolate evaluation tasks, and logging and monitoring services can track runtime status. Baidu AI Cloud also offers general-purpose capabilities in these categories that can be used to host evaluation tasks and manage results. The focus should be on evaluation design, verifiers, and data governance, not on chasing a single benchmark score.
Conclusion
The value of TCS evaluation is not to create another leaderboard, but to distinguish “solving problems” from “doing research.” The fact that no model can yet independently solve open conjectures helps researchers see the boundaries of current capabilities more clearly: large models can be efficient exploration assistants, but they are not yet researchers who can independently sign off on conclusions. For developers, the truly reusable lesson is to build domain evaluation systems that are verifiable, traceable, and contamination-resistant, so that scores serve capability judgment rather than replace it.
