How to Build a Reproducible LLM Capability Evaluation Task Set

I. The Starting Problem: Why “Win or Lose” Is Harder to Judge

In LLM capability evaluation, an increasingly common dilemma is that a single win or loss cannot reliably reflect a model’s true level. Especially when evaluation tasks come from private environments, external observers cannot reproduce the tasks, inspect the data, or verify the results; they can only rely on the conclusions provided by the publishers. This introduces two layers of uncertainty.

First, the closed nature of private tasks means results cannot be verified by third parties. Whether a model has genuinely solved a problem or merely performed better under specific prompts, data slices, or scoring criteria is difficult for outsiders to judge.

Second, public tasks fall into another bias trap. Because the questions, answers, or scoring methods have been public for a long time, the model may have indirectly or directly encountered this content during training. Even without actively “gaming the leaderboard,” a model might achieve inflated scores simply because it has seen more similar samples.

Therefore, rather than continuing to question whether a specific evaluation is objective, it is better to reframe the problem: How can we build a relatively controllable, reproducible, and continuously usable evaluation task set ourselves? This not only avoids relying on conclusions from a single source but also allows for repeated verification of capability changes as models update.

II. Design Principles: Tasks Must Be “Hard Enough” and “Verifiable”

Before building a custom evaluation set, a few principles must be clarified.

1. Tasks Must Be Objectively Verifiable

If a task only evaluates “looks good” or “reads smoothly,” the final score is easily skewed by subjective preferences. Ideally, every task should have descriptive completion criteria. For example, does a 3D model contain specified organs and correct structural positions? Does a coding task pass specific test cases or fix the target defect?

2. Tasks Should Prevent Models from “Memorizing Answers”

When selecting public questions, avoid benchmarks that have already been massively indexed, cited, and discussed. It is highly recommended to construct tasks temporarily from real projects or extract snippets from recently published data. This reduces the probability of the model hitting the right answer purely through memory.

3. Tasks Must Distinguish “Looks Correct” from “Actually Correct”

Many models can generate structurally complete and beautifully formatted content, but upon deeper inspection, critical logic errors or conceptual confusions emerge. Therefore, task design should not only look at the output format but also include checkpoints that penetrate superficial forms.

4. Evaluation Must Be Isolated from Generation

If the same model is allowed to complete the task and score itself, the result is usually lenient. A better approach is to assign task generation to the model under test, while assigning the inspection to a stronger model that did not participate in the generation, supplemented by fixed rules for review.

III. Frontend Task Design: Using 3D Modeling and Video Generation to Assess Spatial and Physical Understanding

Frontend tasks are suitable for evaluating a model’s comprehensive grasp of spatial structures, physical constraints, visual logic, and cross-disciplinary knowledge. Common text-based Q&A is no longer sufficient to differentiate model capabilities. Due to their high complexity and numerous verifiable checkpoints, 3D modeling and video generation are better suited as advanced evaluation methods.

1. Why 3D Modeling is an Effective Test

3D modeling requires the model to simultaneously understand geometric structures, proportional relationships, spatial positions, and inter-object constraints. The generated result is not a block of natural language, but an observable, decomposable 3D object. Inspectors can view the model’s structure from multiple angles to verify if it meets the prompt’s requirements.

More importantly, 3D tasks can incorporate academic subjects. For example:

• Whether human anatomy is correct;
• Whether the relative positions of planets in the solar system are reasonable;
• Whether the meshing direction of gears in a mechanical transmission is correct;
• Whether the load-bearing relationships in an architectural structure conform to basic mechanics.

These elements test not just “aesthetic appeal,” but whether the result holds up physically and logically.

2. The Value of Video Generation Tasks

Video generation adds the dimension of time, requiring the model to handle motion continuity, object occlusion, light/shadow changes, and camera logic. A visually appealing single frame does not mean the video is logically sound. Evaluation can be done through segmented checks, keyframe extraction, and motion trajectory analysis to see if the model truly understands the dynamic process.

For instance, if the model is asked to generate a video of the Moon orbiting the Earth, you shouldn’t only look at the visual style. You must also check:

• Whether the Moon revolves around the Earth;
• Whether the rotation direction is consistent;
• Whether the lighting matches the positional relationship;
• Whether Earth’s rotation and revolution are simultaneously represented.

If the model generates a video that “looks like space” but the Moon’s trajectory is chaotic, it indicates the model did not genuinely understand the underlying astronomical logic.

3. Verification Methods for Frontend Tasks

Every 3D or video task needs predefined checkpoints. Taking a “3D model of a human heart” as an example, verification criteria can include:

• Whether it contains basic structures like atria and ventricles;
• Whether the left-right positions are correctly mirrored;
• Whether the major blood vessel connections are reasonable;
• Whether the model is rotatable for observation without obvious structural errors from any angle.

This type of verification can be assigned to a strong model for an initial round of judgment, followed by manual sampling and review.

IV. Backend Task Design: Constructing Complex Coding Tasks from Real Projects

Compared to frontend tasks, backend coding tasks offer stronger verifiability. Whether code runs, tests pass, or defects are fixed has relatively clear criteria. The problem is that models have likely seen too many common algorithm questions and simple functions from public repositories.

Therefore, a more appropriate approach is to extract real historical defects from large projects to construct “legacy codebase” repair tasks.

1. Why Use Defect History from Complex Projects

Large projects typically contain numerous modules, complex call relationships, and hidden state dependencies. Certain defects in these projects are often not single-line errors but involve interactions across multiple files, functions, or even services. The model must first understand the context, locate the problem, and finally propose a fix.

By locating the defective branch from historical commits or issue logs, you can obtain two versions:

• Defective version: The code before the issue was fixed.
• Fixed version: The code after the official or community fix.

The difference between these two versions forms the core logic of the task.

2. How to Construct “Legacy Codebase” Inputs

Handing the defective version directly to the tested model might not be difficult enough, especially given modern models’ strong code comprehension capabilities. In this case, you can introduce interfering steps. For example:

• Remove some comments from the pre-adjusted code;
• Rename a few key variables to make their semantics more ambiguous;
• Add irrelevant functions or redundant branches;
• Shuffle the file organization.

More importantly, a weaker model can be used to assist in the construction process. Provide the defective code to the weaker model and inject misleading prompts, prompting it to generate code that “appears fixed but actually introduces more issues.” The resulting code is usually structurally chaotic and logically jumpy but retains traces of the original defect, making it perfectly suited as an input for the tested model.

3. Task Requirements Should Focus on Defect Repair, Not Style Optimization

Backend tasks should not stop at vague requirements like “optimize code” or “improve readability.” Clearer task descriptions should be:

• Locate the root cause rendering a specific feature unavailable;
• Fix the specified defect;
• Maintain the original external behavior;
• Do not introduce new regression issues;
• Explain the reasoning for the modification and its impact scope.

Under such requirements, the model must not only generate code but also demonstrate its diagnostic logic. If a model simply rewrites the code without pointing out the actual cause, it indicates insufficient debugging capability.

4. Using Test Cases as Hard Criteria

Every task should come with executable verification methods. You can extract relevant test cases from the original project or write regression cases based on the defect’s behavior. For example:

Input: A request with specific boundary conditions
Expected: Returns a specific error code instead of crashing directly

Before passing, confirm that the defective version reproduces the issue; after passing, confirm that the issue is resolved and other related cases remain unaffected.

V. Scoring Mechanism: Strong Model Review Over Self-Evaluation

After generation tasks are complete, the scoring phase is equally critical. A three-tier structure of “rule-based validation + strong model review + manual spot-check” is recommended.

1. Tier 1: Fixed Rule Validation

For tasks with clear answers, such as whether code passes tests or whether a 3D model contains specified elements, programmatic validation can be directly applied. This type of verification does not rely on model judgment, offers the highest stability, and should be prioritized.

2. Tier 2: Strong Model Inspection

For tasks that are difficult to fully program—such as whether the model’s explanation for a modification is reasonable, or whether a 3D structure is anatomically correct—a stronger model can be used to score it against predefined standards. The scoring prompt should include:

• Task background;
• Completion criteria;
• Scoring range for each checkpoint;
• Requirement to provide reasons for deductions.

This reduces subjectivity while retaining the ability to judge complex content.

3. Tier 3: Manual Sampling Review

Even if the strong model’s scoring is stable, manually spot-checking a certain proportion of results is still recommended. Particularly for structural errors in frontend tasks or motion logic errors in videos, current automated scoring may still have blind spots. The goal of manual review is not to re-score, but to verify whether the automated scoring results are trustworthy.

VI. Task Maintenance and Iteration

An evaluation task set should not remain static once built. A reasonable approach is to update it regularly:

• Frontend tasks can rotate through academic subjects, shifting among medicine, astronomy, mechanics, architecture, biology, etc.;
• Backend tasks can continuously extract new defects from different projects;
• Tasks that have become public or widely circulated should be phased out or transformed into variants promptly;
• After each model update, run regression comparisons using the same task set to observe capability trends.

Through this approach, you can gradually accumulate an evaluation asset of your own. Although the construction process requires a time investment, it significantly reduces the probability of being swayed by one-off promotional results.

VII. Conclusion

The core issue of LLM capability evaluation lies not in whether there is an “absolutely fair” leaderboard, but in whether we possess a reproducible, verifiable, and iterative evaluation methodology. Rather than oscillating between closed conclusions and public benchmark gaming, it is better to take control of the evaluation process ourselves.

Frontend tasks assess spatial understanding, physical logic, and cross-disciplinary capabilities through 3D modeling and video generation; backend tasks construct coding environments close to real engineering scenarios using complex project defects; the scoring phase forms a closed loop through rule validation, strong model review, and manual spot-checks. This methodology does not rely on any single model or platform, but serves as a universal practical approach for engineering teams.

When tasks are sufficiently complex, standards are sufficiently clear, and inspections are sufficiently independent, the gaps between models become easier to see—and much harder to obscure with rhetoric.