The Trust Crisis in LLM Benchmarks and How to Build Your Own Evaluation Tasks

1. Why Evaluation Results Are Increasingly Hard to Trust

As large language model capabilities advance rapidly, evaluation results of all kinds are flooding the landscape. It is common to see a model claim victory over a stronger competitor on some leaderboard, only for a wave of skepticism to follow. Some suspect the evaluation was commercially motivated. Others question how the test set was selected. Still others point out that the task itself is not representative of real-world use.

This trust crisis is not unfounded. It stems from inherent flaws in the two dominant evaluation approaches:

• Private tasks: The task content is not published, so external parties cannot replicate the results. Even if the evaluator presents plausible numbers, third parties have no way to verify them. Subjective judgment, biased task design, and even the risk of data leakage can all distort the outcome.
• Public tasks: The task and dataset are open, so replication is theoretically possible. But precisely because they are public, model developers can optimize against them, effectively gaming the leaderboard. A soaring score does not necessarily mean better performance in real scenarios.

In other words, there is a structural tension between verifiability and resistance to leaderboard gaming. To obtain conclusions that are genuinely useful, the most reliable approach remains to design your own tasks, run them yourself, and evaluate them yourself. The rest of this article presents a practical evaluation methodology covering two representative scenarios: frontend and backend tasks.

2. Frontend Tasks: From “Looks Good” to “Structurally Correct”

Frontend and multimedia generation is a fiercely competitive but not yet saturated area for current models. Using these tasks for evaluation serves a dual purpose: testing the model’s generation capability while using structural constraints to avoid the pitfall of judging only surface-level appearance.

2.1 Choosing Task Types

Two categories of tasks are recommended:

1. 3D modeling tasks: For example, generating a 3D model of human anatomy, a 3D scene of planetary motion, or an exploded view of a mechanical structure.
2. Video generation tasks: For example, generating an animation with physically plausible motion or a short narrative clip with multiple camera cuts.

What these tasks have in common is that the output is judged not only on pixel-level aesthetics, but also on correctness of structure, logic, and physical laws.

2.2 Designing Scoring Dimensions

Using “human anatomy 3D” as an example, the scoring criteria should go beyond “does it look convincing” and be broken down into multiple dimensions:

Dimension	Description	Suggested Weight
Structural correctness	Do organ positions, proportions, and connections conform to anatomical knowledge?	High
Geometric integrity	Are there mesh intersections, broken surfaces, or missing components?	High
Material and lighting	Are materials reasonable and lighting natural?	Medium
Visual quality	Overall visual impression	Medium
Interactivity	Does it support rotation, zoom, and layered viewing?	Low

For a “planetary motion 3D” task, in addition to basic astronomical facts such as orbital shape, relative distance, and orbital period, you should also check:

• Whether the size ratio between planets and the star is reasonable;
• Whether orbital inclination and eccentricity fall within plausible ranges;
• Whether the animation’s time scale corresponds to real physical quantities.

The key point is that aesthetics alone are not enough. Structural correctness must be verified. A beautifully rendered human body model with misplaced organs should receive a lower capability score than a plain-looking but anatomically accurate model.

2.3 Writing Task Descriptions

Task descriptions should be as specific as possible and avoid vague instructions. For example:

Generate an interactive 3D model of a human torso containing five organs: heart, lungs, liver, stomach, and kidneys. The relative positions of the organs must conform to anatomical knowledge. The model should support mouse rotation and zoom, and use different colors to distinguish the organs.

This description provides explicit structural constraints as well as interaction requirements, making subsequent scoring more straightforward.

3. Backend Tasks: Using “Spaghetti Code” to Create Realistic Complexity

The challenge with backend evaluation is that simple algorithm problems do not reflect real engineering capability, while real projects are difficult to standardize. One workable approach is to deliberately construct a sufficiently complex “spaghetti code” task.

3.1 Task Construction Workflow

1. Select a project: Pick a reasonably complex open-source project with many modules.
2. Select a branch: Use the issue tracker to locate a branch with many bugs and frequent changes.
3. Extract the code: Export the code before and after the changes for that branch.
4. Inject the “spaghetti”: Give the pre-change code to a weaker model, along with misleading prompts, so that it produces a tangled mess of illogical structure, arbitrary naming, and heavy coupling.
5. Form the task: Use this obfuscated code as the starting point, and require the model under evaluation to complete a specified fix or refactoring goal.

The purpose of this approach is to give the task the characteristics of a real-world project: legacy baggage, implicit dependencies, misleading naming, and duplicated logic—not a clean algorithm puzzle.

3.2 Examples of Misleading Prompts

Misleading prompts are used to ensure that the weaker model produces sufficiently messy code. For example:

• “Change as little code as possible. Only modify the parts you can understand.”
• “Keep variable names as they are. Do not rename anything.”
• “If you encounter anything uncertain, just add a TODO and skip it.”
• “Prioritize getting it to run. Do not worry about maintainability.”

These prompts push the model toward short-sighted, localized changes without a global perspective, producing the classic “spaghetti code” pattern.

3.3 Designing Scoring Criteria

Scoring criteria for backend tasks should cover both fix correctness and code quality:

• Functional correctness: Is the specified bug fixed? Do regression tests pass?
• Structural soundness: Are duplicated logic and unnecessary coupling eliminated?
• Readability: Are names clear, comments appropriate, and function responsibilities focused?
• Extensibility: Can new features be added easily? Are new implicit dependencies introduced?
• Side effect control: Are existing features broken? Are new edge cases introduced?

Each task should come with an explicit checklist to minimize subjective drift during review.

4. Using a Strong Model as Reviewer: Feasible but with Limits

Once you have constructed your tasks and scoring criteria, you can introduce a sufficiently capable model as reviewer. A concrete approach:

1. Write a structured checklist for each task, including mandatory hard requirements and optional bonus items.
2. Provide the task description, the model output under evaluation, and the checklist to the reviewing model.
3. Require the reviewing model to judge each item individually and provide reasoning, rather than simply returning a total score.

The benefits of this approach include:

• Reproducibility: The criteria are open, and anyone can follow the same process to verify results.
• Scalability: Tasks can be added over time, forming your own evaluation set.
• Resistance to gaming: Tasks are private or semi-private and structurally diverse, making targeted optimization difficult.

However, there are limitations to keep in mind:

• The reviewing model itself can make mistakes, especially when judging complex structures;
• Review criteria need continuous iteration to avoid being exploited by models;
• For highly specialized tasks, manual review is still recommended.

5. A Minimal Practical Implementation Plan

If you want to start building your own evaluation set immediately, the following minimal plan works:

1. Pick two frontend tasks: one 3D modeling task and one video generation task, both with structural constraints.
2. Pick two backend tasks: Extract a defective branch from a complex project for each, and inject “spaghetti code” as the starting point.
3. Write a checklist for each task: no more than 10 hard requirements and 5 bonus items.
4. Standardize the review process: Use the same strong model and the same review template for all tasks.
5. Record and compare: Keep the inputs, outputs, and review justifications for every evaluation run to build a traceable history.

Task ID: BE-001
Task Type: Backend refactoring
Starting Point: A defective branch from a complex project (with injected spaghetti code)
Goal: Fix the specified defect and reduce module coupling
Checklist:
  [Hard] Defect A is fixed
  [Hard] All existing tests pass
  [Hard] No new external dependencies are introduced
  [Bonus] Duplicated logic is extracted into shared functions
  [Bonus] Key functions have single responsibilities
Review Output: Item-by-item judgment + reasoning + total score

6. Conclusion

The credibility of an evaluation ultimately depends on the quality of the task design and the transparency of the review criteria. Rather than agonizing over whether a particular leaderboard result was commercially motivated, it is more productive to build your own tasks, run them, and evaluate them yourself. Use 3D and video tasks on the frontend to assess structural correctness, use “spaghetti code” tasks on the backend to evaluate real-world engineering ability, and combine these with a strong model applying explicit review criteria. This approach requires some upfront effort, but the reproducibility and resistance to interference it provides are qualities that no public leaderboard can replace.