Designing Verifiable Tasks for LLM Evaluation: Moving Beyond Leaderboard Illusions

1. The Evaluation Dilemma: Why Benchmark Scores Are Becoming Less Trustworthy

Over the past year, discussions about large model capabilities have rarely strayed far from various “challenge arenas” and “battle leaderboards.” In one evaluation round, a model with a later version number defeated an opponent widely considered stronger—and the result immediately drew heavy skepticism, with many suspecting the content was a commercial collaboration rather than a neutral test.

What makes such skepticism hard to rebut is the lack of verifiability in the evaluation itself:

• Private tasks cannot be audited: The task creator does not disclose the questions, the scoring scripts, or the raw outputs. Outsiders see only a conclusion—and no one can verify whether that conclusion is correct.
• Public tasks invite leaderboard gaming: Once a benchmark dataset is public, the training side can fold it into optimization objectives. A model may score highly on the leaderboard yet perform modestly on a fresh batch of same-distribution questions—classic overfitting-driven score inflation.
• Subjective scoring criteria: Many evaluations offer only an overall judgment of “which answer is better,” lacking decomposable dimensions, which makes conclusions hard to reproduce.

The conclusion is straightforward: rather than trusting someone else’s leaderboard, build a batch of tasks yourself and run them. With your own tasks, you know exactly what is being tested, and the scoring criteria stay in your hands. Below is an actionable methodology for self-built evaluation.

2. Three Foundational Principles of Task Design

Before creating tasks, establish three principles that determine whether evaluation results are trustworthy.

2.1 Tasks Must Be “Sufficiently Complex” to Resist Simple Pattern Matching

Simple tasks (writing a sort function, fixing a typo) offer very low discrimination—both strong and mid-tier models can get them right. What truly separates models are tasks requiring multi-step reasoning + structured output + domain knowledge.

2.2 Tasks Must Be “Verifiable,” with Scoring Criteria Defined Upfront

Scoring criteria should be written before the task is issued, not decided after seeing model output based on “which feels better.” Verifiable means: given an output, anyone applying the same criteria will reach a similar conclusion.

2.3 Tasks Must Cover “Structural Correctness,” Not Just Surface Appearance

Generation tasks in particular are easily misled by “looking beautiful.” A 3D model may render gorgeously, but if the human skeleton has the wrong bone count or a planet’s orbital direction is reversed, it has failed. Structural correctness often reflects a model’s true understanding far better than aesthetic quality.

3. Frontend Task Design: 3D Modeling and Video Generation

Models have largely converged on “generating a presentable interface,” but clear gaps remain in structured 3D content and long-sequence video—exactly the areas worth evaluating.

3.1 Upgrading from “Good-Looking” to “Correct”

For 3D tasks, for example, you can ask a model to generate:

• An anatomical 3D human model: testing bone count, joint connectivity, and relative organ positions;
• A celestial motion 3D scene: testing planet count, orbital inclination, revolution direction, and proportional relationships;
• A mechanical structure 3D disassembly: testing part hierarchy and assembly order.

The scoring dimensions for these tasks can be decomposed as follows:

Dimension	Description	Example Weight
Structural correctness	Whether part counts, hierarchy, and connection topology are correct	40%
Domain knowledge accuracy	Whether it conforms to subject-matter common sense (e.g., bone count, planet order)	30%
Geometric plausibility	Whether there is interpenetration, floating geometry, or distorted proportions	20%
Visual presentation	Materials, lighting, readability	10%

Note the weighting logic: structural correctness carries the highest weight, while visual presentation carries the lowest. Rendering quality can be compensated for with post-processing tricks, but structural errors cannot be hidden.

3.2 Video Tasks Test Temporal Consistency

Video generation is well suited to testing stability over long time sequences. You can require a short video containing an explicit event order—for example, “an object rises, pauses, then falls”—and focus scoring on:

• Whether the event order matches the instruction;
• Whether there are frame-to-frame jumps or object deformation drift;
• Whether the number and positions of keyframes match expectations.

4. Backend Task Design: Manufacturing “Controllable Spaghetti” from Real Projects

Backend capability evaluation most easily falls into the low-discrimination trap of “ask the model to write a CRUD app.” A better approach is to construct refactoring tasks based on real, complex projects, as follows.

4.1 Choose a Sufficiently Complex Open-Source Project

Pick a medium-to-large project on a code hosting platform. Criteria include:

• Many modules and complex dependency relationships;
• An active issue tracker, making known defects easy to locate;
• Clear branch management, so “before” and “after” versions can be extracted.

4.2 Use Issue Records to Locate Defect-Dense Branches

Browse the project’s issue list to find branches or commit ranges where defect reports cluster. These locations typically correspond to real logic flaws, missing boundary conditions, concurrency problems, and similar issues—naturally valuable for evaluation.

4.3 Manufacture “Spaghetti Code” as a Starting Point

The key technique: do not hand clean code to the model. The correct approach is:

1. Extract the “before” code;
2. Hand it to a weaker model with deliberately misleading prompts, inducing it to produce structurally messy, poorly named, heavily coupled code;
3. Confirm the code runs but is hard to maintain—that is a qualifying piece of “spaghetti.”

Example: a misleading prompt might read, “Just get it running ASAP—don’t refactor, don’t add comments, put all logic in one function.” A weak model will typically comply, and the resulting code is exactly the complex starting point we need.

4.4 Task Delivery and Scoring

Deliver this “spaghetti” code together with the task description to the model under evaluation, asking it to complete a specified refactoring (e.g., fix defects, split modules, add tests). Scoring criteria are likewise defined upfront:

• Whether defects are actually fixed (verified using reproduction steps from the original issue records);
• Whether new regressions are introduced;
• Whether code structure improves (cyclomatic complexity, function length, coupling);
• Whether runnable test cases are added.

5. Automated Scoring: Using a Strong Model as Judge

Once task volume grows, manual scoring becomes impractical. A sufficiently capable model can serve as a judge model, combined with pre-written check criteria for automated scoring.

5.1 Structure of the Judge Prompt

A qualified judge prompt should include:

Role: You are a strict code/content reviewer.
Input:
  - Task description
  - Scoring criteria (by dimension, with weights)
  - Output to be evaluated
Requirements:
  1. Score each dimension independently and give reasoning;
  2. Each dimension score is an integer from 0 to 10;
  3. Compute the weighted total at the end;
  4. If the output cannot be verified for a dimension, mark it "unable to determine" and exclude it from the total.
Output format: JSON, including per-dimension scores, reasoning, and total score.

5.2 Reduce Judge Bias with Structured Output

Requiring the judge model to output JSON rather than free text significantly reduces parsing cost and subjective drift. Also recommended:

• Score the same output multiple times and take the median to reduce randomness;
• Retain the judge’s reasoning for manual spot-checking;
• Periodically calibrate the judge model with human-labeled samples to prevent criteria drift.

5.3 Cross-Validation

If conditions permit, use two different strong models as judges and compare their score differences. Samples with large discrepancies should go to manual review—this is often also a signal that the task design itself is ambiguous.

6. Practical Recommendations and Common Pitfalls

Recommendations:

• Task count need not be large: 10–20 high-quality tasks often offer better discrimination than 100 simple questions;
• Write scoring criteria into a versioned document to ensure comparability across evaluation batches;
• Record full inputs and outputs for every evaluation to support later tracing.

Common pitfalls:

• Task leakage: Self-built tasks, if made public, will likewise be optimized against. Keep a non-public “held-out set”;
• Judge bias: Judge models may favor verbose answers. Criteria should explicitly state that “concise and correct” beats “verbose and partially correct”;
• Testing without analysis: The purpose of evaluation is to find problems. After obtaining scores, analyze failure cases to infer the model’s capability boundaries.

7. Summary

Evaluation credibility comes not from leaderboard rankings but from verifiable tasks, reproducible criteria, and auditable results. Rather than debating whether a given battle was a commercial collaboration, do it yourself: use 3D structure tasks to test spatial and domain knowledge, use real-project refactoring tasks to test engineering ability, and use upfront scoring criteria plus a judge model to automate scoring. Conclusions obtained this way truly belong to you.