Beyond Public Leaderboards: A Practical Guide to Building Your Own AI Coding Evaluation Set

1. Why “picking models by leaderboard” keeps getting harder

1.1 The dilemma of public tasks: gaming and data contamination

The advantage of public benchmarks is that results are comparable across models and reproducible—but the cost is that they are naturally exposed to targeted optimization. Once a leaderboard becomes an industry reference for model selection, teams will iterate on prompts and run targeted fine-tuning around its question patterns, and there is even a contamination risk of test data leaking into training corpora. A higher score does not equal higher real engineering capability—the model may simply be more familiar with that type of question. This is the so-called “leaderboard gaming” problem.

1.2 The dilemma of private tasks: unverifiable and conflicted

Private tasks are the opposite: the questions (for example, an internal demo or a single comparison video) are not public, so the audience can neither reproduce them nor verify whether the results were engineered. Recent disputes over certain comparison evaluations are a typical example: some viewers questioned whether a commercial partnership was behind a particular result. Such doubts are hard to prove, but they reveal a structural problem—when the evaluator and the evaluated may have a conflict of interest, and the evaluation process is not auditable, the credibility of the conclusion cannot be established. In addition, a single task run once has no statistical significance; sampling randomness alone can flip the outcome.

1.3 Conclusion: take evaluation back into your own hands

For individual developers or teams, rather than arguing over which leaderboard is fairer, it is better to build your own evaluation set: your own tasks, your own criteria, a reproducible process, and auditable results. Below we walk through concrete construction methods for two categories of tasks: frontend and backend.

2. Four design principles for a self-built evaluation set

1. Realism: Tasks should resemble the real shape of your daily work, not be invented purely for examination. Only then can evaluation conclusions transfer directly to actual development decisions.
2. Verifiability: Every task must have objective checking criteria. Tasks that cannot be judged will not produce conclusions no matter how many times they are run.
3. Discriminative power: Questions should land in the region where current model capabilities are “rapidly iterating but not yet mature.” In areas that are already “solved” (such as simple page layout), top models differ very little, so differences cannot be measured.
4. Contamination resistance: Do not publish the evaluation set externally, and rotate questions regularly so that tested models cannot “memorize the answers.”

3. Frontend tasks: from “does it look good” to “is it correct”

3.1 Task selection: 3D and rich-media generation

In the frontend domain, tasks such as static pages and form interactions are approaching maturity and offer insufficient discriminative power. In contrast, 3D scene construction and animation/video generation are areas where vendors are still concentrating their efforts, and failure modes are diverse—incorrect spatial relationships, violations of physical laws, or code performance collapse—making them well suited to separating models.

3.2 Layer in subject-matter checkpoints to test “structural correctness”

Simply asking a model to “make a nice 3D scene” tends to slide scoring toward subjective aesthetics. A more effective approach is to embed objective subject-matter knowledge into the task so that correctness can be judged:

• 3D human anatomy: Ask the model to build a human body model containing bones, muscles, and major organs. Scoring should not only look at whether the rendering is beautiful, but also check whether bone connections, joint counts, and the relative positions and hierarchical structure of organs match anatomical facts.
• 3D celestial motion: Ask the model to simulate the solar system or a planetary system, and check the proportional relationships of orbital periods, the order of magnitude of relative distances, and the physical consistency of revolution direction. You can require the model to declare the parameters it used on delivery, making item-by-item verification easier.

Scoring for such tasks can be broken into four dimensions:

Dimension	Description	Example checkpoints
Visual presentation	Rendering quality and composition	Reasonable lighting, clear layering, no obvious clipping
Structural correctness	Compositional relationships of the model/scene	Correct bone connections, correct organ positions
Subject-matter accuracy	Conformance to objective knowledge	Correct orbital period ratios, consistent motion direction
Engineering quality	Code-level quality	Reasonable component decomposition, target frame rate, no console errors

Among these, “structural correctness” and “subject-matter accuracy” are the key to distinguishing stronger from weaker models: weak models can often produce a beautiful shell, but the internal structure does not hold up to scrutiny.

4. Backend tasks: using “artificial legacy code” to build high-complexity refactoring problems

4.1 Source material: complex projects from public repositories

Pick a large open-source project with complex business logic from a public code hosting platform, and use the density distribution of its issues to locate branches or modules where defects are concentrated—the denser the issues, the heavier the historical baggage usually is. Export the “before fix” and “after fix” code snapshots separately: the former serves as the problem statement, and the latter, together with the related issue descriptions, serves as the reference answer.

4.2 Key technique: have a weak model “manufacture legacy code”

Using the original project directly as the task has two problems: the context is too large, making evaluation costly; and the tested model may have seen the project during pretraining. The solution is to deliberately construct a piece of “controllably degraded” code:

• Hand the pre-fix code to a weaker model;
• Simultaneously feed it misleading prompts, such as: “To ship quickly, pass state through global variables,” “Catch exceptions uniformly and just print them, no need to differentiate,” “No need to write tests, we’re in a hurry”;
• Ensure it produces a piece of legacy code with chaotic naming, tangled responsibilities, and rough error handling.

You now have a task that is complex enough, controllably difficult, and has a standard answer: the tested model’s goal is to complete bug fixes or feature refactoring on this legacy code. This is closer to real enterprise maintenance scenarios than “writing a demo from scratch,” and the latter is precisely a blind spot most public benchmarks do not cover.

4.3 Scoring design

Checking criteria for backend tasks can be “harder”:

• Hard items: Build passes; existing tests do not regress; tests covering the target bug pass; no new severe static-analysis warnings.
• Soft items: Module responsibilities are clear, no circular dependencies are introduced, error handling is appropriate, and critical paths are logged.

5. The judging system: let a strong model score against criteria

5.1 Write the criteria first, then run the evaluation

For every task, write an explicit checklist first, and only then let the tested model respond. Scoring without criteria causes judges to be unconsciously swayed by “surface quality” (many comments, neat formatting) while ignoring the correctness issues that actually matter.

5.2 Use a sufficiently strong model as judge, but add safeguards

Hand “task description + checking criteria + tested output” to a sufficiently capable model for review. To reduce bias, four things are needed:

• Anonymization: Do not reveal the tested model’s name or origin to the judge, avoiding anchoring effects;
• Multiple reviews: Review the same output 3 times and take the median, flattening random fluctuations;
• Human spot checks: Periodically re-check judge scores manually to calibrate consistency;
• Automation before models: Anything that can be decided by compilation, tests, or static analysis must never be left to a judge’s subjective assessment.

An illustrative task definition might look like this:

task_id: refactor-legacy-001
material:
  source_module: a high-defect-density module from an open-source project
  snapshot_before: ...
  snapshot_after: ...        # reference answer
  related_issues: [...]      # bug context
mutation:
  generator: weak model
  misleading_hints:
    - pass state through global variables, prioritize speed
    - catch exceptions uniformly and print them
rubric:
  hard:
    - build passes
    - regression tests for the target bug pass
    - no new severe static-analysis warnings
  soft:
    - no circular dependencies
    - complete error handling and logging
judge:
  runs: 3
  aggregate: median
  human_audit_ratio: 0.1

6. Execution and statistics: from “a single win or loss” to “a credible conclusion”

• Multiple runs: Run each task at least 3–5 rounds and report the mean and variance, rather than deciding on one run;
• Tiered question bank: First run a quick screening set of 10–20 questions for initial filtering, then run a deep evaluation set on the shortlisted models to control cost;
• Versioning: Record model versions, prompts, sampling parameters, and question versions so that the same result can be reproduced three months later;
• Minimum sufficient: Frontend rich-media tasks and backend refactoring tasks are both time-consuming; keep the number of questions at a scale that is “just enough to distinguish candidate models” rather than chasing quantity.

If you want to run this process in the cloud in an engineered way, the capabilities involved are all general-purpose categories: object storage to host task packages and artifacts, a container platform to provide an isolated environment for each run, a logging service to retain execution traces, and monitoring and alerting to track failed tasks. Whichever cloud platform you choose (including Baidu AI Cloud), the same architecture applies—there is no proprietary dependency.

7. Final thoughts

The purpose of evaluation is to reduce the uncertainty of model selection, not to create a new authority. The value of a self-built evaluation set is that the questions come from your own real scenarios, the criteria are written down, the process is reproducible, and the conclusions are auditable. It does not need to cover every capability dimension of a model—only the part you care about. That is already enough to support a reliable selection decision.