The Trust Problem in AI Coding Tool Benchmarks and How to Build Your Own Evaluation Suite

Why Benchmarks Often Look Better Than They Are

AI coding tools have been iterating at an accelerating pace over the past two years. Almost every few weeks, a new release claims to “surpass the previous generation” on some benchmark. Alongside these claims, a recurring question keeps surfacing: was that public comparison actually fair, or was there commercial promotion behind it?

These doubts are not unfounded. They reflect two structural problems that have long existed in AI coding tool evaluations:

1. Private tasks cannot be verified. If an evaluation uses a set of undisclosed tasks, outsiders can only see the conclusions. They cannot reproduce the process. Whether the tasks were hand-picked, whether the scoring criteria favored one side, and whether failure cases were hidden all remain impossible to verify.
2. Public tasks invite benchmark overfitting. If a task set is public, it eventually finds its way into training data or becomes a target for deliberate optimization. A model that scores well on a public benchmark does not necessarily perform reliably in real engineering scenarios.

In other words, “private equals untrustworthy, public equals gameable” is the fundamental paradox of benchmark trust. The most practical way to escape this paradox is not to keep debating whether a given leaderboard is fair, but to build your own verifiable, reproducible task set that reflects real engineering work.

The rest of this article presents a practical self-built evaluation framework. It covers task construction for both frontend and backend directions, along with a scoring mechanism design.

Core Principles for Self-Built Evaluations

Before getting started, a few principles should be established to prevent the evaluation itself from becoming another round of guesswork.

Principle 1: Tasks must be complex enough that templates cannot bluff through them.
Simple tasks such as “write a sorting function” have already been conquered by nearly all mainstream models and offer almost no differentiation. Only complex tasks can expose differences in long-context handling, multi-file collaboration, and architectural understanding.

Principle 2: Check criteria must be quantifiable and minimize subjective judgment.
Criteria like “clean code” or “nice UI” are too vague. They should be broken down into checkable items such as structural correctness, functional correctness, edge case handling, and performance constraints.

Principle 3: The scorer must be separated from the model being scored.
Use a sufficiently strong model as the “judge” to inspect the output of other models. This reduces the cost of manual review while maintaining a degree of consistency.

Principle 4: Task sources should be authentic, ideally from real defects in real projects.
Synthetic tasks tend to be too clean. They lack the historical baggage, coupled logic, and implicit conventions found in real projects. Real defect branches are the best testing ground.

Frontend Tasks: From 3D Modeling to Cross-Disciplinary Checks

For the frontend direction, tasks involving 3D modeling, 3D visualization, and video processing are recommended. There are two reasons:

• These tasks remain at the frontier of model capabilities and have not yet been fully saturated, so they provide strong differentiation.
• They demand structured output. It is not enough to render an image; geometric relationships, hierarchical organization, and animation timing must also be correct.

Task Construction Approach

Ask the AI to complete tasks such as:

• Present human anatomical structures in 3D;
• Simulate planetary orbits in 3D;
• Demonstrate mechanical assembly relationships in 3D.

The key point is: do not judge only by visual appeal. Verify whether the structure is correct.

For example, if the task is to “create a 3D model of human anatomy,” scoring should check:

1. Whether the relative positions of organs are correct (for example, the spatial relationship between the heart and lungs);
2. Whether the hierarchy is reasonable (bones, muscles, and organs properly grouped);
3. Whether necessary interactions are supported (rotation, zooming, layer visibility);
4. Whether animations and annotations correspond to the actual structures rather than floating on the surface.

Introducing Cross-Disciplinary Checkpoints

Pure 3D modeling can easily hide structural errors behind visual polish. It is therefore recommended to layer subject-specific checkpoints into the task, such as:

• Planetary orbit 3D: check whether orbital period ratios, orbital inclinations, and rotation-revolution relationships align with basic physics;
• Human anatomy 3D: check whether anatomical terms correspond to actual positions;
• Mechanical structure 3D: check whether gear meshing and linkage motion follow mechanism principles.

The value of this approach is that the model must not only write rendering code but also demonstrate understanding of the domain knowledge behind the task. This is exactly where many current tools fail.

Example Scoring Dimensions

Frontend tasks can be broken down into the following check items:

Check dimension	Description	Suggested weight
Structural correctness	Whether geometric, anatomical, or physical relationships are correct	High
Functional completeness	Whether interactions, animations, and annotations are complete	Medium
Code organization	Whether component separation and state management are clear	Medium
Visual presentation	Whether rendering, lighting, and materials are reasonable	Low

Note that the last item should be deliberately down-weighted to prevent visually appealing but incorrect output from scoring well.

Backend Tasks: Building Legacy Code from Real Defect Branches

The backend approach is closer to real engineering: select complex projects from real repositories, then pick branches with a high density of defects.

Selecting Task Sources

Choose projects with high complexity on code hosting platforms. Criteria may include:

• A large number of modules and complex dependencies;
• A long commit history and many unresolved issues;
• Multiple branches, some of which have notably higher defect density.

Issue trackers can help locate defect-heavy branches. These branches are often ideal raw material for tasks.

Constructing Legacy Code

The concrete steps are:

1. Retrieve both the pre-fix version and the post-fix version of the code;
2. Give the pre-fix code to a weaker AI tool;
3. Simultaneously provide it with misleading prompts that nudge it toward changes that look plausible but actually go in the wrong direction;
4. Ensure the final deliverable is a large pile of messy legacy code: tangled structure, heavy coupling, superficially functional but loaded with hidden risks.

The point of this step is not to belittle weaker models, but to create a sufficiently complex starting point for the task. Real engineering challenges often emerge from exactly this kind of messy code. Only by asking the model under evaluation to refactor, fix defects, and add features on top of such a codebase can its true capabilities be revealed.

Example: Writing a Backend Task Description

A proper backend evaluation task should include:

• Background on the existing code and the observed problem;
• The expected goal (which defects to fix, which constraints to satisfy);
• Existing behaviors that must not be broken;
• Acceptable performance and resource boundaries.

For example:

Task: Fix the abnormal order state transition in the existing service.
Constraints:
1. Do not modify the request/response structure of the external interface;
2. The fix must pass existing regression tests;
3. State transitions must be idempotent;
4. No new external dependencies may be introduced.

This kind of task has clear acceptance criteria while leaving enough room to assess the model’s understanding and design ability.

Scoring Mechanism: Using a Strong Model as the Judge

Once tasks are built, the most critical step is scoring.

Write Check Criteria for Every Task

Each task should come with a set of check criteria that are as specific as possible, such as:

• Whether the functionality works as expected;
• Whether edge cases are handled;
• Whether regressions were introduced;
• Whether the code structure is maintainable;
• Whether the given performance constraints are met.

Use a Strong Model as the Judge

Send the completed work (code diffs, runtime results, documentation) together with the check criteria to a sufficiently strong model. Ask it to review each item and provide a score with justification.

The advantages of this approach:

• Low cost: compared with manual line-by-line review, model-based review can run in batch;
• Good consistency: scoring tends to be stable under the same criteria;
• Traceable: the judge model’s reasoning can be kept as evidence.

Caveats

Using a model as the judge carries risks and requires constraints:

1. The judge model must be strong enough; otherwise it cannot catch advanced errors made by the model under evaluation;
2. Check criteria should be as objective as possible; avoid subjective items like “whether the code is elegant”;
3. Introduce manual spot checks when necessary to calibrate scoring results;
4. Keep original outputs and scoring records for later review.

Turning Evaluation into a Long-Term Capability

A self-built evaluation is not a one-off effort. It should be cultivated as a long-term team capability.

• Keep growing the task library: every real defect or refactoring can be turned into a new evaluation task;
• Keep iterating on scoring criteria: as model capabilities improve, existing tasks may lose differentiation and need to be updated;
• Keep results reproducible: tasks, code versions, check criteria, and scoring records should all be preserved in full.

Returning to the original debate: rather than wondering whether a particular comparison is commercial promotion, run some tasks yourself. Use 3D modeling and cross-disciplinary checkpoints to construct frontend tasks, use real project defect branches to build messy legacy code for backend tasks, write clear check criteria for each task, and hand the results to a sufficiently strong model for scoring. The conclusions you get this way are verifiable and truly your own.

Trust in evaluations is never given by others. It is built by yourself.