Beyond Benchmarks and Sponsored Reviews: Building Your Own AI Coding Evaluation

Why AI Coding Evaluations Keep Ending Up in a “Rashomon” Situation

Not long ago, on a coding challenge show, a newly released model beat a long-standing flagship model on a private task. After the episode aired, rumors began circulating in the community that the segment was sponsored. The claim was hard to trace and impossible to verify. But it exposed a deeper problem: this type of evaluation cannot prove its own credibility by design.

The trust problem in AI coding evaluations can be broken down into two categories.

First, private task demos are unverifiable. Such content usually shows only the final result. The full prompts are not published, intermediate steps are not shown, and no reproducible environment is provided. Viewers cannot confirm whether the result is actually correct, nor whether another model would perform better on the same prompt. Under this information asymmetry, skepticism is inevitable regardless of whether the publisher had a commercial relationship.

Second, public benchmarks carry the risk of leaderboard gaming. Once a benchmark is public, it may end up in training data. Even without direct leakage, vendors can optimize their models specifically for the question types. The result is a gap between leaderboard scores and real-world experience. A model with impressive benchmark numbers may not feel comfortable in your own work.

Both paths have cracks. The only remaining answer is: use your own tasks, run them yourself, and verify the results yourself.

Four Design Principles for Self-Built Evaluations

Before designing tasks, establish four principles. Otherwise, the evaluation can easily become superficial.

1. Stay close to real workloads. The task format should resemble your daily work. If your routine involves maintaining legacy systems, do not evaluate models only on building an application from scratch.
2. Make it verifiable. You must be able to judge correctness for every task. Either you have a reference answer, or the result can be checked automatically. Tasks you cannot verify only lead to more disputes.
3. Resist data contamination. Self-built tasks are naturally absent from any model’s training set. This is their biggest advantage over public leaderboards, so keep them private.
4. Be reproducible. Fix the prompts, context organization, and sampling parameters, then run multiple trials and observe the distribution. A single run is heavily influenced by randomness and cannot support a conclusion.

Frontend Tasks: Combining 3D and Video Challenges with Domain Knowledge

Why Choose These Tasks

3D modeling, animation, and video generation are areas where current models compete fiercely but have not yet fully matured. The tasks are open-ended with a large combination space, making it easier to separate model capabilities. More importantly, these tasks are naturally layered. “Looks good” is only the first layer; “is it correct” is the deeper one.

Key Technique: Embedding Domain Knowledge into the Task

Do not simply ask the model to create a flashy page. Add domain knowledge requirements so that visually impressive but factually wrong outputs cannot slip through. A few directions:

• Human anatomy in 3D: require correct relative positions and proportions of bones, muscles, and organs. The heart leans left, the liver sits in the upper right abdomen, and the lungs have left and right lobes. If the structure is wrong, no amount of rendering polish earns a point.
• Celestial motion in 3D: require correct ratios between orbital and rotational periods, and coplanar orbits. Earth’s orbital period is roughly 365 times its rotational period. Planetary orbital periods increase with orbital radius. Getting these details right shows the model is actually drawing on physical knowledge rather than piling up visual effects.
• Molecular structure visualization: whether bond lengths and bond angles match the geometric parameters of real molecules.
• Mechanical transmission animation: whether the speed ratio of meshing gears follows the inverse ratio of their tooth counts.

An example task could look like this:

Build an interactive solar system demo using web technologies:
1. Eight planets orbit the sun with coplanar, concentric orbits.
2. Orbital periods are scaled to real ratios (Earth = 1, Mercury ~0.24, Jupiter ~11.86).
3. Each planet rotates on its own axis, with axial tilt allowed (Earth ~23.5 degrees).
4. Support drag-to-rotate camera control. Clicking a planet opens its parameter panel.

Scoring in Three Layers

1. Visual quality: lighting, materials, composition, and interaction smoothness.
2. Structural correctness: whether domain facts are accurate. Give this a high weight, or even veto power.
3. Engineering quality: frame rate, memory usage, and code maintainability. Can another engineer pick up the output and continue working on it?

The strength of this design is that you naturally possess the verification criteria. Domain knowledge is public and checkable. You do not need to be an aesthetics expert to confidently mark “the heart drawn on the right side” as wrong. By constraining subjective beauty within an objective correctness framework, the scoring becomes defensible.

Backend Tasks: Mass-Producing Legacy Code

Motivation

In real backend work, building from scratch is the minority. Most daily work happens inside a messy legacy system: fixing bugs and making incremental changes. First understand it, then stabilize it, then consider refactoring. So the right format for backend evaluation is: give the model a sufficiently complex pile of bad code and see whether it can fix it.

The challenge is finding code that is complex enough, bad enough, and for which you know the correct answer. The answer is to manufacture it yourself.

The Manufacturing Process

1. Choose a project. Pick a mid-to-large-sized open source project with multiple modules and some history on a code hosting platform. The closer the tech stack is to your business, the better.
2. Locate the defects. Go through its issue tracker and find branches or historical versions where defects were concentrated and fixes were well documented. The more issues, the better the material.
3. Take two versions. Pull both the pre-fix and post-fix code. The post-fix version naturally serves as the reference answer, and its test suite becomes a ready-made verification tool.
4. Degrade it manually. Feed the pre-fix code to a weaker model with a deliberately misleading prompt, and have it repeatedly patch the code.
5. Assemble the product. A sufficiently messy legacy codebase + a known correct implementation + a test suite = one high-quality evaluation task.

Example of a Misleading Prompt

The goal is to push a weaker model to produce output that resembles real-world legacy code:

You are an engineer under deadline pressure. Modify this code according to the following rules:
1. Do not try to understand the existing architecture. Patch the code wherever the error occurs.
2. Prefer copying existing functions, renaming them, and reusing them directly.
3. Use try-catch liberally to swallow exceptions. The program just needs to not crash.
4. When requirements change, stack more if-else on top of the existing logic. Do not split functions.
5. Hardcode temporary parameters. Just make it work first.
6. Do not write comments, do not write tests, and do not rename any variables.

Why This Method Works

• Sufficient complexity. Real legacy code is full of misleading names, implicit dependencies, and patches stacked on patches. It thoroughly tests context understanding and defect localization, which is exactly where models differentiate themselves.
• Verifiable answers. You have the post-fix version and the test suite. Correctness can be judged by machines, not subjective impressions.
• Close to real work. A model’s performance on this task directly reflects its ability to pay down your technical debt after joining the team.

Two additions. First, prepare multiple legacy codebases covering different tech stacks to avoid having your conclusion skewed by a single domain. Second, pay attention to the license terms of the open source projects you use. Internal evaluation is usually fine, but keeping a record of the source is good professional practice.

Writing a Rubric and Letting a Strong Model Grade

Tasks are ready. The last step is verification. Manually reviewing dozens of outputs line by line is neither practical nor stable. The common approach is to use the strongest currently available model as the judge, provided you give it a sufficiently explicit rubric.

Example Rubric

[Task] Fix defect X in the specified module so that test_suite passes completely
[Hard criteria 60%]
- All test cases pass: 40%
- Public interface signatures unchanged: 10%
- No new heavy dependencies introduced: 10%
[Soft criteria 40%]
- Fix addresses the root cause rather than masking symptoms: 15%
- Exception and edge case handling is thorough: 10%
- Naming and code style consistent with the repository: 10%
- Key changes accompanied by explanations: 5%
[Output format] Itemized scores + deduction reasons + total score (0-10)

Three Cautions

1. Let machines handle deterministic checks first. Anything that can be tested, compiled, or statically checked should not be delegated to the judge model. Programmatic judgment has zero variance. Save the model’s judgment for the genuinely subjective parts.
2. Watch out for judge bias. A judge model may favor outputs similar to its own style. A safer approach is to anonymize outputs and consider cross-scoring with models from different technical approaches, then average the results.
3. Do not change a single word in the rubric. The same rubric must remain identical across all models under evaluation. Any difference in wording introduces noise that can drown out real differences between models.

Execution Details: Making Your Conclusions Stand Up to Scrutiny

• Control variables. All models should use the same prompts, the same context organization, and the same sampling parameters. Different temperatures or different multi-turn strategies create unfair comparisons.
• Repeat runs. Run each task at least 3 to 5 times. Report the median rather than the single best result, and observe the variance. Stability itself is a core dimension of engineering usability.
• Track process metrics. First-pass success rate, number of interaction turns required, context consumption, and generation latency. These reflect real usage experience better than a binary pass/fail outcome.
• Keep a complete trail. Save the full prompts, raw outputs, and process data from every run. This makes review easier and helps you defend your conclusions if challenged.
• Use general infrastructure at scale. As the task count grows, use container platforms to spin up isolated evaluation environments in batches, object storage to retain artifacts, and logging services to record the full process. These are standard capabilities, so use them as needed.

Common Pitfalls Checklist

1. Announcing that one model crushes another after a single run. Given the randomness involved, that is just one coin flip.
2. Creating tasks in domains you cannot verify. An evaluation where you cannot judge correctness is equivalent to no evaluation at all.
3. Writing prompts that are too detailed, feeding the answer directly, or too vague, leaving the model guessing. Both extremes flatten the differences between models. Prepare several semantically equivalent variants and cross-validate.
4. Only checking whether the code runs while ignoring engineering quality. Code that barely works today becomes tomorrow’s legacy mess.
5. Ignoring cost dimensions. For the same task, differences in context usage and generation time translate into real money at scale.

Conclusion

Debates over whether a particular episode was sponsored will never reach a conclusion that satisfies everyone, because the trust chain of private demos cannot be closed by design. Rather than taking sides in comment sections, spend a day or two designing a few frontend tasks close to your own work, manufacturing a legacy codebase, writing a clear rubric, and running your candidate models through the same gauntlet. Leaderboards belong to others; the tasks are yours. Conclusions drawn from your own tasks are the only ones that truly take responsibility for your technology decisions.