Can You Trust AI Coding Benchmarks? Build Your Own Task Set to Measure Real LLM Coding Ability

1. An Evaluation Controversy and the Structural Problems Behind It

A claim has recently been circulating in the community: in a widely watched coding-challenge series where models take on messy legacy codebases, an early episode had a challenger model defeat a model widely recognized as an industry flagship — and afterwards, many viewers suspected that the episode was actually sponsored content. The claim’s origin is hard to trace, and its truth cannot be verified. But the more interesting question is this: why does this kind of suspicion always resonate so quickly with so many people?

Because it hits three structural weaknesses in how large language models are currently evaluated:

1. Commercial conflicts of interest cannot be ruled out. Once evaluation becomes a content business, topic selection, task difficulty, and scoring criteria can all be swayed by commercial interests, while the audience has no way to tell whether a given episode was paid.
2. Private tasks cannot be replicated. Many evaluations rely on privately built tasks: the problems are not published, the process is opaque, and the results cannot be reproduced. Even if the author is completely honest, the mere fact that the work cannot be verified should reduce the trust you place in the conclusions. By the standards of scientific methodology, an experiment that cannot be replicated is not a conclusion — it is an anecdote.
3. Public benchmarks are easily distorted. In the other direction, public benchmarks suffer from gaming and data contamination: the problems may have long since leaked into training corpora, or been specifically optimized for, so leaderboard scores drift away from real capability.

To put it in one line: private tasks cannot be verified, public tasks are easily contaminated, and commercial evaluations carry questionable incentives. If you want to know whether a model actually works for your business, the most reliable method remains almost embarrassingly simple: assemble a batch of tasks yourself and run them by hand. The rest of this article lays out a complete, practical way to do that.

2. Four Design Principles for a Self-Built Evaluation Task Set

Before you start, be clear about what makes a task a “good task”:

1. Verifiability first. A task must have objective, decidable success criteria — ideally expressed as checklists or automated tests. Anything that can only be judged by “I feel it’s right” injects subjective noise into the verdict.
2. Sufficient discrimination. A task cannot be too easy — easy tasks are passed by everyone and reveal no gaps — and it should not rely purely on trick questions detached from real scenarios. The ideal task is one where strong models get it right and weaker models fail.
3. Adequate complexity. The hard parts of real work are never isolated small functions; they are complex systems with long contexts, many modules, and historical baggage. If the complexity is not there, the evaluation degenerates into a memorization test.
4. Close to your real workload. The tasks should mirror the kind of work you actually need done. If your scenario is frontend visualization, do not test only algorithm puzzles; if your scenario is maintaining legacy systems, do not test only writing new code from scratch.

3. Frontend Tasks: 3D Modeling and Video Generation, Plus Subject-Matter Checks

Why choose this direction? 3D modeling and video generation are areas where model vendors are investing heavily but capability has not yet converged. These tasks have low saturation and relatively scarce public training data, so models can hardly pass by memorization — which makes them effective at exposing real capability boundaries.

How to raise the discrimination? Simply asking a model to “render a nice-looking 3D scene” separates models poorly. Instead, layer subject-matter checks onto the task so that “beautiful” and “correct” both count toward the score:

• Build a 3D human anatomy demo: whether the relative positions and hierarchical structure of bones, muscles, and organs are correct is a far stronger test of cross-disciplinary knowledge integration than lighting and shading.
• Build a 3D celestial motion demo: are the orbital eccentricities and the ratios between orbital and rotational periods accurate?
• Build a mechanical motion video: are the gear ratios and linkage trajectories physically plausible?
• Build a data visualization animation: are the axis semantics and data mappings consistent?

The scoring must be two-dimensional: first, visual quality and interactive completeness (frame rate, interactivity, resource consumption); second, structural and factual correctness. An organ placed in the wrong position or absurd orbital parameters are classic cases of “polished on the surface, broken underneath.” Scoring “looks” and “correctness” separately is what lets you tell the aesthetic performers from the engineering performers.

4. Backend Tasks: Deliberately Build a Legacy Code Mess That Meets the Bar

The difficulty of backend evaluation is that real tasks are either too simple (change an interface, fill in a function) or uncontrollable (test against production code and there is not even a reference answer). One workable method: deliberately construct a legacy code mess that is complex enough and comes with its own answer key (a “legacy code mess” here means patch-on-patch, structurally chaotic, hard-to-maintain leftover code). The process:

1. Pick a host project. Find a genuinely complex project on an open-source code hosting platform: many modules, real users, a long iteration history. Make sure the license permits internal testing use, and keep the modified result strictly for internal evaluation — do not distribute it.
2. Locate a defect-dense point in the project’s history. Use the project’s issue tracker to find problems that have already been fixed, then locate the branch or commit before the fix. That revision preserves a real defect — a natural “original exam question.”
3. Pull both versions of the code. Get the pre-fix and post-fix code. The post-fix version is the reference answer, which lets you check whether the model’s fix converges with the community’s actual solution.
4. Reprocess the code with a weak model. Hand the pre-fix code to a deliberately weaker model together with misleading prompts — encourage it to “tidy up the structure,” “do a bit of refactoring,” or keep obsolete idioms — and let it rework the code. The weak model’s capability ceiling guarantees the degree of chaos: drifting naming, redundant abstractions, self-contradictory comments, patch wrapped upon patch. A fully loaded legacy mess is born.
5. Assemble the task. Finally, give the model under test an instruction like the ones it will face in the real world: “Here is a piece of legacy code containing a defect (quoting the original bug report). Fix it, and make sure existing functionality does not regress.”

The elegance of this method: the complexity is real (it comes from a real project’s historical bugs), the verification is objective (whether the defect is gone, whether the original test suite still passes, whether regressions were introduced — all can be judged automatically or semi-automatically), and the difficulty is tunable (switch to an even weaker intermediary model, or stack on more misleading changes, to push the difficulty higher). It is essentially a simulation sandbox for enterprise legacy systems — production legacy messes are, after all, the accumulated sediment of years of “not-quite-right” changes.

5. Checklists and Strong-Model Arbitration: Let a Flagship Model Do the Grading

With the tasks in hand, one link remains: how do you grade? Human grading is the most reliable, but it becomes the bottleneck once the task volume grows. A reusable approach: write an explicit checklist for each task and hand the grading to a currently strong flagship model acting as the examiner. The premise that makes this mechanism work is that grading is easier than answering. Three points when writing the checklist:

1. Itemized and decidable. Break the success criteria into atomic check items, each of which can be independently judged pass or fail, minimizing the grader’s room for subjective discretion.
2. Weighted priorities. Core correctness items should carry far more weight than style items, so that “pretty code that doesn’t actually fix the bug” does not score high.
3. Cross arbitration. Where possible, have two different strong models grade independently; items on which they disagree go to human review, further reducing the error rate.

An illustrative checklist might be organized like this:

Task: Fix defect Y in legacy module X
Core items (weight: 80%)
- [ ] The defect reproduction script no longer triggers the error
- [ ] The project's existing test suite passes in full
- [ ] Changes are limited to modules related to the defect, with no out-of-scope edits
Bonus items (weight: 20%)
- [ ] A regression test covering the defect has been added
- [ ] The root cause of the defect has been eliminated rather than locally patched
Veto items
- [ ] New compile errors / runtime exceptions have been introduced
- [ ] External interfaces have been removed or broken

6. Execution Details: Don’t Let Statistics Ruin a Week of Preparation

• Run each task multiple times and report the statistics. Generative models are stochastic. Run each task at least 3–5 times and report the mean and variance of the pass rate. Never present only the best run — that is, in essence, no different from the cherry-picking that sponsored reviews rely on.
• Control the variables. Give every model under test the same context length limit, the same tool-call permissions, and the same prompt template. Otherwise you are measuring configuration differences, not model differences.
• Record the full cost. Beyond right and wrong, record token consumption, latency, and API cost. “Expensive but passes on the first try” and “cheap but needs five retries” are completely different engineering conclusions.
• Pin the version snapshot. Whether you deploy models locally or call them through a cloud API platform (for example, a model invocation service such as Baidu AI Cloud), always record the model version number — a single silent vendor upgrade can invalidate your conclusions.

7. Conclusion

The credibility crisis in evaluation comes down to this: the barrier to testing keeps falling, while the barrier to testing credibly never has. Leaderboards can be gamed, reviews can be bought, and private results cannot be checked. Rather than arguing over whether someone else’s episode was a paid promotion, take the reins of evaluation back into your own hands: use frontier tasks to find capability boundaries, use real legacy messes to test engineering quality, use explicit criteria to squeeze out subjectivity, and use repeated runs to counter randomness. A score you ran yourself is the only one that deserves to be called “measured.”