The Rashomon Effect in LLM Benchmarking: Building a Trustworthy, Reproducible Stress-Test Framework for Complex Coding Tasks

Why Benchmark Conclusions Are Always Questioned

A recent controversy illustrates the problem: in a challenge focused on refactoring messy legacy code, one model defeated a widely recognized stronger model. Soon after, questions arose about whether the evaluation had commercial backing. The root cause is not whether any single evaluation was fair, but that the entire benchmarking paradigm contains structural flaws.

For developers evaluating coding capabilities, the meaningful question is not “who beat whom,” but “how do I obtain conclusions that are useful for my own needs?” This article focuses on verifiability in evaluation mechanisms, discusses why both private tasks and public leaderboards fail, and presents an actionable stress-testing framework covering two tracks: front-end generation quality and back-end complex refactoring.

The Double Bind: Private Tasks vs. Public Leaderboards

Evaluation systems generally fall into two categories, and both carry unavoidable problems.

The core defect of private tasks is non-verifiability. When the organizer designs tasks, executes them, and publishes conclusions all in-house, external readers cannot re-examine whether task difficulty is reasonable, whether scoring criteria were consistent, or whether failure cases were fully disclosed. Even if the organizer is completely honest, readers can only choose to “trust” or “not trust”—this does not constitute engineering-grade reproducibility. Once evaluation content becomes associated with commercial partnerships, skepticism naturally amplifies—not because there is necessarily a problem, but because the mechanism cannot rule one out.

The core defect of public tasks is overfitting and leaderboard gaming. When task sets, scoring scripts, and test cases remain openly available for extended periods, model providers can improve scores through targeted training, prompt-template tuning, or even feature fitting to the evaluation set itself. A rising leaderboard score does not necessarily correspond to a genuine capability increase. Especially when the task distribution is fixed and the scoring dimension is narrow, the correlation between score and practical utility steadily decays.

Both problems lead to the same conclusion: any evaluation performed by a third party can only serve as a reference signal, not as a decision basis. The reliable path is to construct your own tasks, define your own acceptance criteria, and observe results yourself.

Overall Design of a Self-Built Stress-Test Framework

A usable self-built evaluation must satisfy three conditions simultaneously: the tasks are sufficiently complex, the tasks can be constructed automatically, and acceptance criteria can be clearly defined. Below is a two-track plan covering front-end and back-end tasks.

Front-End Tasks: Replace “Does It Look Good” with Structural Correctness

For the front-end track, focus on 3D modeling and video generation tasks. These areas are evolving rapidly but are not yet stabilized, so capability differences between models are more likely to surface. At the same time, these tasks have explicit structural constraints that enable objective verification.

The key principle: do not evaluate visual aesthetics alone—evaluate structural correctness. Aesthetics are subjective and easily fooled by surface-level polish; structural correctness can be tested. The practical approach is to embed cross-disciplinary checkpoints into the task. Examples:

• Generate a human anatomy 3D model, requiring bone count, joint-connection relationships, and relative organ positions to conform to anatomical conventions.
• Generate a 3D solar-system scene, requiring correct orbital hierarchy, orbital-period ratios, and satellite-parent relationships.
• Generate a mechanical-assembly demonstration, requiring correct part-mating relationships and degrees of freedom under mechanism constraints.

Verification can be performed in two layers. The first layer checks structural assertions—for example, “does the scene graph contain the specified number of identifiable parts” and “does the parent-child hierarchy match expectations.” The second layer checks rendering quality and interaction fluency. The prompt should explicitly require the model to output a structured scene description (e.g., a node tree or parameter table) so that automated comparison is possible, rather than just returning a screenshot.

Example task (structured output required):
Generate a 3D schematic of the solar system and output the scene structure as JSON:
{
  "bodies": [
    {"name": "...", "type": "star|planet|moon", "parent": "...", "orbit_period_ratio": 1.0}
  ]
}
Acceptance assertions: number of planets, satellite assignments for each planet, tolerance range for orbital-period ratios.

The value of this design: structural assertions are automatable, results are reproducible, and “drawing nicely” cannot pass the test.

Back-End Tasks: Construct Messy Codebases from Real Projects

The core idea for the back-end track is to build tasks from complex defect branches in real open-source projects, rather than hand-crafting toy-level problems.

The workflow is as follows:

1. Select projects on code-hosting platforms that have high structural complexity and tight module coupling.
2. Use the issue tracker to locate defect-dense branches or commit intervals, and pick a defect that spans multiple modules and requires a large fix.
3. Export both the pre-fix and post-fix versions of the code.
4. Feed the pre-fix code to a weaker model, appended with deliberately misleading instructions, so that the model produces a modified result that runs but has chaotic structure, arbitrary naming, and blurred responsibility boundaries—the “messy codebase.”
5. Use this output as the input task for the model under evaluation.

The design rationale: defect fixes in real projects often span multiple abstraction layers, and the chaotic code produced by a weak model with misleading prompts faithfully simulates the “legacy technical debt left by a predecessor” scenario. The complexity of such tasks cannot be replicated by hand-written exercises, and because they originate from real projects, the structure and dependency relationships are naturally coherent.

When exporting code, strip project-identifying information, keeping only the structure and logic, to avoid any association between the evaluation and a specific project.

Acceptance: Use a Strong Model for Rule-Based Verification

After task construction, write a clear checklist for each task. The checklist must be decidable—for example:

• Whether the specified defect is fixed and whether the corresponding regression tests pass.
• Whether the refactored code introduces new failing tests.
• Whether module responsibilities are consolidated as expected and whether obvious duplicated implementations remain.
• Whether all structural assertions for the front-end task are satisfied.

Then, provide the checklist and the model output to a sufficiently strong model for verification. The purpose of using a strong model here is not to give a subjective score, but to check each item against rules that a human has pre-defined, and to output a structured acceptance report. The evaluation actually relies on the rules you wrote; the strong model merely executes them as a rule interpreter. This reduces manual review effort while keeping conclusions re-checkable.

Engineering Details for Implementation

First, task quantity and diversity. Results from a single task are highly incidental. Prepare multiple tasks per track, covering different complexity levels and knowledge domains, and observe the pass-rate distribution rather than a single win/loss outcome.

Second, the boundary of misleading prompts. When constructing the “messy codebase,” the misleading prompts should simulate realistic ambiguous requirements and wrong assumptions in development—not deliberately create impossible tasks. The goal is to produce code that “runs but is messy,” not code that cannot execute.

Third, independence of acceptance rules. The checklist must be written before you see the output of the model under evaluation. Adjusting criteria after seeing results destroys objectivity.

Fourth, result recording and reproducibility. For every task, retain the full set: task description, input code, checklist, model output, and acceptance report. This allows others to reproduce the conclusion under identical conditions. This is the core advantage of self-built evaluation over private evaluation.

Summary

Disputes about model coding capabilities cannot be resolved by argument alone. Private evaluations lack verifiability, and public leaderboards face overfitting risks—neither can independently support engineering decisions. The practical approach is to take evaluation into your own hands: use 3D generation tasks with cross-disciplinary checkpoints to test structural correctness on the front-end, use real-project defect branches combined with weak-model output to construct complex back-end tasks, and then use pre-written checklists with a strong model to automate acceptance. This framework is cost-controlled, reproducible, and does not depend on the credibility of any external leaderboard—which, at the current stage, is the most pragmatic way to judge a model’s real capability.