The Credibility of LLM Benchmarks and a Framework for Building Your Own

1. The Problem: Why Benchmark Scores Aren’t Directly Trustworthy

During large language model (LLM) selection, benchmark leaderboards are usually the first reference material consulted. However, anyone who has conducted independent validations will discover an awkward reality: the deviation between benchmark scores and actual user experience is often much larger than anticipated.

This discrepancy primarily stems from two sources:

First, the unverifiability of private tasks. Many evaluations use “internal private task sets” as a selling point, claiming the tasks closely resemble real production scenarios and are uncontaminated. The problem with these evaluations is that they cannot be externally reproduced. You do not know the specific composition of the tasks, how the scoring criteria are defined, or whether the scoring process is manual and subjectively lenient. When the evaluator is also the model provider or has a commercial affiliation, the credibility of the conclusions becomes even harder to guarantee. Recent debates over whether certain evaluation periods are essentially commercial promotions fundamentally reflect this structural issue—without reviewable materials, no party can prove their objectivity.

Second, the risk of gaming public benchmarks. The advantage of public leaderboards is transparency, but the disadvantage is that transparency itself leads to overfitting. As long as the task sets and scoring scripts are public, there is ample room for targeted data cleaning, prompt tuning, and even training set contamination. A model might score highly on the leaderboard but perform noticeably worse on a similar, unpublicized task.

The conclusion is straightforward: evaluation conclusions from any single source should serve only as a reference. What truly holds decision-making value is constructing your own tasks, defining your own criteria, and running the tests yourself. Below is a cost-controllable, repeatable framework for building your own evaluation pipeline.

2. Overall Design of the Evaluation Framework

The core of building an in-house evaluation is not to “create a larger leaderboard,” but to create a set of sufficiently complex tasks with definitive correct answers that are difficult to pass with superficial tricks. The overall process can be broken down into four steps:

1. Task Construction: Design frontend and backend tasks by capability dimension, ensuring sufficient complexity.
2. Difficulty Injection: Actively introduce “spaghetti code” baselines for backend tasks to raise the bar for the model’s comprehension and refactoring capabilities.
3. Criteria Definition: Write executable checklists for each task, clearly defining what constitutes “completion.”
4. Automated Scoring: Use a sufficiently capable model as a judge to score against the checklist item by item, supplemented by script validation where necessary.

The key to this process is that you control both the tasks and the criteria. Although the review phase is executed by a model, the scoring basis is text that you can audit, making the entire process verifiable.

3. Frontend Task Design: Shifting from “Looks Good” to “Structurally Correct”

Frontend and creative tasks are highly competitive areas among model providers that have not yet fully converged, making them ideal for differentiating capability levels. 3D modeling and video generation are two typical starting points.

3.1 Why Choose 3D Modeling?

3D modeling tasks simultaneously test three abilities: spatial structure comprehension, structural organization of generated code/descriptions, and aesthetic control of rendering results. Many models can generate “passable” visuals, but as soon as structural correctness is required, problems surface.

3.2 Introducing Academic Constraints

Simply asking a model to “create a 3D scene” is too broad. A more effective approach is to embed explicit academic constraints into the task, for example:

• Human Anatomy 3D: Require correct relative positions of bones, muscles, and organs, with joint ranges of motion conforming to anatomical facts.
• Celestial Motion 3D: Require orbital inclinations, orbital period ratios, and axial tilts to align with basic astronomical facts, with reasonable proportional relationships for planet sizes.

The benefit of these tasks is that aesthetics are subjective, but structural correctness is objective. Scoring can clearly distinguish between “visual presentation” and “structural accuracy,” preventing models from scoring high simply by piling on visual effects.

3.3 Suggested Scoring Dimensions

Dimension	What to Check	Evaluation Method
Structural Correctness	Whether part count, relative positions, and proportions meet task constraints	Check against constraint list item by item
Parameter Accuracy	Whether periods, tilts, and sizes fall within reasonable ranges	Value comparison
Interactivity	Whether basic operations like rotation, zooming, and part highlighting are supported	Run and verify
Visual Presentation	Whether materials, lighting, and composition are reasonable	Subjective scoring, weight should not be too high

It must be emphasized: the weight of visual presentation must be significantly lower than structural correctness, otherwise the evaluation degrades into “who renders better.”

4. Backend Task Design: Using “Spaghetti Code” for Complex Bug Fixing

The challenge of evaluating backend capabilities is that real projects are too complex, while toy projects are too simple. A compromise is to extract a poorly conditioned segment of code from a real, complex project as the baseline for repair.

4.1 Sourcing Task Materials

The selection principle is “sufficient complexity, numerous defects, and traceable history.” Specific steps include:

1. Select open-source projects with complex business logic from code hosting platforms, prioritizing repositories with many modules and deep dependencies.
2. Locate heavily bugged branches or modules through issue tracking records, identifying commits with clear before-and-after repair differences.
3. Extract both the “pre-fix” and “post-fix” code versions as the task input and reference respectively.

This provides a set of natural control samples: the pre-fix is real code with defects, and the post-fix is a correct implementation verified by the community.

4.2 Difficulty Injection: Deliberately Creating a “Spaghetti Code” Baseline

Just using the pre-fix code is not ruthless enough, as the lower bound of code quality in real projects is still limited. To further increase difficulty, you can actively degrade the code:

• Give the pre-fix code to a weaker model and ask it to perform a round of “refactoring.”
• Apply misleading prompts simultaneously, such as implying an incorrect direction is a “performance optimization” or a redundant abstraction is a “decoupling design.”
• The goal is to have the weak model output a code baseline with chaotic logic, arbitrary naming, deep nesting, and unclear responsibilities.

The value of this step is that it shifts the task from “fixing a bug” to “locating and fixing a bug amidst a pile of noise.” This closely mirrors real engineering scenarios and better distinguishes the model’s problem localization capabilities and code comprehension depth.

4.3 Task Description and Constraints

The task description for the model under test should include:

• Project background and business goals (brief, without revealing the defect location).
• Reproducible failure symptoms (error messages, incorrect output examples).
• Clear acceptance criteria (functionality restored, no regressions introduced, interface compatibility maintained).

It should not include: the file containing the defect, root cause hints, or repair strategies. This information should be left for the model to infer.

4.4 Example Scoring Criteria

Each task comes with a checklist. The judging model evaluates item by item:

Task: Fix order status synchronization exception
[ ] Reproduction path works, original failure no longer occurs
[ ] Fix does not alter external interface signatures
[ ] Existing test cases are not deleted or bypassed
[ ] New logic includes boundary condition handling (null values, concurrency, timeouts)
[ ] Code readability: clear naming, no deep nesting, no copy-paste duplication
[ ] No obvious performance degradation introduced (e.g., repeated queries inside loops)

The benefit of a checklist is that the scoring process is auditable: each item has a clear basis for judgment, and the judging model’s output can be manually spot-checked.

5. Organizing the Automated Scoring Phase

The scoring phase uses a sufficiently capable model as the judge, but three constraints must be applied, otherwise the judge itself becomes a new untrustworthy source.

First, scoring inputs must be structured. Do not just throw the code at the judging model and say “score it.” Provide the checklist, the code to be evaluated, and the reference implementation (optional) together, requiring item-by-item judgment outputs and reasons.

Second, scoring outputs must be parseable. Require the judge to output in a fixed format for easy script aggregation:

{
  "task_id": "backend-001",
  "items": [
    {"check": "Reproduction path works", "pass": true, "reason": "..."},
    {"check": "No external interface signature altered", "pass": true, "reason": "..."}
  ],
  "score": 5,
  "total": 6
}

Third, critical items must have script fallbacks. Whether an interface signature changed, whether tests pass, and whether syntax errors exist can be completely determined by static checks or test scripts without relying on model judgment. Model review is better suited for items requiring semantic understanding, such as “code readability” or “sufficiency of boundary handling.”

A pragmatic approach is dual-layer scoring: scripts handle hard factual judgments, while models handle soft quality judgments. The two are weighted and aggregated. If a hard item fails, the overall result is judged as a failure, preventing models from giving lenient evaluations on obvious errors.

6. Execution Advice and Common Pitfalls

6.1 Task Quantity and Coverage

Do not pursue an exhaustive approach. Preparing 3 to 5 tasks per capability dimension is sufficient to establish initial differentiation. Focus on the quality of the tasks rather than the quantity. Frontend tasks should focus on structural constraints, while backend tasks focus on defect localization and refactoring. Keep the two categories scored separately; do not mix them into a total score.

6.2 Avoiding Same-Origin Bias Between Judge and Tested Model

If the judging model and the tested model originate from the same technical lineage, a lenient “in-group favoritism” tendency may occur. Feasible mitigation methods include: fixing the judging model to a single strong model, manually spot-checking scoring reasons, and introducing a second judge for cross-validation on disputed items.

6.3 Prompt Fairness

All tested models must receive the exact same task description and constraint text. No prompt optimization should be applied for any specific model. This seems basic, but it is easily overlooked in practice. Once differentiated prompts appear, the evaluation conclusions lose comparability.

6.4 Boundaries of Result Interpretation

The conclusions of an in-house evaluation have limited scope: they reflect performance on the specific batch of tasks you constructed and cannot be directly extrapolated to a “model’s overall capability ranking.” The correct usage is to treat it as supplementary evidence for selection, combining it with public benchmarks and actual business trial runs.

7. Conclusion

The credibility issue in evaluations is fundamentally not about “which leaderboard is more accurate,” but rather “whether you master a verification method you can audit yourself.” Public benchmarks provide broad references, and private evaluations provide scenario alignment, but neither can replace running the tests yourself.

The framework provided in this article can be summarized in one sentence: Use 3D and video tasks with academic constraints to evaluate frontend structural capabilities, use intentionally degraded real project code to evaluate backend bug-fixing capabilities, and use checklist-based criteria with script fallbacks to complete automated scoring. This process is cost-controllable, clearly structured, and auditable, making it suitable as a routine method for internal team model selection.

It is worth reminding that any evaluation framework has its blind spots. The preferences of the task designer will permeate the task set, and the strictness of the scoring criteria will affect the final ranking. Maintaining prudence regarding conclusions and regularly updating the task set is more important than pursuing an “absolutely accurate” leaderboard.