Building a Reliable LLM Code Evaluation System: From Subjective Impressions to Objective Verification

Cutting Through the Fog of LLM Code Evaluation: An Automated Guide from Subjective Impressions to Objective Verification

Recent discussions about large language models performing impressively in “spaghetti code refactoring challenges” have sparked debate. Some voices question whether such evaluations may involve commercial interference, undermining the objectivity of the results. Setting aside the specific background of any individual evaluation, the controversy itself exposes a deeper pain point in the current landscape of LLM code capability assessment: private tasks cannot be reproduced or verified, while public leaderboards carry the risk of “leaderboard gaming” through test-set-specific optimization.

For developers and enterprise technical decision-makers, relying solely on external evaluation reports to select intelligent coding tools is clearly insufficient. The most reliable path is to build a controlled and objective evaluation system of one’s own. This article describes in detail how to construct such an evaluation framework—covering everything from front-end multimodal generation to back-end complex engineering refactoring—to comprehensively and realistically test a large model’s code processing capabilities.

1. Core Pain Points in Current LLM Code Evaluation

Before building a custom evaluation system, we need to clarify why existing mainstream evaluation methods are difficult to trust:

1. The “black box” problem of privatized tasks
   Many comparative evaluations use private tasks defined by the evaluator. Such tasks often do not disclose test cases, input/output standards, or even the system prompts used. This makes it completely impossible for external developers to reproduce the testing process. Regardless of which model ultimately wins, the lack of transparency makes the conclusions easy to label as “customized tuning” or “commercial promotion,” greatly diminishing their practical reference value.

2. The “leaderboard gaming” phenomenon on public datasets
   In pursuit of public transparency, the industry has introduced well-known public code evaluation benchmarks such as HumanEval. However, as the scale of training data for large models grows exponentially, public datasets are highly susceptible to being mixed into model training corpora. This leads to “leaderboard gaming”—models perform exceptionally well on specific public benchmarks but often fall short when facing real code with intricate logic in actual production environments, creating a serious evaluation bias.

Given these pain points, the only way to break through the information barrier is for developers to design tasks themselves and run the full pipeline.

2. Design Principles for High-Barrier Evaluation Tasks

To effectively test a large model’s true code understanding and generation capabilities, task design must have sufficient complexity and “traps.” Tasks cannot be as simple as “implement a quicksort”—they should closely resemble real spaghetti code scenarios. The following are two recommended directions for custom task design:

(A) Complex Front-End Scenarios: Cross-Disciplinary, Multi-Dimensional Structured Generation

Most large models have reached a relatively mature level in basic front-end UI generation; simple dashboards and forms no longer differentiate them. To test advanced front-end capabilities, the focus should be on areas like “3D modeling and video production”—domains that vendors are actively working on but have not yet fully mastered.

Design strategy:

1. Introduce cross-disciplinary checkpoints: Require the model not only to generate visually appealing pages but also to incorporate hard knowledge checkpoints from a specific discipline.
2. Dual verification criteria: Check not only the rendered visual output (whether it looks good) but also the underlying code logic and data structures (whether the structure is correct).

Practical examples:
You can ask a large model to build an interactive application using a mainstream WebGL or front-end 3D rendering library. For example:

• Human anatomy 3D model: Require the model not only to render 3D meshes of organs but also to accurately reflect spatial hierarchy relationships (such as the relative position of the heart and lungs) and vascular pathways based on anatomical knowledge. If the generated model can rotate but organ positions are anatomically incorrect, it fails.
• Celestial motion 3D simulation: Require the implementation of planetary revolution and rotation in the solar system. The checkpoint lies in applying Kepler’s laws—planetary orbits must be elliptical rather than perfect circles, with faster speeds at perihelion and slower speeds at aphelion. This directly tests a large model’s grasp of physical laws and its ability to translate them into mathematical logic in code.

(B) Complex Back-End Scenarios: Spaghetti Code Refactoring Tests on Real Open-Source Projects

The focus of back-end code testing is on understanding existing complex logic, locating bugs, and performing non-destructive refactoring. We can synthesize an “extremely complex spaghetti code project” as a test set.

Steps for synthesizing spaghetti code:

1. Find a base sample: On an open-source code hosting platform, select a project with complex business logic and a long commit history.
2. Locate a defective branch: Using the project’s issue list, find a historical branch containing a known, unfixed bug. Download its source code as the “ground truth” after adjustments.
3. Create spaghetti code: Take the pre-adjustment code containing the bug and feed it to a less capable open-source model or an early-version model. At the same time, provide misleading system prompts (e.g., “Please do not use design patterns, try to write all logic in a single function, use as many global variables as possible, and do not write comments”). Through this approach, the weaker model produces a large chunk of unreadable, logically chaotic “spaghetti code” that still contains the original bug.

This synthesized spaghetti code, due to its extreme complexity and abundance of anti-patterns, is an excellent litmus test for evaluating a top-tier large model’s engineering refactoring capabilities.

3. Objective Verification Mechanism: Introducing a Strong Reasoning Model as an Automated Judge

Once tasks are designed, the biggest challenge is objectively judging whether the model’s refactored or generated code is correct. Manually reviewing large engineering codebases is time-consuming and prone to subjectivity. At this point, you can introduce the strongest closed-source large model available for code (assume Model-O) as an “automated judge.”

Evaluation process:

1. Write inspection criteria: For each test task, write a detailed acceptance checklist.
2. Construct the judge prompt: Package the acceptance criteria, the spaghetti code content, and the refactored code output by the model under test, and feed them all to Model-O.
3. Automate the inspection: Require Model-O to check each criterion one by one, indicating whether the code under test fixed the original bug, introduced new logic errors, or improved code quality.

Illustrative checklist example:

# Task: Refactor the user permission verification module
## Acceptance Checklist
1. [Critical Bug] Must fix the privilege escalation vulnerability mentioned in Issue #123 (the original code did not verify the admin's secondary permissions).
2. [Logic Constraint] Must not change the external interface signature (function parameters and return value structures must remain compatible).
3. [Code Quality] Must remove the global variable `$global_user_state` and replace it with context passing.
4. [Performance Constraint] The permission tree traversal depth must not exceed 3 levels to avoid stack overflow.

By having Model-O output a structured JSON evaluation report, developers can quickly quantify the actual scores of different large models in this “spaghetti challenge.” This “strong model evaluating weaker/peer models” approach not only removes suspicion of human intervention but also greatly improves verification efficiency.

4. Conclusion

Facing a mixed landscape of LLM evaluations, developers need to stay clear-headed. Whether suspecting a particular challenge of being commercially customized or worrying about leaderboard manipulation, the only way to break through is to “get hands-on.” By building cross-disciplinary front-end complex generation tasks, synthesizing customized “spaghetti code” from real open-source defects using weaker models, and combining the strongest model as an automated judge, we can build a code capability evaluation system that is logically rigorous, difficult to game, and closely aligned with real production environments. Only through such rigorous sandbox exercises can we provide truly valuable reference points for a team’s technology selection decisions.