How to Build Verifiable Private Complex Tasks to Evaluate AI Coding Capabilities

1. Why Build Your Own Private Complex Tasks

When evaluating AI coding capabilities, many developers simply refer to public leaderboards or third-party benchmark results. These are cheap to obtain, but they come with two problems that are hard to avoid.

First, public benchmarks are vulnerable to gaming and contamination. Once an evaluation set becomes widely used, some models may have seen similar samples during training—intentionally or not—so the final score no longer reflects true generalization ability. Benchmarks built on fixed questions with fixed answers are especially exposed: models can achieve inflated scores simply by memorizing specific patterns.

Second, private tasks avoid data leakage, but usually lack verifiable standards. When you design a task yourself and hand it to a model, a task that is too simple cannot differentiate between models, while a task that is too complex makes it hard to judge whether the output is genuinely correct. For example, if a model generates business logic code, the code may run fine while many subtle logic errors slip through unnoticed.

The more reliable approach, therefore, is not to depend on a single source, but to build a set of complex tasks under your own control—one that is reproducible and quantitatively judgeable. This frees you from the noise of public leaderboards, while explicit checking criteria keep evaluation results consistent.

2. Frontend Task Design: Starting from 3D Modeling and Video Generation

Frontend scenarios are well suited to examining how well a model understands complex structures. Compared with traditional page layouts or component-writing exercises, 3D modeling and video generation involve far more complexity, and current models still show clear capability gaps in these two areas.

1. Why Choose 3D Modeling Tasks

A 3D modeling task naturally embeds spatial structure, geometric relationships, and physical constraints. The generated artifact cannot be judged by appearance alone; it must also pass structural validation. Example tasks:

• Ask the model to generate a 3D model of human anatomy, labeling the relative positions of major organs.
• Ask the model to generate a celestial motion simulation, checking whether planetary orbital inclinations, revolution directions, and relative distances are plausible.
• Ask the model to generate a mechanical transmission structure, checking gear meshing, rotation directions, and transmission ratios.

These tasks share one key property: they have objective structural-correctness criteria instead of relying on subjective aesthetics. An anatomical model that looks beautiful but has organs in the wrong places must be judged a failure.

2. Adding Interdisciplinary Knowledge to Increase Discrimination

To raise the difficulty further, you can embed knowledge points from different disciplines into the 3D scene. For example:

• Medicine: generate the four-chamber structure of the heart and label the pulmonary and systemic circulation paths.
• Astronomy: simulate the positional relationship among the Earth, the Moon, and the Sun during a solar eclipse.
• Physics: generate a mechanism that satisfies the lever principle and output the numerical relationships among the fulcrum, lever arms, and load.

The core idea is that the model must do more than generate code—it must correctly understand and apply the relevant subject knowledge during generation. A mistake in any part of this chain shows up in the final artifact.

3. How to Define Checking Criteria

For frontend 3D tasks, criteria can be defined along the following dimensions:

Dimension	What to Check
Structural correctness	Whether the relative positions, proportions, and connections of components meet the task requirements
Executability	Whether the generated code can be parsed and executed normally by the target runtime
Interaction and animation logic	Whether motion trajectories and timing follow physical laws, when animation is required
Domain knowledge accuracy	Whether the embedded knowledge points are expressed correctly

After each dimension is broken down into explicit pass and fail conditions, the criteria can be handed to another, stronger model for automated checking, reducing the subjectivity of manual judgment.

3. Backend Task Design: Recreating Real Legacy Code Refactoring Scenarios

In backend evaluation, asking a model to write a simple API or algorithm from scratch no longer differentiates models effectively. A better approach is to simulate the legacy code maintenance and refactoring work that dominates real projects.

1. How to Source Complex, Real Codebases

An effective approach is to select projects from open-source code hosting platforms that have a complex structure, a long history, and a real maintenance burden. When choosing a project, consider:

• A codebase of a few thousand to a few tens of thousands of lines—too small fails to reflect complexity, while too large makes evaluation prohibitively expensive.
• Multiple historical branches or commit records, so you can retrieve versions before and after a change.
• A well-populated issue tracker, whose bug reports help you locate the most defect-prone code areas.

Using the bug descriptions in the issue tracker, you can fairly accurately identify the modules with the messiest logic, the most problems, and the highest risk of introducing new errors.

2. Deliberately Manufacturing “Spaghetti” Code

Simply handing the original project code to a model makes the task too clean, and it will not adequately test the model’s comprehension and repair ability. To raise the difficulty, deliberately construct a degraded version using the following process:

1. Select a defect-heavy module or file from the project.
2. Retrieve the pre-fix historical version of that code as the base.
3. Use a weaker model as the “code polluter” and feed it the selected code.
4. Add misleading instructions to the prompt, such as “optimize the naming but preserve the logic,” “simplify the structure without changing behavior,” or “remove duplicate code while keeping compatibility.”
5. The model’s output will usually pile a large number of new defects on top of the original ones—incorrectly merged conditional branches, deleted valid code, infinite loops, or faulty type conversions.

The result is a “spaghetti” sample with sufficiently high complexity and defect density, while still preserving the original business context. Because it is derived from real project code, the realism of this artifact is impossible to match with hand-designed exercises.

3. Task Formats and What They Test

With the degraded code in hand, you can design tasks such as:

• Bug-fix tasks: given code containing several known and unknown defects, the model must locate and fix all of them.
• Readability refactoring: without changing external behavior, the model must reduce cyclomatic complexity, eliminate duplicated logic, and improve naming.
• Test-writing tasks: the model must write thorough tests for messy logic, and those tests must expose the pre-existing problems.

Bug-fix tasks are the easiest to quantify, because the fix can be compared behaviorally against the reference version that the upstream project eventually shipped.

4. Building Independent Checking Criteria and an Automated Judgment Process

For both frontend and backend tasks, relying on manual item-by-item inspection is extremely inefficient, and evaluation quality fluctuates with the reviewer’s condition. The checking process should therefore be automated as much as possible.

1. Design Principles for Checking Criteria

Checking criteria should meet the following requirements:

• Objectivity: every check can be clearly judged as pass or fail, without relying on subjective impressions.
• Codability: the checking logic can be written as scripts or prompts and executed automatically by a strong model.
• Coverage: the checks must cover the core requirements of the task, not just edge issues.

For example, a backend bug-fix task could define checks like these:

1. Does the fixed code pass the relevant build or static analysis checks from the original project?
2. For every issue listed in the bug report, does the submission provide a reproducible verification of the fix?
3. On a given set of inputs, does the output of the fixed code match the reference (upstream-fixed) version?
4. Does the fix introduce new dead code, unused variables, or duplicated logic?

2. Using a Strong Model as the Checker

The checker itself should be a currently strong model—one with stable performance in reasoning and code analysis. Provide the task requirements, the checking criteria, the code under test, and the necessary context, and require the checker to output a structured verdict.

If you are concerned about the checker’s own judgment bias, you can adopt a two-round review mechanism:

• In the first round, the checker independently gives a pass/fail conclusion for each check item.
• In the second round, the checker’s conclusions are submitted again together with the original task, and it is asked to re-examine the disputed items.
• If the two rounds still disagree, the item is escalated to human review.

This significantly improves the reliability of the verdicts.

5. Putting It Together: A Complete Evaluation Workflow

Combining the methods above yields a relatively stable private evaluation process:

1. Build the task set: prepare several 3D modeling and video generation tasks for the frontend, and several degraded refactoring tasks for the backend.
2. Freeze tasks and criteria: before evaluation begins, freeze all task descriptions, input code, and checking criteria as versioned files.
3. Run each model separately: distribute the same tasks to different models, requiring complete runnable artifacts or code patches.
4. Automated checking: use a strong model combined with script tools to judge each output item by item against the criteria.
5. Record and compare: compile each model’s pass rate across tasks and record failure modes for later analysis.

The core value of this process is that every task is under your own control, with no dependence on third-party evaluation results; the checking criteria are explicit, so results can be re-verified repeatedly; and the tasks themselves come from real projects or complex scenarios, providing genuine discrimination between models.

6. Conclusion

The key to evaluating AI coding capability is not finding one “authoritative” leaderboard, but establishing an evaluation system you control. Public leaderboards can remain a reference, but they should not be the sole basis for decisions. By introducing complex 3D and cross-disciplinary tasks on the frontend, real legacy code refactoring and repair tasks on the backend, and independent, automatically executable checking criteria, developers can obtain a capability profile much closer to real engineering scenarios at relatively low cost. The purpose of this evaluation approach is not to prove which model is better, but to help you make the right technical choice for your specific application scenario.