Benchmarking LLM Coding Ability: Frontend Visual and 3D Generation Evaluation

1. Why Build Your Own Coding Benchmark

Public leaderboards answer the question “how many problems did the model solve on a static dataset,” while engineering teams actually care about “can a single generation be used directly on my task.” There is significant information loss between the two:

• Corpus contamination risk: Popular problems have likely already entered training data, causing systematic upward bias in leaderboard scores.
• Task shape mismatch: Most benchmarks evaluate algorithm and function-level completion, while real frontend and 3D tasks are composite tests of “math + art + engineering.”
• Missing production constraints: Leaderboards do not include variables essential to engineering decisions such as unit cost, latency, context length, and number of correction rounds.

Therefore, constructing a “small but accurate” internal benchmark around your own business is currently the most cost-effective selection approach. Among these, frontend visual and 3D generation are precisely the two task domains with the highest discriminative power: they require the model to simultaneously understand coordinate systems and matrix transformations, the animation main loop, event systems, and rendering pipelines. Any misunderstanding in any one area is directly exposed on screen and can hardly be faked.

2. Evaluation Dimension Framework: Five Dimensions and Weights

The following weighted structure is recommended, with weights adjustable according to your team’s business focus:

Dimension	Weight	0-point anchor / 5-point anchor
Functional correctness	30%	0: cannot run; 5: passes on first attempt with stable edge cases
Visual fidelity	25%	0: broken layout; 5: structure, hierarchy, and color scheme match the target
3D / interaction motion	20%	0: no perspective, numerous occlusion errors; 5: stable frame rate, natural interaction feel
Engineering quality	15%	0: global pollution, no resource release; 5: clear semantic naming, recoverable resources
Cost efficiency	10%	0: still below standard after multiple rounds; 5: meets standard in one round with low token consumption

Functional correctness means not only “the page opens” but also the interaction loop (correct state after press-drag-release) and edge cases (recovery after window resize or tab switching).

Visual fidelity should evaluate “structural consistency” rather than pixel-level identity: given a text description or reference screenshot of the expected result, assess the layout skeleton, visual hierarchy, whitespace proportions, and color logic. Pixel-level replication is not a reasonable expectation for generative models and should not be an evaluation goal.

3D and motion focus on three things: whether perspective is reasonable (near-large far-small, correct occlusion relationships), whether particle or mesh distribution is uniform, and whether interaction has physical feel such as damping and spring-back.

Engineering quality is most easily overlooked: unbound event listeners and uncancelled animation frame requests will evolve into memory leaks in long-lived pages. Low-quality code is extremely expensive to maintain even if it “runs.”

Cost efficiency requires recording token consumption throughout as raw data for the cost accounting in Section 6.

3. Test Case Design: Covering Three Difficulty Tiers

Test cases should cover a difficulty gradient rather than pile up in quantity. Three categories are recommended:

1. Static components (low difficulty): Navigation bars, card lists, responsive grids. Acceptance points: correct breakpoint switching, no unexpected horizontal scrollbars.
2. Dynamic interaction (medium difficulty): Data-driven charts or Canvas 2D animations. Acceptance points: the main loop uses the standard animation frame API rather than timers; pauses when the page is hidden and resumes at normal frame rate.
3. 3D scenes (high difficulty): Implement rotating star rings, solar systems, and similar scenes using browser-native 3D rendering capabilities. Acceptance points: correct projection matrix (no face flipping), correct depth relationships, pointer dragging changes the viewpoint.

A key constraint: each problem should explicitly prohibit external dependencies. Otherwise, “can the model recall the API usage of some third-party library” would contaminate the true measurement target of “spatial understanding and mathematical ability.” Test case cards can be templated as:

{
  "case_id": "fe-3d-001",
  "domain": "frontend_3d",
  "difficulty": "high",
  "prompt": "Using native HTML/CSS/JavaScript, implement an interactive 3D particle star ring: about 800 particles orbiting along a tilted circular track, supporting pointer drag to adjust pitch angle, with damped spring-back on release. Single-file delivery, no external libraries allowed.",
  "acceptance": [
    "Zero console errors",
    "Particle count within the 750–850 range",
    "Main loop driven by requestAnimationFrame",
    "pointerdown / pointermove / pointerup events bound",
    "Animation recovers normally after the page is hidden for 30 seconds"
  ],
  "sampling": 5
}

4. Combining Automated Acceptance with Blind Human Review

The automated portion intercepts low-level failures. An illustrative script follows:

// Structured acceptance executed after loading generated output in a headless browser (illustrative)
async function runAcceptance(page) {
  const checks = [];
  // 1. Zero console errors (collected by listeners during load)
  checks.push({ name: "console_clean", ok: page.consoleErrors.length === 0 });
  // 2. Static feature checks for main loop and event binding
  checks.push({ name: "anim_loop", ok: /requestAnimationFrame/.test(page.source) });
  checks.push({ name: "pointer_events", ok: ["pointerdown", "pointermove", "pointerup"]
    .every((evt) => page.listenerCount(evt) > 0) });
  // 3. Resource release check: residual listeners should be zero after teardown
  await page.teardown();
  checks.push({ name: "listener_cleanup", ok: page.residualListenerCount === 0 });
  return { pass: checks.every((c) => c.ok), checks };
}

The human portion handles judgments that are difficult to automate, such as “does it look good.” Strip identifiers from different models’ outputs and randomly order screenshots, then have three or more reviewers score independently and take the median. Blind review significantly weakens brand priors and order bias from “which one was seen first,” and is a key safeguard for visual evaluation credibility.

5. Quantitative Scoring and Statistical Robustness

• Each dimension is scored 0–5, weighted and summed to obtain a test case total score, then averaged across cases in the same domain to obtain a domain score.
• Each test case should be sampled at least 3–5 times, reporting a “pass rate” (e.g., 4 of 5 runs executable) rather than a single result. Generative output has high variance, and a single sample hardly constitutes evidence.
• Record two metrics simultaneously: pass@1 (passes on first attempt, reflecting out-of-the-box usability) and “passes after correction” (reflecting iteration cost). Models with a large gap between the two suit teams patient enough for multiple rounds of refinement; models with a small gap suit pipelines pursuing one-shot completion.

6. The Cost Perspective: Value Is Not the Same as Low Unit Price

The most commonly overlooked equation in model selection is:

Total cost to meet standard = (first generation + correction rounds) × token unit price

A model with a high unit price may not have higher total cost if its first-pass rate is significantly higher; conversely, a low-price model will see token totals expand rapidly if it requires multiple rounds of correction. Therefore “value for money” must be measured by “total cost to meet standard” rather than unit price.

Another methodological principle that must be emphasized: conclusions such as “low-priced models approach flagship performance on specific tasks” must be limited to a task domain and accompanied by sample counts. Empirically, structured frontend tasks (cards, forms, grids) place relatively moderate demands on model capability, and mid-tier models are often sufficient; while high-difficulty 3D tasks involving projection matrices, normal calculations, and particle dynamics will significantly amplify capability gaps. Any “flagship replacement” judgment that does not limit the task domain and does not perform multiple sampling has no extrapolation value.

7. Four Common Evaluation Pitfalls

1. Problem leakage: Classic examples (such as the standard rotating cube) are highly likely in training corpora. Create your own variants: change particle counts, orbital tilt angles, and color constraints so that “memorizing answers” fails.
2. Survivorship bias from single sampling: The one run that happened to work may just be luck; multiple samples are needed to see the distribution.
3. Looking only at “it runs” while ignoring engineering quality: Memory leaks and global variable pollution are invisible in demos and only erupt in production.
4. Subjectivity in visual review: Fix lighting and environment for screenshots (eliminating GPU and monitor differences), then layer on multi-person blind review to keep subjective judgment within controllable error.

8. Practical Recommendations

• Build an internal benchmark set: 20–50 problems covering three difficulty tiers is sufficient; avoid over-engineering. The value of a benchmark set lies in “stable comparability,” not “big and comprehensive.”
• Unified access reduces engineering cost: Execute the same case set against multiple candidate models through a unified inference interface, masking differences in calling conventions. Enterprises can use model services on cloud platforms such as Baidu AI Cloud for unified access, batch inference, and usage statistics, concentrating evaluation effort on cases and scoring themselves.
• Version regression: Re-run the benchmark after model service upgrades to prevent capability drift from causing production regressions.
• Result archival: Archive outputs, scores, and correction records from each evaluation round to gradually form the team’s reusable knowledge assets.

9. Conclusion

Models iterate quickly, and “which model is strongest” is a question with a very short shelf life; “how to continuously, objectively, and cost-effectively verify whether a model matches my business” is the long-term valuable engineering capability. With a small but accurate internal benchmark, combined with automated acceptance, multi-person blind review, and total-cost-to-standard accounting, any team can make technical decisions that withstand retrospective review under budget constraints.