Benchmarking LLM Coding Skills in Practice: A 3D and Front-End Visual Test Suite

Why Build a Custom Coding Benchmark

Model vendors often claim “significant coding improvements” or “industry-leading cost performance.” However, for development teams, the core question is: Which model can reliably deliver usable code at an acceptable cost for our specific task distribution?

Public leaderboards rarely answer this directly for three reasons:

1. Distribution bias: Benchmarks typically focus on algorithm problems and function-level completion, while enterprise use cases revolve around pages, components, scripts, and data visualization—engineering-level tasks with different capability profiles.
2. Contamination risk: Training corpora may contain content highly similar to public evaluation sets, inflating scores.
3. Missing dimensions: Leaderboards rarely report metrics like first-pass success rate, average repair rounds, or effective unit cost—the numbers engineering teams actually care about.

Spending half a day building a small, precise internal benchmark before model selection translates marketing claims into quantifiable data relevant to your own context. This article uses two high-discrimination task types—3D scene generation and front-end visual reproduction—to demonstrate a complete methodology.

Why 3D and Front-End Visual Tasks Are Effective Test Cases

Broad Capability Coverage

A single 3D or complex front-end generation task simultaneously evaluates multiple underlying model abilities:

• Spatial and mathematical skills: 3D coordinates, transformation matrices, lighting and material parameters, particle motion equations—any computational error becomes immediately visible;
• API proficiency: Whether the model correctly calls rendering library or front-end framework APIs and organizes component lifecycles directly determines if code runs;
• Engineering structure: High-quality output should have clear modules and sensible state management rather than all logic piled into one file;
• Aesthetics and details: Spacing, color palettes, whitespace, animation curves—these “soft” metrics matter most in real products.

Verifiable and Blindly Scorable Results

Algorithm problems require writing test cases, but 3D and front-end tasks produce a screenshot-able page. Reviewers don’t need to read code line by line; a screenshot and a few seconds of interaction suffice for stable judgment, making them ideal for multi-person blind evaluation. Screenshots also serve as the minimal unit for automated acceptance testing.

High Discrimination

These tasks amplify capability differences effectively: weaker models either fail outright or render visibly distorted scenes, while capable models generate interactive, detailed output in one pass. Scores don’t cluster in a narrow “usable enough” range—selection conclusions become clearer.

Test Case Design: Layered, Controllable, Reproducible

Three-Level Difficulty Structure

Organize cases into an entry–intermediate–advanced gradient:

• Entry level (static presentation): E.g., “Implement a self-rotating geometric shape with basic lighting using browser-native 3D rendering capabilities”—assesses capability floor and runnability;
• Intermediate level (interactive control): E.g., “Simulate a starfield with a particle system, supporting mouse drag rotation and scroll-wheel zoom”—assesses interactive animation and event handling;
• Advanced level (comprehensive and data-driven): E.g., “Render a 3D bar chart from JSON data with transition animations and hover highlighting,” or “Generate a complete landing page from a text-based design spec, including responsive layout and dark/light themes”—assesses requirement decomposition and satisfying multiple constraints simultaneously.

Prompt Standardization

Evaluation validity depends on variable control. Four principles are recommended:

1. Unified prompt template: Requirement description, constraints (single-file delivery, no external assets, modern browser compatibility), and acceptance criteria—three sections;
2. Unified inference parameters: Temperature and maximum output length stay consistent across all models under test;
3. Multiple sampling: Each case is independently sampled 3–5 times per model, averaging results to suppress randomness;
4. Full traceability: System prompts, each feedback round, and retry counts are all recorded for reproducibility.

Scoring Dimensions: Decomposing “Good Looking” into Addable Numbers

A multi-dimensional weighted score avoids subjective “overall impression” judgments:

Dimension	Score	Assessment Criteria
Runnable	0–3	No errors; clean console; interactions work
Visual fidelity	0–5	Layout, color, detail, and animation alignment with requirements
Functional completeness	0–5	Check against requirement checklist, deduct per missing item
Code quality	0–3	Structural clarity, maintainability, redundancy level
Instruction adherence	0–2	Compliance with explicit constraints (single file, no external assets, etc.)

Beyond dimension scores, track two engineering metrics:

• First-pass success rate: Proportion of runs meeting acceptance criteria without any human intervention;
• Average repair rounds: Number of feedback iterations needed after a failure.

Use blind evaluation during scoring: hide model names, shuffle order, and have 2–3 reviewers score independently, then average results to avoid brand bias.

Engineering the Evaluation Pipeline

Manually opening pages one by one doesn’t scale. Automate the generate–run–screenshot–archive cycle. Core loop pseudocode:

for model in candidate_models:
    for case in benchmark_suite:
        for run in range(sampling_count):
            # 1. Generate code via a standard chat-completion style API
            code = chat(model, case.prompt, temperature=fixed_value)
            # 2. Write output to a single-file workspace
            write_workspace(case.id, run, code)
            # 3. Load page in headless browser: capture screenshot + console errors
            shot, errors = capture(case.entry_file)
            # 4. Archive screenshot, code, errors, and token usage
            archive(case.id, model.id, run, shot, errors, usage)

Additional recommendations:

• Environment isolation: Use clean container environments per run to avoid state interference;
• Artifact archiving: Store screenshots and code in object storage; write scoring records to a database or spreadsheet service for quarterly regression comparison;
• Cost tracking: Log input/output token counts per call to support cost calculations;
• Cloud execution: Batch workloads fit naturally in the cloud—e.g., using GPU compute to issue inference calls in parallel, object storage for artifacts, and logging and monitoring services to track pipeline health.

Cost-Effectiveness Calculation: Turning “Good Enough” and “Worth It” into Arithmetic

From Sticker Price to Effective Cost

Single-task cost ≈ input token price × input token count + output token price × output token count. But real spending must incorporate first-pass success rate:

Effective cost = single-run cost ÷ first-pass success rate

Illustrative example: Model A costs 0.2 CNY per run with a 40% pass rate; Model B costs 0.5 CNY with an 80% pass rate. Effective costs are 0.5 CNY for A and approximately 0.63 CNY for B—a 2.5× sticker price gap narrows substantially. Adding input tokens from repair rounds and human feedback time, a “5× price difference” can shrink to under 2× in effective-cost terms.

Cost-Effectiveness Index and Scenario-Based Selection

For horizontal comparison, define: Cost-effectiveness index = weighted capability score ÷ effective cost. This index must be computed separately per task family. For visually verifiable tasks like 3D and front-end generation, some mid-tier models may score several times higher than flagship models. For complex architecture design or long-chain refactoring, however, flagship advantages often remain irreplaceable. Claims like “a fraction of the price for flagship performance” only merit trust after you reproduce them on your own benchmark—that is the entire point of building one.

Traffic Routing Strategy

• High-difficulty, high-value tasks: use flagship models to ensure pass rates;
• High-frequency, routine tasks: use high cost-effectiveness models for volume;
• Route via an unified gateway with automatic fallback on failures.

Five Common Mistakes in Interpreting Results

1. A single impressive sample ≠ overall capability: Demo cases often survive multiple rounds of selection; evaluate distributions, not extremes;
2. Single sampling ≠ stable performance: Output variance for the same prompt can be large; always sample multiple times;
3. Visual appeal ≠ correctness: A stunning scene may hide physics simulation errors, memory leaks, or frame-rate issues—check performance metrics explicitly;
4. Benchmark contamination: Rotate materials and wording periodically to prevent models from having memorized similar public cases;
5. Static scores ≠ production experience: Also assess multi-round repair context handling, long-document stability, and error recovery behavior.

Action Checklist

• Build an internal benchmark set: 10–30 business-relevant cases is enough to start, covering core task families;
• Automate the pipeline: Script generation, execution, screenshot, and archiving end-to-end, with version control;
• Quarterly regression: Models and versions iterate frequently; schedule benchmark regressions to track capability drift and price changes;
• Dual cost tracking: Record both “sticker price cost” and “effective cost including retries” for future selection and negotiation;
• Conservative rollout: Route new models to low-risk tasks first, observe two weeks of online metrics, then gradually scale up.

Model selection is ultimately about letting data replace marketing language. A self-built benchmark on the order of a hundred lines of scripting can reduce claims like “flagship-level capability” or “a fraction of the price” to actual scores, pass rates, and effective costs on your own task distribution—which is what truly reliable model selection depends on.