LLM Selection Testing: A Three-Dimensional Evaluation of Coding, Vision, and Cost

    This article presents a reproducible methodology for evaluating large language models across three dimensions—coding, front-end and 3D visual generation, and cost efficiency—instead of relying on leaderboard rankings. It explains why public benchmarks and marketing claims fall short, introduces a four-dimensional evaluation framework, and shows how to use pass@k to distinguish a capability gap from a sampling gap. It also covers objective and subjective front-end scoring, structured validation for 3D scenes, effective-cost modeling that accounts for repair loops and pass rates, a comparison of self-hosting versus pay-as-you-go inference, and a continuous-regression workflow. The goal is to help teams build their own benchmark set so that every model update and price change maps quickly to business cost and product quality.

    Introduction: What an Evaluation Really Needs to Answer Is Never “Who’s the Strongest”

    When new models ship in rapid succession, vendor messaging tends to cluster around three points: overall intelligence rankings, price cuts, and multimodal capability. Phrases like “globally leading aggregate score,” “coding ability close to flagship,” “more polished 3D and UI output,” and “substantially lower per-token pricing” carry a lot of information but do little to guide an actual selection decision. What developers really need to answer is a different question: on their own workloads, which model can consistently produce acceptable results at an acceptable latency and cost?

    That answer can’t be read off a leaderboard or a marketing page. This article takes the coding, front-end, and 3D visual-generation capabilities of mainstream large language models as its subject and proposes a reproducible evaluation methodology covering test-case design, automated scoring, cost estimation, and continuous regression. One caveat up front: the absolute ranking and price of any vendor-promoted model version change over time, so this article does not discuss specific products. It focuses on how to test, how to calculate, and how to use the results.

    1. The Limits of Public Benchmarks and Marketing Data

    Public benchmarks and leaderboards are a good starting point for selection, but they shouldn’t be the finish line, for four main reasons.

    1. Training-data contamination. The cutoff for most model training corpora postdates the release of public benchmarks. Test questions have very likely entered the training data, so the model may be recalling answers rather than genuinely possessing the ability being tested.
    2. Mismatch between question distribution and real workloads. Capability rankings built on general-purpose questions differ markedly from real tasks like “multi-file refactoring,” “turning a design mockup into an interactive page,” or “generating a scene from a text description.”
    3. A single aggregate score hides weak spots. A composite score averages everything out. A high code-generation score does not mean instruction-following, self-correction across multiple turns, or cross-file dependency understanding is equally reliable.
    4. Conclusions have a shelf life. Inference pricing, model versions, and context windows keep changing; a cost-effectiveness conclusion drawn six months ago may have reversed today.

    Recommendation: Use public benchmarks for initial screening and an internal business test set for the final decision. The selection conclusion is only trustworthy when both agree.

    2. The Four-Dimensional Evaluation Framework

    Break the evaluation into capability, cost, and engineering, which yields four dimensions:

    Dimension Core question Key metrics Main evaluation method
    Coding & engineering Can the code compile and pass tests? Compilation pass rate, pass@k, defect rate, static-check findings Automated build and test in an isolated environment
    Front-end generation Is the UI usable and attractive? Element completeness, layout correctness, accessibility conformance items, human aesthetic score Headless-browser rendering, rule checks + pairwise blind comparison
    3D & visual generation Is the scene realistic and sensible? Import success rate, geometry validity, error-free render rate, multi-view consistency Geometry validation, multi-view rendering, human scoring
    Cost efficiency Does it reliably meet targets within budget? Task cost, effective cost, time-to-first-token (TTFT), generation speed Usage tracking, load testing, end-to-end accounting

    Beyond these four dimensions, also pay attention to stability and compatibility:

    • variance in pass rate across repeated runs of the same input;
    • whether outputs honor agreed formats such as JSON Schema or function signatures;
    • the maximum supported context length;
    • compatibility with mainstream interface conventions and how much change migration requires.

    3. Coding: From “Can Write” to “Can Ship”

    Coding is the highest-weighted and most objectively measurable dimension in LLM selection, but testing only whether the model “can produce a correct snippet of code” is nowhere near enough.

    3.1 Three-Tier Test-Case Design

    • L1, function level: Given a function signature and docstring, implement a specific algorithm or data-processing logic. These cases are easy to judge automatically, cover a lot of ground, and reflect baseline coding ability.
    • L2, module level: Produce a module with complete error handling, type annotations, logging, and unit tests, which tests engineering discipline.
    • L3, task level: End-to-end tasks close to real development, such as “add a new endpoint to an existing repository and update the tests and docs accordingly” or “fix an already-located bug without introducing regressions.” Scoring should include engineering dimensions such as control over the scope of changes, whether existing tests were broken, and whether dependencies stayed consistent.

    The three tiers should be weighted according to your business distribution. Too high a share of pure algorithm problems tends to select “test-optimizing” models rather than “engineering-oriented” ones.

    3.2 Using pass@k to Separate “Capability Gaps” from “Sampling Gaps”

    Two behaviors show up often when evaluating coding models: the model frequently errs on a single generation but a few candidate samples yield a correct answer; or it can’t get it right no matter how many times you sample. The first is a sampling-variance problem, which can be eased by raising temperature or increasing the number of candidates. The second is a capability problem that retries cannot fix.

    The industry-standard pass@k means: for each question, sample k candidates independently, and the question counts as passed if at least one candidate passes all unit tests. To reduce the randomness introduced by sampling, an unbiased combinatorial estimator is typically used:

    1. pass@k = 1 - C(n - c, k) / C(n, k)

    Here n is the total number of samples per question, c is the number that pass all tests, and C is the binomial coefficient. In practice you can avoid computing large combinations directly by using the recurrence below:

    1. def pass_at_k(n: int, c: int, k: int) -> float:
    2. """Unbiased pass@k estimate based on combinatorics.
    3. n: total samples per question; c: number passing all tests; k: number of candidates.
    4. Recurrence: prod(n - c - i + 1) / prod(n - i + 1), i from 1 to k.
    5. """
    6. if n - c < k:
    7. return 1.0
    8. ratio = 1.0
    9. for i in range(1, k + 1):
    10. ratio *= (n - c - i + 1) / (n - i + 1)
    11. return 1.0 - ratio
    12. # Example: 10 samples per question, 3 of which pass all tests
    13. # Result is ~0.708, i.e., the probability that at least one of 3 randomly
    14. # chosen candidates is correct is about 70.8%
    15. print(round(pass_at_k(10, 3, 3), 3))

    Comparing pass@1 with pass@5 and pass@10 reveals a model’s “ceiling” and “floor.” A model with a low pass@1 that rises markedly at pass@10 suits scenarios that can add multi-candidate filtering and automated repair; a model that scores low across all k values is not suited to critical-path tasks.

    3.3 Execution Environment and Engineering Gates

    Automatically evaluating coding ability requires a stable, isolated, reproducible execution environment:

    • Pre-installed or offline dependencies. Avoid network fluctuations causing false “pass” or false “fail” results.
    • Time and memory limits. Prevent infinite loops and oversized memory allocations from taking down the evaluation machine.
    • Test timeouts. Record the timeout rate as a stability metric.
    • Uniform test execution. Have the evaluation framework run the test cases, rather than letting the model under test write and run its own tests. This avoids inflated scores from a model “grading its own work.”

    On top of passing tests, layer in engineering gates:

    • static type checking and style/lint checks;
    • security scanning, focused on common defects such as command concatenation, hard-coded secrets, and unsafe deserialization;
    • change-scope checks—in task-level cases, whether the model touched unrelated files on its own;
    • regression checks—whether all pre-existing tests still pass.

    3.4 Typical Coding Workflow and Cost Model

    In AI coding tools, a model often doesn’t “finish in one shot” but runs inside a “generate–execute–observe–repair” loop. This workflow breaks into roughly four steps:

    1. Retrieve relevant snippets from the codebase;
    2. Generate a patch;
    3. Run the build and tests;
    4. Keep correcting based on error messages until it passes or hits the maximum number of rounds.

    This loop directly shapes the cost structure. Let I_t be the input tokens in round t, O_t the output tokens, p_in the input unit price, p_out the output unit price, and E[T] the expected number of rounds to complete a task. The average cost per task is:

    1. C_task = ( E[Σ I_t] * p_in + E[Σ O_t] * p_out ) / 1,000,000

    Note especially: in the repair loop, the input in round t contains not only the original prompt but also all code, logs, and patches from previous rounds. If each round’s context accumulates in roughly equal amounts, the total input tokens grow approximately quadratically with the number of rounds:

    1. Σ(t=1..T) I_t ≈ I_1 * T * (T + 1) / 2

    This means that each additional self-repair round increases cost by more than the round increase alone. When evaluating and configuring for production, record both the average number of rounds and the total token consumption, not just the “per-call unit price.” In addition, some inference services offer context caching or prefix reuse, where reused portions are billed at a lower rate. When reusing, put stable system prompts, repository conventions, and shared dependency notes at the very front of the prompt to maximize hit probability (exact billing follows your platform’s rules).

    4. Front-End Generation: Making “Visual Aesthetics” Measurable

    “Stunning visual aesthetics” is among the most attention-grabbing and most contentious claims in model evaluation. Different reviewers can perceive the same page very differently. To bring visual ability into an evaluation system, subjective impressions must be decomposed into objective, judgeable items.

    4.1 Objective Checks: Rule-Based and Reproducible

    • Element completeness: Whether the rendered page contains the key regions and controls required by the design, such as navigation, hero section, forms, and footer.
    • Layout correctness: Whether the relative position and hierarchy of blocks match the brief, and whether there are element overflows, occlusions, misalignment, or abnormal scrollbars.
    • Responsive behavior: Whether the layout adapts across typical viewports—narrow, standard, and wide—and whether horizontal scrolling appears.
    • Visual consistency: Whether the spacing system is consistent (e.g., follows multiples of 4 or 8), the number and hierarchy of colors are restrained, the type scale is clear, and alignment is clean.
    • Accessibility basics: Whether images have alt text, form controls have labels, text/background contrast meets requirements, and the page works with a keyboard.
    • Interaction and semantics: Whether buttons and links have correct semantics, and whether interactions like form validation, dialogs, and collapsible panels work properly.

    Most of these checks can be automated with a headless browser: render the page, extract the DOM and computed styles, take screenshots, capture console errors, then judge item by item against assertions preset for the case. Assertions can be written by evaluators based on the design intent, or a baseline can first be generated from a reference implementation and the output compared against it.

    4.2 Subjective Checks: Pairwise Comparison and Model Review

    Aesthetic judgment can’t be fully rule-based, so two complementary methods are recommended:

    • Pairwise comparison. Reviewers look only at two versions, A and B, and judge which is better, without giving an absolute score. This is more stable than direct scoring, and results can be aggregated into a relative ranking using a pairwise win/loss rating system (such as Elo or the Bradley-Terry model).
    • Multi-reviewer aggregation. Have at least three reviewers score independently, drop the extremes and take the mean, and report a consistency metric. Samples where reviewers disagree should be reviewed separately; they often reveal ambiguities in the case description.

    If you bring in automated model review, follow these principles:

    • The reviewing model should be different from the model under test, to reduce same-source bias;
    • Give the reviewing model explicit scoring criteria and examples;
    • Periodically sample-check by hand to calibrate the reviewing model against human judgment.

    4.3 Guarding Against Gaming in Front-End Evaluation

    Front-end generation evaluation is especially prone to results that “score high but are actually unusable.” Common failure points include:

    • the page chases only static appearance while interactions don’t respond;
    • heavy use of placeholder content, so the layout breaks once real data is connected;
    • directly copying a well-known site’s visual style while ignoring brand and compliance requirements.

    So it’s advisable to provide two kinds of cases: first, “design-reproduction” cases that give a layout description or even a reference image to assess reproduction ability; second, “product-requirement” cases that give only business context and target users to assess autonomous design and information organization. Also embed verifiable specifics in the cases—particular copy, color schemes, or block counts—so a model can’t score well just by applying a template.

    5. 3D and Visual Generation: Realism Comes from Structure

    Generating 3D scenes from text is a focus area for multimodal models in recent years. Evaluating this ability should move from “looks like it” to “structurally sound and usable.”

    5.1 Evaluation Pipeline

    Run structured checks on the generated results in this order:

    1. text description ──> scene generation ──> import validation (scene file parsing, dependency-resource integrity)
    2. ──> geometry validation (non-manifold edges, flipped normals, self-intersections, dangling vertices)
    3. ──> render checks (multi-camera rendering, console/render logs)
    4. ──> view-consistency comparison (entity list vs. rendered screenshots)
    5. ──> human scoring (realism, material plausibility, compositional aesthetics)

    5.2 Key Checks

    • Importability: Whether the generated scene file can be read properly by general 3D tools or engines, and whether external resources such as textures and materials are complete.
    • Geometry validity: Whether common mesh problems are present, such as non-manifold edges, incorrect normal orientation, self-intersecting faces, and dangling vertices.
    • Render stability: Whether rendering from multiple camera positions and under varied lighting produces broken faces, black screens, missing materials, or abnormal flicker.
    • View consistency: Extract an “entity list” from the text description—object types, counts, relative positions, and material attributes—and compare it item by item against rendered screenshots. The key to realism is not the polish of a single frame but whether objects remain present, positions stay stable, and occlusion relationships are correct across different angles.
    • Physical common sense: Whether gravity direction, support relationships, scale/proportions, and light direction are reasonable.

    5.3 Applying the Method Elsewhere

    This approach transfers to evaluating other forms of graphical content:

    • For diagram generation, check whether diagram elements correspond one-to-one with text labels and whether arrows and connectors are correct;
    • For video or motion generation, extract keyframes to check temporal coherence and visual stability.

    The core principle is a single one: first use structured validation to filter out “unusable” results, then use human review to distinguish good from better among the usable ones.

    6. Cost Efficiency: Getting “Cheap” Down to the Business Level

    Price cuts are the most eye-catching part of vendor messaging, but unit price is only one factor in cost. What really drives the budget is the cost per unit of effective output.

    6.1 Cost Components

    The base cost of a single inference call can be written as:

    1. C_call = (T_in * p_in + T_out * p_out) / 1,000,000

    where T_in and T_out are input and output tokens, and p_in and p_out are the corresponding unit prices (per 1M tokens). In real business, three factors further adjust this:

    1. Retries and repair. Multiply task-level cost by the expected number of rounds; see Section 3.4.
    2. Cache reuse. The input portion that hits the cache is billed at a lower price.
    3. Hidden costs. Human intervention, waiting/delay, and rework caused by failed tasks.

    A more meaningful metric is therefore effective cost:

    1. C_eff = C_task / P_task

    where P_task is the first-pass task success rate. Two models with unit prices 5× apart don’t necessarily differ 5× in business cost; if the cheaper model needs more repair rounds, the final gap may be far smaller than 5×, or even reverse.

    6.2 A Reproducible Estimation Method

    1. Using the same batch of internal cases, tally the input tokens, output tokens, and rounds each model needs to complete a task;
    2. Estimate expected cost given the retry strategy;
    3. Plug in the first-pass success rate to get effective cost;
    4. Convert to a monthly budget based on the business’s daily call volume.
    1. from dataclasses import dataclass
    2. @dataclass
    3. class ModelStat:
    4. name: str # model codename, for internal evaluation only
    5. avg_in: float # average input tokens per task (including repair loop)
    6. avg_out: float # average output tokens per task
    7. p_in: float # input price (per 1M tokens)
    8. p_out: float # output price (per 1M tokens)
    9. p_task: float # first-pass task success rate
    10. def effective_cost(s: ModelStat, daily_tasks: int) -> dict:
    11. cost = (s.avg_in * s.p_in + s.avg_out * s.p_out) / 1_000_000
    12. eff = cost / s.p_task if s.p_task > 0 else float("inf")
    13. return {
    14. "model": s.name,
    15. "cost_per_task": round(cost, 4),
    16. "effective_cost": round(eff, 4),
    17. "monthly_budget": round(eff * daily_tasks * 30, 2),
    18. }
    19. # All values below are illustrative and do not represent any real product's pricing;
    20. # actual values should come from your own business testing
    21. stats = [
    22. ModelStat("M1", avg_in=12000, avg_out=3000, p_in=4, p_out=12, p_task=0.90),
    23. ModelStat("M2", avg_in=15000, avg_out=4000, p_in=2, p_out=8, p_task=0.75),
    24. ]
    25. for s in stats:
    26. print(effective_cost(s, daily_tasks=10000))

    As the logic shows, the pass rate sits between per-task cost and effective cost as an “amplifier.” A model with a lower unit price but a noticeably lower pass rate is not necessarily the more economical choice.

    6.3 The Cost Angle on Performance and Stability

    Cost also shows up on the time axis, and two metrics need attention at once:

    • Time to first token (TTFT): Determines how quickly the first token appears in an interactive coding assistant, directly affecting perceived responsiveness;
    • Generation speed: The token throughput during output, which determines how soon a long piece of code is fully rendered.

    Also examine how much latency and throughput degrade under different concurrency loads. When responses slow noticeably at peak, teams are forced into more retries or larger buffers, indirectly pushing cost up.

    6.4 Self-Hosting vs. Pay-As-You-Go

    For open-source or open-weight models, self-hosting is another cost path. It avoids pay-as-you-go billing but takes on compute, ops, and headcount costs, which can be estimated roughly:

    1. required GPU-days = daily tokens processed / (per-GPU throughput × 86400 × target utilization)
    2. monthly compute cost ≈ required GPU-days × 30 × per-GPU daily rent (or depreciation cost)

    For example (the numbers below are illustrative only):

    • A team needs to process 100 million output tokens per day;
    • A single GPU’s output throughput is about 1000 tokens/s;
    • Target utilization is set to 40%.

    Then each day needs about 2.9 GPUs of capacity (1e8 ÷ (1000 × 86400 × 0.4) ≈ 2.89), which with redundancy roughly rounds to 3–4 cards. If a single GPU costs 10,000 per month, the monthly compute cost is about 30,000–40,000, which works out to roughly 10–13 per million output tokens.

    This estimate excludes ops headcount, storage, network, model updates, and elastic scaling. When comparing it with a pay-as-you-go unit price, watch that the units match: the former targets output tokens, while the latter must be converted to the same basis using the business’s actual input/output ratio.

    In practice, teams with low and volatile call volume and fast-changing model versions are better suited to pay-as-you-go; teams with stable, large-scale volume, a hard requirement that data not leave their environment, or mature inference-ops capability can focus on evaluating self-hosting, ideally with a hybrid strategy of “self-host core scenarios, pay-as-you-go for elastic traffic.”

    7. Continuous Evaluation: Keeping Selection Decisions Valid Over Time

    7.1 Build an Internal Benchmark Set

    The internal benchmark set is the core asset of the evaluation system and should have these traits:

    • Business-relevant: Cases come from real requirements and are organized by business module;
    • Difficulty-tiered: Covering L1–L3, with an expected difficulty distribution;
    • Private and controlled: Rotated regularly and refreshed with new cases to lower the risk of being targeted by optimization;
    • Automatically judgeable: Every case comes with an automated judging script or explicit human scoring criteria.

    7.2 Wire It into Continuous Integration

    Connect the benchmark set to a CI pipeline so that changes to model versions, prompts, or inference parameters automatically trigger a regression run and produce a report containing each dimension’s score, the diff versus the previous version, cost changes, and typical cases (including failures). This lets teams manage a “model quality baseline” much like they manage code.

    Illustrative flow:

    1. code / prompt / model version change
    2. │
    3. ▼
    4. trigger regression ──> run internal benchmark set (coding / front-end / 3D / instruction-following)
    5. │
    6. ├──> automated judging: compilation, unit tests, rendering, rule checks
    7. ├──> human/model review: aesthetics, realism
    8. └──> cost accounting: token usage, rounds, latency
    9. │
    10. ▼
    11. generate comparison report ──> block release if any metric falls below baseline

    7.3 Fix Parameters and Control Randomness

    Before evaluating, fix parameters such as temperature, maximum output length, and system prompt, and record the full model version identifier. Sampling parameters introduce randomness, so sample each case multiple times and report the mean and variance rather than drawing conclusions from a single run.

    7.4 Gradual Rollout and Production Feedback

    When a new model goes live, roll it out to a small share of traffic first and run it in parallel with the current model, comparing key metrics such as task success rate, average repair rounds, user adoption rate, and latency. At the same time, route anonymized production failure samples back into the internal benchmark set, forming a closed loop of “evaluate → release → feedback → re-evaluate.”

    8. Common Pitfalls and How to Avoid Them

    • Generalizing from a narrow sample: Testing only algorithm problems and then concluding the model fits all coding scenarios. Weight cases according to your business distribution.
    • One-shot verdicts: Letting a single evaluation decide a full year of technology selection. Build a regular regression mechanism.
    • Unit price only: Ignoring repair rounds, retries, and human fallback costs. Use effective cost as the decision basis.
    • Capability over engineering: Overlooking context limits, output-length limits, interface compatibility, and rate-limiting policies, causing rework at integration time.
    • Ignoring compliance: For scenarios involving sensitive data, prioritize evaluating on-premises deployment or data-stays-in-region options.
    • Subjective scoring: Vague criteria and too few reviewers. Provide explicit scoring criteria, add more reviewers, and report consistency.

    Conclusion

    A systematic evaluation across coding, front-end visuals, 3D generation, and cost efficiency comes closer to the business truth than any single leaderboard ranking. For most teams, rather than chasing the “strongest model,” it’s better to build your own benchmark set and regression process so that every model iteration and every price change maps quickly and accurately to changes in business cost and product quality.

    For rollout, a three-step approach is recommended:

    1. Run a first comparative evaluation with 20–50 business-relevant cases;
    2. Bring the passing candidate models into a gradual-rollout process for small-traffic validation;
    3. Add the benchmark set to continuous integration for long-term tracking.

    At the platform level, an enterprise can use a mature cloud platform such as Baidu AI Cloud to select appropriate large-model services according to business needs, and combine them with general cloud capabilities such as object storage, logging, and monitoring/alerting to build a complete pipeline from evaluation-data management to online quality monitoring. Once evaluation becomes part of everyday R&D, model selection is no longer a one-time decision at release season but a continuously controllable engineering practice.