LLM Coding Evaluation and Cost-Effective Model Selection: A Practical Framework

    Top-of-leaderboard rankings rarely translate into real engineering value. This article proposes a self-built, reproducible evaluation system for measuring LLM coding ability. It breaks coding capability into six dimensions, shows how to build a layered private task set (L1–L4), and describes a sandboxed, reproducible evaluation loop with a fixed six-metric scorecard. Its core contribution is a cost model built around cost per successful task rather than unit price, along with visual and artifact-level evaluation for frontend and 3D tasks, tiered model routing with continuous regression in CI, and the common pitfalls that invalidate comparisons.

    1. The Starting Point: Why “Top of the Leaderboard” Rarely Means “Good in Production”

    When people debate which model writes the best code, the most common mistake is reducing evaluation to a single benchmark run. Public leaderboards differ in their problems, evaluation methods, and context assembly, and they all carry a risk of training-data contamination, so a single score rarely transfers to your own codebase, tech stack, and engineering constraints. More importantly, a model that is cheap or free per call may actually cost more per unit if its task success rate is low and it requires multiple rounds of rework.

    The right question from an engineering perspective is: on your task distribution, what is the cost and end-to-end latency per successful delivery, and is the quality acceptable? Answering it requires a self-built, reproducible, regression-capable evaluation system, not a copy of third-party conclusions.

    2. Breaking Down Capability: Six Dimensions of Coding Model Evaluation

    Turning “can write code” into measurable dimensions is the first cornerstone of the evaluation system.

    1. Correctness and executability. Whether generated code passes predefined unit tests is the highest signal-to-noise metric. It requires the evaluation to be a closed “generate–execute–judge” loop, not a human reading code to judge its quality.

    2. Algorithmic and engineering quality. Beyond passing tests, observe whether complexity degrades (for example, solving in O(n²) a problem that could be solved in O(n log n)), whether exception handling is complete, and whether meaningless duplicate code is introduced.

    3. Long-context repository understanding. In real development, the input is often multiple files, type definitions, and interface contracts. Evaluation must cover capabilities such as “cross-file dependency inference” and “following existing abstractions rather than inventing a parallel set,” which completion-style tests cannot reveal.

    4. Frontend UI generation quality. This includes layout fidelity, responsive breakpoint behavior, component semantics, completeness of accessibility attributes, and rendering consistency of the generated page across mainstream browsers.

    5. 3D and graphics code generation. These tasks require the model to output geometry data, shading logic, or scene-description code, testing its grasp of mathematical consistency across coordinates, normals, transformation matrices, and material parameters. The output can be objectively validated: whether the mesh is closed, whether normal directions are consistent, whether the face count and file size fall within a reasonable range, and whether the multi-view rendered images are self-consistent.

    6. Instruction following and tool calling. In the agent form of coding tools, the model must issue function calls in an agreed format, read the returned results, and self-correct based on errors. The core metrics here are the “number of repair rounds” and the “format compliance rate,” not a single output.

    3. Task Set Design: A Layered Private Evaluation Set

    A self-built task set determines the ceiling of the evaluation. Build it in four layers:

    Level Task type Evaluation method
    L1 Function-level algorithm problems, boundary handling All unit tests pass / fail
    L2 Single-file module implementation (with exception handling and logging) Tests + static checks (complexity, cyclomatic complexity thresholds)
    L3 Cross-file refactoring, interface implementation, bug fixes Regression tests + change size + human review
    L4 End-to-end requirements (pages, small tools, script pipelines) Runnable acceptance + visual/artifact checks + human scoring

    Three design principles are easily overlooked:

    • Tests should be private and dynamic. Public problems are easy to memorize; a private test set combined with randomized parameters (randomized inputs, randomized field names and requirement wording) significantly reduces the impact of “memorizing the answer.”
    • Difficulty should be layered with a fixed ratio. Using one vague aggregate score to mask structural differences leads to the useless conclusion that “they all work” or “none of them work.”
    • Tasks should carry metadata. Tag the tech stack, the number of files involved, and whether a specific output format is required, so that attribution analysis can be done later.

    4. Execution and Metrics: A Reproducible Evaluation Loop

    Stability matters more than coverage. Set the temperature to 0 or very low, fix the context assembly strategy, and fix the timeout and retry limits; otherwise, data across batches is not comparable.

    1. # Illustrative code: an evaluation loop that operates per task, not per completion
    2. def run_one(client, task, cfg):
    3. prompt = render_context(task.spec, task.repo_files, cfg.max_ctx)
    4. usage = {"prompt_tokens": 0, "completion_tokens": 0}
    5. for attempt in range(cfg.max_attempts + 1): # includes repair rounds
    6. t0 = time.perf_counter()
    7. resp = client.chat(prompt, temperature=0.0)
    8. latency = time.perf_counter() - t0
    9. usage["prompt_tokens"] += resp.usage.prompt_tokens
    10. usage["completion_tokens"] += resp.usage.completion_tokens
    11. code = extract_code(resp.text) # structured parsing
    12. verdict = sandbox.run_tests(code, task.tests, timeout=cfg.timeout)
    13. if verdict.passed:
    14. return record(task, True, attempt, latency, usage)
    15. # feed back the compile errors / failing test summary and enter a repair round
    16. prompt = prompt + feedback_block(verdict)
    17. return record(task, False, cfg.max_attempts, latency, usage)

    The execution environment should run in a containerized sandbox: network disabled by default, input mounted read-only on the filesystem, a separate writable directory, and process-level timeouts and resource quotas. Generated code is inherently untrusted and must never be executed directly on a machine that holds credentials.

    The metric system should be fixed at six items:

    1. First-pass rate (passes tests on the first generation)
    2. Pass rate after repair (passes after n rounds of error feedback)
    3. Average number of interaction rounds (measures efficiency in the agent form)
    4. End-to-end latency P50 / P95
    5. Cost per successful task (see the next section)
    6. Regression introduction rate (the proportion of times fixing old problems breaks existing functionality)

    5. Cost Model: From “Unit Price” to “Cost per Successful Task”

    This is the core of cost-effectiveness judgment, and the step that is most often miscalculated. All of the following belong outside the denominator and must be accounted for: retries, context expansion, and the tokens consumed by failed tasks.

    1. Cost per successful task = (total input tokens × input price + total output tokens × output price) / number of successful tasks
    2. Effective throughput = number of successful tasks / total wall-clock time

    A hypothetical scenario illustrates the magnitude of the difference (the numbers are illustrative only, not the pricing of any specific product): Model A has a high per-call price and a first-pass rate of 0.85; Model B’s price is about one-fifth of A’s, with a first-pass rate of 0.55 and an average of 1.4 repair rounds. If B’s input tokens balloon to 1.6× A’s due to long context and retries, then the gap in cost per successful task between the two becomes markedly smaller than the gap in unit price; once you also factor in the rework labor cost of failed tasks, the conclusion may reverse entirely.

    Therefore, the selection conclusion must be jointly determined by “unit price × usage structure ÷ success rate,” not by a unit-price ranking. The evaluation report should output this comparison table directly.

    6. Visual Evaluation of Frontend and 3D Tasks

    These tasks cannot be judged by unit tests alone and require “artifact-level checks”:

    • Rendered screenshot comparison: use a headless browser to capture screenshots at multiple breakpoints (mobile / tablet / desktop), compute perceptual similarity and structural difference against reference images, and output a difference heatmap to aid localization.
    • Structure and accessibility checks: DOM hierarchy depth, proportion of semantic-tag usage, contrast and label association (labels paired with form controls), and the number of console errors.
    • Human scoring rubric: score each of four items — visual fidelity, interaction completeness, code maintainability, and edge-case handling — on a 1–5 scale, with a fixed rubric and cross-sampled scorers to reduce subjective bias.
    • 3D artifact checks: mesh watertightness, normal consistency, number of non-manifold edges, face and vertex counts, number of materials and textures, multi-view rendering consistency, and a file-size cap. All of these metrics can be scripted and are the most reliable objective signal for 3D generation capability.

    7. Putting It into Practice: Tiered Routing and Continuous Regression

    The final output of the evaluation is not a report but a routing policy. A common approach is to build a three-layer structure of “capability tiers — task tiers — routing decisions”:

    1. def route(task):
    2. if task.level in ("L1", "L2") and not task.requires_long_context:
    3. return light_model # cheaper model handles high-frequency, low-risk tasks
    4. return flagship_model # complex refactoring, cross-file, and 3D/graphics tasks

    Routing conditions can be tightened further: return directly on a cache hit; automatically escalate to a stronger model when format validation fails or unit tests fail beyond a threshold. Supporting engineering practices include:

    • Prompt and context caching: put stable repository summaries, type definitions, and coding conventions into reusable context to reduce repeated input tokens. This kind of optimization often has a greater impact on cost than switching models.
    • Evaluation set in CI: run the full set nightly and a smoke set before release, to prevent model or prompt changes from introducing quality regressions.
    • Artifact archiving and observability: store evaluation artifacts, screenshots, and logs uniformly in object storage, schedule tasks asynchronously via a message queue, and feed metrics into a monitoring and alerting system, forming a traceable evaluation pipeline.

    8. Common Pitfalls

    • Looking only at first-pass rate: ignoring repair rounds and token expansion severely distorts cost estimates.
    • Inconsistent prompt engineering: using different context strategies for different models on the same batch of tasks invalidates the comparison.
    • Evaluation set contamination: long-term use of public problems gradually distorts conclusions, so you need to continuously add private and randomized cases.
    • Ignoring security and compliance: generated code may introduce insecure dependencies, hardcoded credentials, or snippets with unclear licensing, so it must be included in static scanning and license checking.
    • Ignoring latency experience: P95 latency directly determines the feel of interactive coding tools and should be evaluated with the same weight as accuracy.

    Conclusion

    The essence of high-quality coding ability evaluation is to translate the subjective “works well” into executable, reproducible, attributable metrics: a layered task set guarantees coverage, a sandbox closed loop guarantees objective judgment, the six metrics guarantee comparability, “cost per successful task” guarantees the economics of selection, and visual and artifact-level checks cover scenarios such as UIs and 3D that are hard to unit-test. Once this system is running, model selection shifts from “reading leaderboards and listening to word of mouth” to an engineering decision based on your own data — which is precisely the basis for judging whether a low-cost model can serve as the main workhorse in real business.