From Leaderboards to Production: A Practical Guide to Evaluating LLM Coding and Visual Capabilities

1. Why Leaderboards Are No Longer Enough

In the past, evaluating whether a model was “powerful” typically meant checking general intelligence indices or human preference voting leaderboards. This approach was sufficient during the era dominated by pre-training: parameter scale and data volume determined the capability ceiling, and leaderboard gradients generally matched real-world experience. However, as post-training—especially reinforcement learning (RL)—becomes the primary iteration method, three substantive changes have occurred.

First, iteration cycles have drastically shortened. Pre-training often takes months, while RL using verifiable tasks as reward signals can complete a round of refinement in just weeks to two or three months. The density of model versions is rising rapidly, making leaderboard updates informational noise—today’s number one might be surpassed next month.

Second, the risk of benchmark overfitting has intensified. When the optimization goal is explicitly to “score high on public benchmarks,” the boundary between training data and evaluation data blurs. Rising scores do not necessarily equate to general capability improvements, let alone improvements in your specific business scenarios.

Third, capability profiles are highly uneven. The same model might approach top-tier performance in algorithmic reasoning but perform mediocrely in frontend visual aesthetics or long-chain tool calling, and vice versa. A single comprehensive index cannot capture these “specializations,” yet these biases determine the model’s true value to your team.

The conclusion is straightforward: use leaderboards only for initial screening, and return to your own task distribution for final selection. This is the core theme of this article—building a reusable evaluation system focused on two high-frequency scenarios: coding and frontend/3D visual capabilities.

2. Understanding the Capability Source: RL “Refinement”

To design an evaluation, you must first understand the evaluated object. The current mainstream LLM training pipeline can be summarized in three stages:

Pre-training (Massive corpus, acquiring language and world knowledge)
   → Supervised Fine-Tuning SFT (Instruction following, output format normalization)
   → Reinforcement Learning RL (Policy optimization towards verifiable goals)

The third stage is key to the capability leaps seen in the past two years. The core idea is to provide an automatically verifiable reward signal for the model’s output, allowing it to repeatedly trial-and-error and adjust its strategy towards higher rewards. Code is the perfect example—whether a generated program passes test cases can be judged by a machine at zero cost, providing a naturally clean reward signal.

This explains two industry phenomena: the growth rate of coding and reasoning capabilities is significantly higher than other skills because their rewards are easiest to scale; whereas capabilities like visual aesthetics and interactive experience, which are “hard to programmatically judge,” rely more heavily on the quality of human preference data, leading to more obvious differentiation among models.

The implication for evaluation is: do not assume capabilities are uniform. Evaluation dimensions must correspond one-to-one with your business scenario’s capability profile. Furthermore, the gap between open-weights and closed-source models continues to narrow, offering teams a new “self-hosting” option. However, self-hosting requires additional evaluation of inference costs, throughput, and operational complexity, which should also be included in the practical evaluation scope.

3. Coding Capability Evaluation Framework

3.1 Task Design: Three Essential Levels

Level	Task Type	Evaluation Focus
L1	Single-file algorithmic problems	Basic reasoning and code correctness
L2	Multi-file engineering tasks	Context understanding, dependency handling, code style consistency
L3	Agent-style tasks	Multi-turn planning, tool calling, failure recovery

Testing only L1 will severely overestimate productivity—a high pass rate on single-file problems does not mean the model can understand context or follow engineering standards in a real repository. L2 tasks should be anonymized and extracted from your own codebase, while L3 should simulate the complete loop of “read requirements → modify code → run tests → fix errors.”

3.2 Metric Design

• pass@1: Single-sample pass rate, reflecting “out-of-the-box” readiness;
• pass@k: Probability of at least one pass in multiple samples, reflecting “will it succeed if we try a few times”;
• rounds-to-green: Number of rounds of human feedback needed to become usable, reflecting collaboration cost;
• token efficiency: Tokens consumed per accepted line of code, reflecting the economics of long-context tasks.

3.3 Reference Implementation

import math
from dataclasses import dataclass
@dataclass
class CodeTask:
    task_id: str
    prompt: str
    test_cases: list          # [(input, expected_output), ...]
def evaluate_task(model, task: CodeTask, n: int = 5) -> list[bool]:
    """Sample n times for the same task, recording whether all test cases pass each time.
    The sampling temperature should match production settings to avoid disconnects 
    between evaluation and online behavior."""
    return [
        run_in_sandbox(model.generate(task.prompt, temperature=0.2),
                       task.test_cases)
        for _ in range(n)
    ]
def pass_at_k(results: list[bool], k: int) -> float:
    """Standard pass@k: the probability of at least one success in k samples from n trials."""
    n, c = len(results), sum(results)
    if n - c < k:
        return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)

Two engineering reminders: run_in_sandbox must ensure container isolation, resource limits, and timeout control to prevent generated code from damaging the evaluation machine. The evaluation dataset itself must be version-controlled to prevent “evaluation set drift” from making cross-version conclusions incomparable.

4. Frontend and 3D Visual Capability Evaluation

4.1 Why Visual Aesthetics Are Hard to Evaluate

Code can be objectively judged via test cases, but there is no unit test for “whether a page looks good.” Differences in aesthetic capability can often only be measured through structured human judgment—meaning unregulated evaluations will introduce significant subjective noise.

4.2 Scoring Rubric: Breaking Down “Aesthetics” into Verifiable Dimensions

Dimension	Key Judging Points	Weight
Layout & Hierarchy	Whether the visual flow matches information priority	25%
Spacing & Alignment	Whether the grid is self-consistent and whitespace is even	20%
Color Palette	Contrast, readability, color scheme consistency	20%
Component Consistency	Whether similar elements share the same structure	15%
Responsiveness	Whether the layout collapses at breakpoints	10%
Usability Details	Click states, focus states, empty states	10%

4.3 Blind Testing Process

1. De-branding: Uniformly remove comments and characteristic traces from different models’ outputs, and assign random numbers;
2. Independent Multi-person Scoring: 3 to 5 reviewers first conduct a calibration round on sample outputs to unify the judging scale for each dimension;
3. Consistency Check: Calculate the inter-rater reliability coefficient. A low score indicates that the rubric dimensions are still vague and need further refinement;
4. Record Distribution, Not Mean: A model that consistently outputs a 70-point page has entirely different engineering value than one that “sometimes scores 95 and sometimes 40”—the “variance” of aesthetics is itself decision-making information.

4.4 Fidelity and 3D Scenarios

• Design Mockup to Code: Compare screenshots of generated pages with original mockups region by region. Pixel differences and layout tree differences can be programmatically calculated as objective supplements for fidelity;
• 3D Scene Generation: Focus on structural correctness (normal directions, texture coordinates), runtime performance (polygon count, frame rate), and interaction completeness (camera controls, event responses). The first two can be automatically detected, while the latter requires a manual checklist review.

5. Cost Calculation: Turning “Cost-Performance” into a Computable Number

Inference prices in the industry are dropping rapidly; “cheap unit pricing” is no longer rare. However, unit price does not equal Total Cost of Ownership (TCO). It is recommended to standardize on Cost Per Successful Task as the metric:

Cost Per Successful Task = (Expected Call Count × Token Cost Per Call) / Task Success Rate

def cost_per_successful_task(price_in_per_m, price_out_per_m,
                             avg_in, avg_out,
                             success_rate, avg_retries) -> float:
    """Convert unit price, average length, retry count, and success rate 
    into a single comparable metric."""
    expected_calls = 1 + avg_retries
    per_call = (avg_in * price_in_per_m + avg_out * price_out_per_m) / 1e6
    return expected_calls * per_call / max(success_rate, 1e-6)

Three easily overlooked factors:

1. Hidden Costs of Failed Retries: A cheap model with a low success rate might end up with a higher cost per successful task than a flagship model with a high success rate;
2. Context & Caching: In long-context tasks, input tokens dominate. Whether requests hit the cache can impact costs by multiples, so real hit rates should be recorded during evaluation;
3. Routing Strategies: A common practice is to route based on task difficulty—boilerplate code goes to lightweight models, while engineering tasks and visual drafts go to flagship models. A sound routing strategy can often compress overall costs to a fraction of a “full flagship” approach. This is the engineering path to “getting flagship-level experiences at lower prices,” rather than just marketing rhetoric.

6. Productivity Boundaries: What to Delegate to Models, What to Keep In-House

After evaluation, a more important output is a “boundary map”:

• Safe to delegate to models: Scaffolding and boilerplate code, one-off prototypes, batch style migrations, visual drafts, and test case generation;
• Human-machine collaboration: Cross-module refactoring, exploration and comparison of architectural solutions, and security-sensitive code—models draft, humans review line by line;
• Not yet suitable for full automation: Domain-specific tasks lacking verifiers, and long-chain unattended Agent flows—the error recovery cost might exceed the benefits.

Recommended rollout pace: small-scale pilot (2-4 weeks, fixed evaluation set) → establish regression baseline → gradual rollout → real-world load data flows back to update the evaluation set. The evaluation set should evolve in sync with the business; otherwise, after three months, it will degrade into “yet another overfitted leaderboard.”

7. Infrastructure: Making Evaluation a Pipeline, Not a Movement

One-off evaluations are easy; continuous evaluation is hard. Engineered approaches include:

• Evaluation as Code: Tasks, test cases, and scoring rubrics are all placed under version control, making changes traceable and reviewable;
• CI Integration: Automatically trigger regression tests when model versions or prompts change, preventing silent capability degradation;
• Compute & Operations: When building your own evaluation or self-hosted inference pipelines, leverage general cloud capabilities—such as container services, GPU compute, and log monitoring/alerting provided by platforms like Baidu AI Cloud. Containerizing the evaluation pipeline allows on-demand scaling, eliminating the cost of maintaining a resident cluster;
• Result Visualization: Output capability radar charts and cost curves for each evaluation, ensuring every selection decision is evidence-based.

Conclusion

Leaderboards answer “who is gaining points,” but practical evaluation answers “who is doing the work for you.” As model iteration enters the fast lane driven by reinforcement learning, coding, visual, and cost curves will evolve independently, and capability differentiation will only intensify. Aligning evaluation dimensions with business distributions, breaking aesthetics down into verifiable scoring rubrics, and converting cost-performance into cost per successful task—once these three things are done, model selection transforms from a “matter of faith” into an “engineering problem.”