How to Evaluate Large Language Models for Coding and Visual Generation: A Practical Guide
A practical, three-layer framework for evaluating large language models (LLMs) on coding and visual generation tasks. It breaks these abilities into measurable sub-dimensions, explains how to build a private task set instead of relying on public leaderboards, and combines automated checks, LLM-as-judge scoring, and human review into a repeatable pipeline. It also covers pass@k sampling, cost-per-effective-task metrics, and security risks—so you can choose models that are good enough and predictable under your own business constraints.
1. Background: Models Iterate Fast, So Selection Can’t Rely on Leaderboards
LLM development is accelerating. Leading vendors now ship new versions—or lighter-weight variants—on a scale measured in weeks or even days. These releases usually bring longer context windows, stronger reasoning and tool use, and some vendors keep improving real-task performance through multi-round reinforcement learning. At the same time, API pricing across mainstream model platforms has generally been falling, with some new-generation models priced at roughly one-third of their predecessors.
For developers and enterprises, this is both an opportunity and a new challenge: versions turn over quickly and price gaps are wide—should you migrate? Many teams base the decision on leaderboard rank or parameter count, but there’s often a clear “migration gap” between generic benchmarks and actual business workloads.
1.1 Why Top-Ranked Models Aren’t Necessarily Right for Your Business
Take frontend page generation as an example. For the same request—“build a data dashboard page”—different models may all produce code that runs, but the differences show up in the details: whether grid spacing is consistent, whether component hierarchy is sensible, and whether empty and error states are handled thoroughly. Generic scores rarely capture these details.
Coding tasks are the same. Real development work involves a lot of cross-file changes, adherence to existing architecture conventions, and compatibility with specific framework versions—constraints that seldom appear in public evaluation sets. So a more useful question isn’t “which is the strongest model?” but “under a given cost, which model is stable, reliable, and maintainable on my tasks?”
1.2 Evaluation Approach: Build in Layers, Close the Loop
This guide recommends organizing evaluation into three layers that progressively narrow toward real business:
- Layer 1 — General capability screening: Use public code and multimodal benchmarks for quick horizontal comparison to rule out candidates that clearly fall short.
- Layer 2 — Validation on your own task set: Draw tasks from real business, score them with quantifiable rules plus human review. This is the main basis for selection.
- Layer 3 — Production canary monitoring: After launch, continuously track online metrics such as adoption rate and rework rate to catch regressions or version drift, then feed the results back as new evaluation samples.
2. Capability Breakdown: Split “Writing Code” and “Building UIs” into Measurable Sub-Items
Saying vaguely that a model has “strong coding skills” or “good taste” can’t guide decisions. You first need to break these into observable, scorable sub-items.
2.1 Six Dimensions of Coding Ability
| Sub-dimension | Typical task | How to judge |
|---|---|---|
| Code correctness | Implement a function or interface per requirements | Unit tests and type checks all pass |
| Multi-file engineering changes | Add a feature or fix a bug in an existing project | Project builds, tests pass, change scope is reasonable |
| Instruction following | Implement to a specified framework, style, and interface contract | Dependencies and signatures match; no unauthorized new libraries |
| Error diagnosis and repair | Given a stack trace, locate the root cause | Reproduction case passes after the fix; no regressions introduced |
| Reasoning and planning | Complex algorithms, solution design, and trade-offs | Human review of the reasoning chain and edge-case handling |
| Security and conformance | Handle user input, enforce permission checks | Whether flaws like injection, privilege escalation, or hard-coded secrets are introduced |
Engineering tasks deserve particular attention. In a real repository, whether a model can first read the relevant files and then make a localized, accurate change often reflects practical value better than “how nicely it solves a single problem.”
2.2 Four Dimensions of Visual and Frontend Generation Ability
These abilities can likewise be broken into four observable dimensions:
- Layout fidelity: Whether page structure, module hierarchy, and relative positioning match the requirement description or design. For layout-generation tasks, if the sidebar, top navigation, and main content area are misplaced or missing in the output, fidelity is insufficient.
- Visual quality: Includes whether spacing is systematic (for example, consistently using multiples of 4), whether typographic hierarchy is clear, whether colors are harmonious, and whether the visual focus stands out. Take a data card: if the number size, label color, and whitespace are poorly handled, the result can look poor even when functionally complete.
- Interaction completeness and accessibility: Whether form validation, loading and empty states, and error messages are all present; whether semantic tags, focus order, and text-to-background contrast meet accessibility requirements (for example, contrast of at least 4.5:1 for normal text). For public-facing systems, accessibility is often a hard compliance requirement.
- Code quality: Whether the output is clearly structured, has reusable components, contains no redundant inline styles, and follows the project’s existing design conventions and component patterns.
For 3D content generation, the evaluation should focus on “usability”: can the model accurately understand the text description and output renderable meshes, materials, or scene structures (such as the common glTF format)? Viewpoint, lighting, and material appearance are only bonus points. Whether the model can be imported into mainstream rendering engines and whether assets are complete and undamaged are the baseline for engineering deployment.
3. Method: Build a Three-Layer Scoring System
3.1 Replace Public Leaderboards with Your Own Task Set
Public benchmarks are good for “comparing side by side”; your own task set is good for “making the decision.”
Task sources: Mine high-frequency, high-value scenarios from places like:
- Recurring issues in support tickets and user feedback;
- Requirement documents and designs, which can be turned into page-generation or frontend-restoration tasks;
- Common defect types in code review;
- Historical incidents and rollback records.
Task design: Start with 50–200 tasks, evenly stratified across capability dimensions. Each task contains three parts—input, constraints, and expected result:
# Page-generation task exampleTask ID: FE-0007Input: An e-commerce admin order-list page with a filter area, bulk actions, and pagination; the table must keep its header fixed when there are many rowsConstraints: Semantic tags; keyboard support; text contrast >= 4.5:1; no extra frameworksExpected: A runnable page + structure notesScoring dimensions: build passes (0/1) | fidelity (0-5) | visual quality (0-5) | interaction completeness (0-5) | code quality (0-5)# Coding task exampleTask ID: BE-0021Input: A description of an authentication flaw in an internal service, plus a minimal reproduction caseConstraints: Modify only necessary files; don't change external interfaces; don't add dependenciesExpected: A patch + root-cause explanationScoring dimensions: bug reproduced as fixed (0/1) | regression tests pass (0/1) | change scope (0-5) | root-cause accuracy (0-5)
Three points to keep in mind when designing the task set:
- Rotate and continuously add new samples on a regular basis to reduce the risk of models overfitting to the evaluation set;
- Privatize and de-identify evaluation tasks so they retain discriminative power if made public;
- Version the evaluation set itself—otherwise cross-time comparisons can’t be traced.
3.2 Automated Checks: Executable, Reproducible, Low-Cost
Whatever can be judged programmatically should not be left to subjective judgment. Both frontend and code tasks can run automatically in isolated containers.
Automated checks for frontend tasks:
- Build check: Whether the code compiles and bundles, and whether there are syntax or type errors;
- Structure check: Parse the generated page’s structure tree to confirm key modules exist and nesting levels are reasonable;
- Visual-rule check: Extract values like spacing, font size, and color, and judge whether they meet design and accessibility requirements;
- UI regression: Render screenshots at fixed resolutions and compare against a baseline via pixel or structural diff to catch layout shifts;
- Interaction check: Drive the browser with scripts to verify key paths such as form validation, dialogs, and pagination.
Automated checks for code tasks:
- Install dependencies, compile, and run unit tests in an isolated environment;
- Apply regression tests to the patch to confirm the fix hasn’t broken existing functionality.
Here it’s worth stressing the need for sandbox isolation: model-generated code is untrusted and may have side effects like outbound connections or file writes, so it must run in a restricted environment. Techniques like network restrictions, a read-only filesystem, resource quotas, and one-time containers can control the risk.
3.3 Model-Assisted Scoring: Trading Controllable Bias for Efficiency
Subjective dimensions like “visual quality” and “code readability” can be scored by another strong model acting as a judge. This significantly reduces manual cost but requires guarding against several known biases:
- Position bias: Judge models tend to favor candidates ranked first or last;
- Length bias: Longer responses with more comments are easily rated as “better”;
- Style preference: Some models rate responses in their own or a similar style more highly;
- Self-preference and fragility: Results may be inconsistent across different judge models and across different runs.
Mitigations include: randomly shuffling candidates and using double-blind review; fixing the judge model version and prompt template to ensure cross-batch comparability; setting hard thresholds (for example, a failed build is automatically disqualified); and periodically comparing sampled results against expert review to calibrate consistency.
3.4 Human Review: Handling the “Last Mile”
Whether the build passes and whether all tests pass can be judged automatically. But semantic fidelity, aesthetic preference, and maintainability still need human judgment. Recommended practices:
- Define detailed scoring rubrics where each score maps to a clear observable feature;
- Use double-blind review with 2–3 reviewers scoring independently;
- Compute inter-reviewer agreement; a high disagreement rate signals that the rubric needs revision or reviewers need training;
- Feed review results back to calibrate model-based scoring, gradually capturing the parts that can be automated.
3.5 Composite Scoring: Weight, Don’t Just Add Up
Each sub-dimension carries different value for the business—for example, correctness comes first in a trading system, while visuals come first on a brand showcase page. So the total score should be a weighted composite of the dimension scores by business weight. Moreover, code quality and interaction completeness in frontend tasks are only meaningful once the build passes. Set an “admission threshold” first (a failed build is disqualified), then compute the weighted score over the qualifying samples.
3.6 Multiple Sampling: Conclude from Pass Rates, Not Single Runs
Generative model output is stochastic, so a single sample carries a lot of chance. Sample each task n times (for example, n=5) and record pass@k: the fraction of times at least one of k samples passes. The classic estimation formula for code tasks is:
# For each task: n is total samples, c is the number that passedpass_at_k = 1 - C(n - c, k) / C(n, k) # C denotes the combination function
Teams still need to interpret these two metrics in the context of the business:
- pass@1 measures out-of-the-box reliability and suits automated pipelines;
- pass@k measures the upper bound of capability when paired with human selection.
4. Engineering: Set Up an Automated Evaluation Pipeline
When there are many candidate models and versions update frequently, manual evaluation can’t keep up. It’s worth engineering the evaluation flow. Sample code:
# Model evaluation pipeline (pseudocode)def run_eval(model_id, tasks, repeats=3):records = []for task in tasks:for _ in range(repeats):# 1. Record the input fingerprint and parameters to ensure reproducibilityrequest = build_request(task, temperature=0.7, seed=42)# 2. Call the model through a unified adapter to hide differences between vendor APIsraw = model_client.chat(request)records.append({"model": model_id,"task_id": task.id,"request_hash": hash(request),"raw": raw,"input_tokens": raw.usage.input_tokens,"output_tokens": raw.usage.output_tokens,})# 3. Compile, build, and run tests in an isolated sandboxexec_results = run_in_sandbox(records, limits={"cpu": "2", "mem": "4Gi", "timeout": "300s"})# 4. Merge automatic rule checks + judge-model scores + human review resultsscores = merge_scores(exec_results, rule_checker, llm_judge, human_labels)# 5. Compute pass@k, average score, and cost per task by business weightsreport = aggregate(scores, weights=task.weights)# 6. Persist results and trigger dashboards and alertssave_report(report)alert_if_regression(report, baseline="last_release")return report
Unified adapter: Wrap a consistent calling interface for different models, standardizing parameters like message format, temperature, and max output length. Switching models then only requires a config change—neither evaluation code nor business code needs to change—and it lays the groundwork for later multi-model routing.
Isolated sandbox: Each evaluation runs in a one-time container. Network access is restricted during execution to prevent model-generated code from stealing secrets or making outbound requests. CPU, memory, and timeout limits are also set to avoid infinite loops bringing down the whole pipeline.
Results and metrics management: Store raw outputs, individual scores, and token usage together, queryable and comparable by model version, task type, and time. Wire core metrics (pass rate, average score, P95 latency, cost per task) into dashboards and alerts, with automatic triggers when a new version regresses.
Infrastructure choices: Enterprises can build this pipeline on cloud platforms such as Baidu AI Cloud using general cloud capabilities. The evaluation dataset can live in object storage, execution tasks can be scheduled via a message queue to isolated environments on a container platform, and invocation and execution logs can go to a logging service with metrics feeding monitoring and alerts. Alternatively, reuse the internal infrastructure your organization already has.
5. Cost Evaluation: Turn “Price Cuts” into Quantifiable Gains
API prices on mainstream platforms keep falling, and new-generation models sometimes deliver equal or better performance—some priced at roughly one-third of the previous generation. But a lower unit price doesn’t mean total cost falls proportionally. When evaluating, focus on these points:
- Unit price must be considered together with token consumption. Frontend generation often includes a large amount of context (design notes, component code) and produces long outputs, so actual cost isn’t necessarily linear with unit price.
- Measure cost per effective task. This metric = (input cost + output cost + retry and failure cost) ÷ number of effective outputs. Judge real returns by this metric rather than unit price.
- Distinguish billing for sync vs. async and batch requests. Refer to each platform’s latest public terms for pricing, and watch for price changes.
- Watch vendor lock-in and migration cost. Prefer models with compatible calling methods that can be switched quickly through an adapter.
6. Risk Control: Write Security Requirements into Test Cases
6.1 Generated-Content Safety
- Code defects: Add sensitive scenarios to the evaluation set—hard-coded secrets, SQL injection, privilege-escalation access—to test whether the model proactively adopts secure practices like parameterized queries and input validation;
- Dependencies and licenses: Check whether the model carelessly introduces dependencies of unknown origin or with incompatible licenses;
- Information protection: De-identify code and designs before submission, stripping secrets, personal data, and other sensitive content.
6.2 Usage-Process Risks
- Version drift: Integrate evaluation into the CI flow so a new release automatically runs regression tests and blocks the release if a regression appears;
- Vendor risk: Keep switchable via the adapter and periodically review terms;
- Over-reliance by humans: Humans are always the final reviewers, and critical paths must go through human confirmation.
6.3 Agent-Oriented Scenarios
Once a model is wired into an agent, the risk surface expands further. Tool calls and code execution bring risks like unauthorized operations and prompt-injection attacks (where an attacker hides malicious instructions in a webpage or document the model reads). These need dedicated test cases: can the model recognize and refuse suspicious instructions, and can it distinguish “user intent” from “data content”? The evaluation scope should be updated in step with how the application evolves.
7. Common Pitfalls and How to Avoid Them
- Judging only by leaderboard rank: Public benchmarks diverge widely from your own business—rely on your own task set;
- Judging only by pass@1: Single-sample results fluctuate a lot—combine them with pass rates from multiple samples;
- Comparing only unit price: Ignoring token consumption and rework cost can actually raise total cost;
- Testing only correctness, not maintainability: Code that passes tests isn’t necessarily maintainable long-term—score code quality separately;
- Treating evaluation as one-off: Models keep evolving, so the evaluation set must keep updating and be re-run regularly;
- Scoring only with a judge model: Subjective dimensions need cross-calibration between a judge model and double-blind human review.
8. Summary
The rapid iteration of LLM capabilities and the continued drop in prices give developers a window of “higher capability, lower cost.” But only technology selection built on rigorous evaluation can turn these gains into real productivity.
Recommended steps to build your own evaluation system incrementally:
- Capability breakdown: Split coding and visual generation into observable sub-items;
- Task-set construction: Draw samples from real business into a stratified set of 50–200 tasks;
- Scoring system: Build a three-layer scoring system combining automated checks, model-assisted scoring, and human review;
- Engineering: Stand up an automated evaluation pipeline for continuous regression and continuous monitoring;
- Decision loop: With cost per effective task as the core metric, form a decision loop of “evaluate—select—canary—feedback.”
The ultimate goal of evaluation isn’t to find a “strongest model” but a model combination that is “good enough and predictable” under your business constraints. Only by turning evaluation capability into team infrastructure can you keep making rational decisions and evolving quickly through every wave of technology.
