A Practical Guide to Evaluating LLMs: Frontend Code Generation and Agent Capabilities

1. Beyond Benchmark Scores: The Need for Hands-On Testing

The release cadence of large language models is accelerating, and benchmark scores are continually rising. However, for engineering teams, there is often a significant gap between benchmark performance and real-world business usability. Benchmarks typically measure general capabilities, whereas real-world business requirements demand specifics: “Can it generate a visually appealing, logically correct page in a single prompt that runs immediately?” or “Can it stably plan, call tools, and self-correct during multi-step tasks?”

Frontend code generation serves as an excellent hands-on testing benchmark because it simultaneously stresses-tests multiple capabilities, and the results are visible and executable:

• Language Understanding: Translating vague requirements into specific UI layouts and interactions.
• Aesthetics and Design Intuition: Layout, color schemes, font hierarchy, whitespace, and alignment. These are difficult to achieve by rote memorization and best reflect the model’s internalized capabilities.
• Spatial and Geometric Imagination: Relative object positions, camera angles, perspective, and lighting in 3D scenarios provide a comprehensive assessment of spatial understanding.
• Engineering Capability: Whether the code is structurally clear, runs directly, and uses dependencies judiciously.

This approach also applies to Agent capabilities: assign the model a multi-step task requiring tool calls and observe its planning, self-reflection, and recovery performance. Compared to abstract Q&A, the output of such tasks is verifiable and scoreable, making it ideal for solidifying a team’s internal evaluation assets.

Hands-on testing must achieve three goals: Reproducibility (stable conclusions across multiple executions of the same task), Quantifiability (using scorecards and metrics rather than “feeling good”), and Comparability (horizontal comparison of different models and versions on the same scale).

2. Evaluation Dimension Design: Define the Rubric Before Running Tasks

2.1 Frontend Generation: Four Dimensions

1. Visual Presentation: Is the overall layout reasonable? Is the color scheme harmonious? Is the font hierarchy clear? Are whitespace and alignment well-executed? It is recommended to score these as independent sub-items to prevent one good aspect from masking other flaws.
2. Spatial Understanding: In 3D scenario tasks, assess whether object positioning is correct, camera angles fit the requirements, and perspective proportions are distortion-free. This is a frequent point of failure for many models.
3. Lighting and Animation: Are light source settings reasonable? Are shadows natural? Are animation curves and trigger timings smooth? Lighting quality often significantly differentiates model performance.
4. Engineering Quality: Can it run directly without modifications? How many console errors are there? How is the code organization and maintainability? Are unnecessary dependencies introduced?

2.2 Agent Capability: Four Dimensions

1. Task Decomposition: Faced with an ambiguous goal, can the model first break it down into clear execution steps rather than rushing in blindly?
2. Process Self-Reflection: Is the model aware of its current step, what has been completed, and what information is missing? This is often overlooked but heavily impacts user experience—a model with clear self-reflection makes its execution process transparent and predictable.
3. Tool Usage: Is the timing of the call appropriate? Are the parameters correct? Does it validate the returned results before proceeding?
4. Exception Handling: Does it adjust its approach and retry after a failure, or does it loop in place? Does it possess the judgment to terminate promptly and report accurately?

3. Building a Reproducible, Tiered Benchmark Task Set

It is recommended to tier tasks by difficulty, increasing the pressure progressively:

• L1 Static Display Page: Tests the aesthetic baseline, such as product landing pages or event pages.
• L2 Interactive Application: Small-to-medium pages with form validation, list filtering, and state management.
• L3 3D and Lighting Scenarios: Implementing rotatable, illuminated demo scenes using native browser graphics APIs or mainstream 3D rendering libraries.
• L4 Multi-File Micro-Project: Component splitting and module referencing to test structural organization and self-consistency.

Task descriptions should be “specifically constrained and verifiable.” Example template:

Task: Implement a single-file HTML product landing page
Hard Requirements:
1. No build tools required; runs directly in the browser;
2. Include top navigation, Hero section, three-column feature cards, and a footer;
3. Dark background with only one accent color allowed;
4. At least one scroll-triggered fade-in animation.
Deliverable: Complete code, no omissions.

The key point: the more specific the constraints, the better the real capability differentiation. Instructions like “write a beautiful page” are unscorable and should be avoided.

4. Scoring System and Automated Evaluation Pipeline

4.1 Visual Scorecard

Score each dimension from 0 to 5, providing clear scoring anchors. For example, “5: Ready for production; 3: Structurally correct but rough in details; 1: Layout broken.” Anchors ensure consistency across multiple reviewers.

4.2 Automated Metrics

• First-Pass Success Rate: The proportion of generated code that runs successfully without modification.
• Runtime Error Count: Console errors and failed resource loads.
• Screenshot Archiving: Fixed-viewport screenshots for blind reviews and cross-version regression comparisons.

4.3 Pipeline Schematic

# Evaluation pipeline schematic (pseudo-code)
tasks = load_tasks("benchmark/frontend_L1_L4.json")
for task in tasks:
    for i in range(N_SAMPLES):            # Multiple samples to test stability
        code = llm.generate(task.prompt, temperature=0.7)
        run  = sandbox.run(code)          # Render in sandbox
        shot = headless_browser.screenshot(run.url)
        record(task.id, i, {
            "run_ok": run.ok,
            "errors": run.console_error_count,
            "visual": score_by_rubric(shot, task.rubric),
        })
report = aggregate()   # Output pass rate, mean score, and variance

Two points to emphasize: First, multiple sampling. Single-run performance is highly affected by randomness; you should report the mean and variance. Second, sandbox isolation. Generated code must run in a controlled environment to mitigate security risks.

5. Agent Task Evaluation Practices

Design a typical multi-step pipeline task: “Read local data file -> Generate visualization page -> Self-check rendering result -> Fix identified issues -> Report delivery.”

Observation points include:

• Does it output a plan before taking action, and is the plan’s granularity appropriate?
• Does it explain current progress and next steps at each phase (externalizing self-reflection)?
• Are tool call parameters correct? Does it validate and handle returned exceptions?
• Upon failure, does it switch approaches or repeat the exact same operation?
• Does it proactively summarize results and list remaining issues at the end?

It is recommended to log the complete trajectory, formatted like this:

[step 1] plan: parse data -> generate chart page -> open self-check -> fix
[step 2] action: read_file(data.csv)      observation: 120 rows, 3 columns
[step 3] action: write_file(chart.html)   observation: write successful
[step 4] action: render_and_check()       observation: 1 console error
[step 5] think: Error due to unhandled empty data branch, rewrite with added logic
[step 6] action: write_file(chart.html)   observation: fix complete
[step 7] summary: Delivered, remaining issues: none

Trajectory logs transform “process self-reflection” from a subjective impression into reviewable evidence, making it easier to track efficiency metrics like average step count and invalid retry attempts.

6. Interpreting Results and Implementation

• Prioritize Stability Over Peak Performance: When mean scores are similar, the model with lower variance is better suited for production environments.
• Factor in Cost and Latency: Interactive scenarios are sensitive to response times. For equal quality, a faster solution provides a better user experience.
• Test Prompt Sensitivity: Describe the same task using three different phrasing styles. High result volatility indicates strong prompt dependency, requiring extra investment in prompt engineering before deployment.
• Gradual Rollout: Start with a limited rollout on internal tool pages. Validate before gradually expanding to user-facing scenarios.

Whether your team chooses to self-host inference services or integrate cloud-hosted LLM services (such as the general model access methods provided by platforms like Baidu AI Cloud), the task sets, scorecards, and pipelines outlined above can be reused as-is. This transforms “switching models” from a one-off decision into an ongoing regression test.

7. Common Pitfalls

1. Drawing Conclusions from a Single Task: One impressive demo does not represent overall capability. You must cover a tiered task set.
2. Ignoring Prompt Sensitivity: It is common for conclusions to flip entirely simply by changing the phrasing of a prompt.
3. Focusing Only on Success Cases: Long-tail failure cases (broken layouts, infinite loops, hallucinated APIs) often have higher diagnostic value.
4. Using “Feelings” Instead of Scorecards: Without a unified standard, cross-model and cross-version comparisons are impossible.

Conclusion

Models iterate rapidly, and new hotspots emerge constantly, but evaluation methodologies are durable team assets. By standardizing dimensions, task sets, scorecards, and automated pipelines, you can obtain real answers tailored to your business within hours of a new model release—instead of being led astray by benchmark scores and marketing hype.