A Practical Guide to Evaluating LLM Frontend Code Generation and Agent Capabilities

Why Frontend Generation and Agent Capabilities?

Historically, teams have benchmarked Large Language Models (LLMs) using conversational tasks like Q&A, summarization, and translation. However, as models transition into production environments, the importance of two specific capabilities has surged:

1. Frontend Code Generation: This simultaneously tests code correctness and visual aesthetics. It is one of the few capabilities where quality is immediately apparent. Non-technical stakeholders can intuitively judge whether the generated UI is visually appealing, the interactions are smooth, or 3D scenes align with spatial intuition. This makes it an excellent visceral benchmark for model performance.
2. Agent Task Execution: As models evolve from “answering questions” to “completing tasks,” they must autonomously plan steps, invoke tools, and handle exceptions. This capability directly determines whether a model can be integrated into automated workflows.

After conducting a systematic round of empirical testing on these two capabilities, I have abstracted the methodologies and observations into a reusable evaluation framework for teams needing to build internal assessment mechanisms.

Frontend Generation: Four Evaluation Dimensions

1. Aesthetics and Design Fidelity

Frontend generation is not just about “code that runs.” When evaluating aesthetic capabilities, use a fixed prompt. For example: “Generate a dark-theme homepage for a product launch event, requiring a gradient background, frosted glass cards, and scroll animations.” Score the output based on the following sub-items:

• Layout Rationality: Grid alignment, appropriate whitespace, and clear information hierarchy.
• Color and Contrast: Primary colors, accent colors, and text contrast ratios meeting accessibility and readability standards.
• Modern Design Elements: Natural implementation of dark mode, gradients, rounded corners, shadows, and micro-interactions.
• Multi-Sample Consistency: Generate 5 to 10 times using the same prompt to observe if the aesthetic baseline remains stable. Excellent performance means being “consistently online,” not just “occasionally amazing.”

2. Spatial Understanding and 3D Modeling

3D scenes test a model’s spatial reasoning capabilities. Browser-side 3D typically relies on standard Web rendering APIs and mature frontend 3D libraries. You can design progressive test cases:

• Basic: Render a rotatable geometric shape that supports mouse dragging to change the perspective.
• Intermediate: Build a scene with a ground plane, light sources, and multiple objects with correct occlusion relationships.
• Challenge: Implement camera follow, object collision, or basic physics effects.

Evaluation focuses on: correct usage of coordinate systems and transformation matrices, reasonable camera parameters, and object relative positions aligning with natural spatial intuition. Models with weak spatial understanding often exhibit typical errors like “objects embedded in the ground” or “light direction contradicting shadows,” which are the most easily quantifiable deduction points.

3. Lighting and Visual Effects

Lighting capabilities directly dictate the “premium feel” of a page. Evaluate this across three levels:

• Light Sources and Materials: Correct differentiation between ambient light, directional light, and point light; proper rendering of material roughness and metallic properties.
• Effects Implementation: Successful realization of particle systems, light streaks, glows, and shader animations based on text descriptions.
• Performance Awareness: Proactive control of rendering overhead, such as limiting particle counts and avoiding per-frame object creation. A visually stunning effect that drops the framerate to single digits should be penalized.

4. Code Quality and First-Run Success Rate

• First-Run Success Rate: The percentage of attempts that execute directly in a browser without any manual modifications. It is recommended to test over 20 samples.
• Self-Containment: Outputting a single HTML file, inline styles and scripts, and no missing dependencies.
• Engineering Literacy: Naming conventions, necessary comments, and viewport adaptation for mobile devices.

Agent Capabilities: Three Evaluation Layers

1. Task Decomposition and Planning

Provide a multi-step task (e.g., “Scan all log files in a specified directory, count error code distributions, and generate a summary report”). Observe if the model outputs a clear execution plan before taking action and if the plan covers boundary conditions (e.g., file not found, encoding anomalies).

2. Tool Invocation and Execution Loop

• Are the call parameters correct, and is the input organized according to tool constraints?
• Can the model determine the next action based on the previous tool’s return result, forming an “Observation-Thought-Action” loop?
• Does it proactively verify results upon task completion, rather than simply declaring success?

3. Self-State Awareness and Failure Recovery

This is the most noteworthy trait during empirical testing: can the model “know what it is currently doing as an Agent”? Specifically observe:

• Before each execution step, can it briefly state which step of the plan it is on and what the goal is?
• After multiple rounds of execution, can it accurately recap completed and pending items without state drift?
• Upon encountering an error, does it blindly retry the original command, or does it first locate the cause, adjust the strategy, and then retry?
• When consecutive failures reach a threshold, does it proactively report and request human intervention instead of looping infinitely?

An Agent with good self-state awareness significantly reduces human intervention costs in production environments. It is recommended to assign a separate weight to this dimension.

Actionable Empirical Methods

1. Standardized Test Cases

Using frontend 3D as an example, fix a prompt template like the following:

Implement a 3D display page using a single HTML file:
1. The main body should be a rotatable geometric combination, supporting mouse drag to adjust the perspective;
2. Provide at least two lighting mode toggles (e.g., daylight/night);
3. Use CSS gradients for the background and frosted glass effects for cards;
4. Inline all styles and scripts so it runs directly in a browser;
5. Compatible with common desktop and mobile viewports.

Once the template is fixed, different models or different versions of the same model can be compared horizontally.

2. Quantitative Scoring System

Dimension	Weight	Description
Visual Fidelity & Aesthetics	30%	Layout, color, and design element completion
Interaction & Functional Integrity	25%	Drag, toggle, and animation requirements met
Code Quality	20%	Structure, naming, and self-containment
First-Run Success Rate	15%	Ratio of successful runs without human intervention
Performance	10%	Framerate, resource usage, and rendering overhead

Agent evaluation can similarly assign weights across five items: planning quality, tool invocation accuracy, task completion rate, self-state awareness, and failure recovery.

3. Automated Evaluation Script Approach

# Pseudo-code: Agent execution trajectory evaluation framework
def evaluate_agent(task: str, max_steps: int = 10):
    trajectory, state = [], init_env(task)
    for step in range(max_steps):
        action = llm_plan_and_act(state, task)
        # Evaluation point 1: Output a clear step self-description before execution
        record(step, has_self_description(action))
        result = execute(action)
        trajectory.append((action, result))
        # Evaluation point 2: Trigger reflection and strategy adjustment after failure
        if result.failed and not action.adjusts_strategy:
            record_issue("Lacks failure recovery strategy")
        if task_completed(result):
            break
    return final_score(trajectory)

In actual engineering, whether the model is accessed via local deployment or through LLM services provided by cloud platforms like Baidu AI Cloud, these evaluation scripts can be hooked into Continuous Integration (CI) pipelines. With every model version update, a round of benchmark test cases runs automatically, outputting a scoring report to form a regression testing mechanism.

Common Pitfalls and Recommendations

1. Outputs Diverging Due to Vague Prompts: Test cases must explicitly state hard verifiable conditions (quantities, dimensions, interaction methods). Otherwise, scoring will be polluted by subjective noise.
2. Single Results Are Not Representative: Due to sampling randomness, any model should be sampled multiple times to calculate an average. Pay attention to the “worst performance” rather than only looking at the best cases.
3. Do Not Ignore Long-Tail Scenarios: Mobile adaptation, performance degradation on low-end devices, and accessibility support are often where engineering usability gaps truly widen.
4. Balance Performance and Cost: Record the token consumption and time taken for each task. When performance is similar, prioritize the more cost-effective solution.

Conclusion

“Is the LLM good enough?” should not remain a matter of subjective impression. Frontend generation provides an intuitive visual acceptance surface, while Agent capabilities provide an observable execution process. By breaking these down into scorable dimensions, fixing them into reproducible test cases, and supplementing them with automated scripts for regression testing, teams can use a stable yardstick to measure model iterations—truly turning empirical testing into accumulable engineering practice.