Evaluating LLM Frontend Generation and Agent Coding Capabilities: A Practical Methodology

1. Why High Benchmark Scores Don’t Equal “Usability”

In recent years, Large Language Models (LLMs) have iterated at a rapid pace, often updating monthly. With every new release, vendors provide benchmark scores and capability descriptions. However, for developers, daily experience is rarely determined by leaderboard numbers. Instead, it hinges on details that are hard to quantify:

• When generating a page, is the default output visually ready, or does it require manual style adjustments?
• In 3D scenes, are coordinates and lighting correct on the first try?
• When assigned a multi-step task, does the Agent clearly know its current step and what remains?
• Does the Agent secretly omit critical steps to appear efficient, causing issues to surface later?

These questions have no ready-made benchmarks. They can only be answered through systematic testing. This article consolidates practical testing methods into a reusable evaluation framework covering frontend generation and Agent behavior, intended to guide model selection and engineering implementation.

2. Frontend Generation: Four Dimensions from “Runnable” to “Visually Pleasing”

The core approach to evaluating frontend capabilities is to provide only functional descriptions—no visual mockups or reference implementations—and observe the model’s “default aesthetic baseline.”

2.1 Aesthetics and Visual Design

Aesthetic capability refers to the model’s default output level regarding color palettes, whitespace, typography scale, and component density when explicit visual guidelines are absent. Evaluation points include:

• Color consistency: Does it use a unified color palette rather than assigning random colors to each element?
• Hierarchical rhythm: Do the font sizes and weights of headings, subheadings, and body text create a clear visual order?
• Spacing system: Does the whitespace follow a 4/8px grid system instead of random values?
• Accessibility baseline: Is the contrast ratio between text and background readable?

Common failure modes include defaulting to a “generic blue” theme, overusing shadows and rounded corners, and unbalanced information density. These issues are syntactically correct in code and run without errors, only becoming visible upon rendering. Therefore, “viewing the rendered result” must be the final criterion.

2.2 Spatial Understanding and Layout Reasoning

Layout capability tests the model’s mental model of the box model and flow layouts. You can design a set of progressive test cases:

1. Horizontal and vertical centering (using both Flexbox and Grid);
2. Equal-width columns with controllable gaps;
3. Sticky footer (footer remains at the bottom even if content is shorter than the viewport);
4. Complex layering (order of overlays, masks, and z-index).

Typical spatial errors include abusing absolute positioning for forced placement, uncontrolled container nesting, and layout breaking at breakpoints. These errors often look correct locally but chaotic globally. Evaluation should involve screenshot verification at each breakpoint.

2.3 3D Modeling and Scene Organization

3D capability is a definitive differentiator for frontend proficiency. Evaluation points include:

• Coordinate system and transformations: Are combinations of translation, rotation, and scaling intuitive?
• Geometric assembly: Can it build structurally sound complex objects using basic geometries?
• Camera and interaction: Does it include default draggable camera controls for rotation and zoom?
• Animation organization: Are hierarchical animations (e.g., revolution vs. rotation) clearly distinguished?
• Performance awareness: Does it consider common optimizations like on-demand rendering and instancing?

A common test task is “build an interactive solar system demo in one sentence.” A capable model will consider orbital radius proportions, distinguish between self-rotation and revolution, and include an interactive camera and basic lighting. Weaker models might generate code that “seems to run” but has completely scrambled coordinates. It must be emphasized: 3D generation quality must be verified by running it; reading the code alone is highly prone to misjudgment.

2.4 Lighting, Effects, and Performance Balance

Lighting and effects provide the “cool factor” but are also the most easily overused. Evaluation should score two aspects simultaneously:

• Visual correctness: Are shadow directions consistent with the light source? Are gradients and materials coordinated? Does particle density serve the content rather than distract from it?
• Performance budget: Can it maintain a smooth frame rate on mid-range devices? Do effects provide degradation fallbacks?

A practical principle: effects should serve a clear “narrative purpose.” Models whose default output exercises restraint typically possess a higher aesthetic baseline.

3. Agent Coding Capability: From “Code Quality” to “Behavioral Quality”

The difficulty of evaluating Agents lies in the fact that we are no longer assessing the static quality of a single output, but the dynamic behavior composed of a series of decisions. The following four observation points have the highest discriminative value in practice.

3.1 Self-State Awareness: The Agent’s “Context Ledger”

An excellent Agent should always maintain a clear “ledger”: What is the task goal? What steps are completed? Where are the intermediate artifacts? What is the next step? Signals of good performance include:

• Restating the goal and current plan before each action;
• Not repeating completed steps in long-chain tasks;
• Correctly recovering the original context after being interrupted or given a new directive.

Corresponding failure signals include: calling the same tool in a loop, forgetting the storage path of intermediate artifacts, and falsely claiming “task completed.” The verification method is straightforward: design an engineering task with five or more steps (build project → write module → run tests → fix → deliver), inject an interference command midway, and observe if state consistency is broken.

3.2 The True Boundary of “Wishful Programming”

Recent tests across multiple models have revealed a phenomenon: requirement descriptions can be very close to natural language, even as vague as “making a wish,” yet still yield a runnable implementation. This is fundamentally a leap in requirement completion capability, but two metrics must be clearly distinguished:

• First-time runnable rate: Can the code run directly?
• Semantic compliance rate: Is what runs actually what you wanted?

A high score in the former does not guarantee a high score in the latter—the model will fill in omitted details using its own “default understanding.” Therefore, retaining explicit acceptance criteria in the prompt is the critical defense against “wishful programming” failures.

3.3 Resource Usage Tendencies: Beware of the “Over-Frugal” Bias

This is a behavioral trait worth isolating during testing: some model Agents exhibit an excessively strong “tendency to save”:

• Repeatedly trying to stuff code into existing files when creating new files is necessary;
• Attempting to bypass dependency installation and run directly;
• Outputting only a brief summary instead of the complete error log when needed;
• Reducing feature completeness just to minimize “overhead.”

The probable cause of this bias is that rewards for “conciseness and efficiency” during training were over-generalized—the model treats “saving” as the goal itself, forgetting it is merely a means to complete the task. Its harm is insidious: errors are silently swallowed, tasks are abandoned halfway, and subsequent steps fail due to insufficient information, with root causes difficult to locate from surface logs.

The design principle for test cases is “must spend resources to complete”: e.g., require delivering a multi-file project with external dependencies and complete tests, and observe if it tries to cut corners. Mitigation strategies exist at the prompt engineering level:

[Resource Policy]
1. Allow and encourage creating necessary files and directories; do not merge unrelated modules just to reduce file count.
2. Fully install required dependencies; do not bypass them by cutting features.
3. Output complete error logs when commands fail; do not provide only summaries or silently retry.
4. "Saving" serves task completion; it must not sacrifice task completion.

3.4 Decision Explainability

The final observation point is whether the reasoning text before and after tool calls is traceable: Is planning consistent with execution? Are strategy adjustments visible after a failure? An Agent with poor explainability is almost impossible to debug once an error occurs, multiplying operational costs.

4. A Reusable Test Case Set and Scorecard

Synthesizing the dimensions above, it is recommended to standardize four test cases to form a regression test set:

Case	Covered Dimensions	Key Checkpoints
A: Marketing Landing Page	Aesthetics, Responsive, Accessibility	3-breakpoint screenshots, contrast ratio, spacing grid
B: Interactive 3D Demo	Spatial, Lighting, Performance	Coordinate correctness, frame rate, camera interaction
C: Multi-stage Engineering Task	Agent State Awareness	Context consistency after midway interference
D: Resource Overhead Task	Frugality Bias	Bypassing necessary dependencies, swallowing errors

Each case should be paired with a prompt template that hardcodes strict requirements and resource policies to reduce noise from the model’s “free improvisation”:

[Task] Implement an interactive product launch landing page, including a hero section, features area, data area, and registration form.
[Strict Requirements]
1. Responsive: 375 / 768 / 1440 breakpoints, no horizontal scrollbars at any breakpoint;
2. Accessibility: Body text contrast ratio >= 4.5:1, form controls with labels;
3. Resource Policy: Allow creating files and installing dependencies; do not cut features to skip steps.
[Acceptance Criteria]
- [ ] Core information fully displayed in hero section at 1440 width
- [ ] Form submission includes local validation and success feedback
- [ ] Deliverables include execution instructions

For scoring, it is recommended to break down subjective judgments into weighted dimensions. Score each item manually first, then aggregate to prevent an “overall impression score” from masking specific shortcomings:

{
  "case": "landing-page-v1",
  "model": "<tested_model_version>",
  "dimensions": [
    {"name": "aesthetics",   "weight": 0.25, "checks": ["color_consistency", "typography_rhythm", "spacing_system"]},
    {"name": "layout",       "weight": 0.20, "checks": ["breakpoint_behavior", "stacking_order"]},
    {"name": "spatial_3d",   "weight": 0.10, "checks": ["coordinate_correctness", "interactive_camera"]},
    {"name": "agent_state",  "weight": 0.20, "checks": ["no_step_repetition", "recovery_after_interference"]},
    {"name": "resource_bias","weight": 0.25, "checks": ["no_dependency_bypass", "no_swallowed_errors", "no_feature_reduction"]}
  ],
  "pass_threshold": 0.75
}

5. From Testing to Implementation: Engineering Recommendations

A single test only forms an impression. To turn impressions into team assets, three things are required:

1. Fix variables: Lock sampling parameters and prompt versions to ensure reproducible and comparable results;
2. Establish regression: Run the fixed test case set on every model version update, focusing on preventing capability regression and behavioral bias recurrence;
3. Build continuous evaluation infrastructure: Schedule evaluation tasks in a container platform, store rendered artifacts and logs in object storage, and integrate metric changes into logging and monitoring alert systems. If your team is already using Baidu AI Cloud or a similar cloud platform, you can directly reuse general capabilities like containers, object storage, logging services, and monitoring alerts. Hooking evaluation into existing CI/CD pipelines incurs very low marginal cost.

Additionally, it must be emphasized: aesthetic metrics currently cannot be reliably evaluated automatically. A dual-track approach of “automated metrics + manual blind review” is the more prudent practice today.

6. Conclusion

Some compare LLMs to the “Philosopher’s Stone,” expecting it to turn stone into gold. After conducting numerous tests, you will find a more pragmatic answer: it is not a stone, but a measurable, constrainable, and iteratively verifiable engineering component. Look at “default quality” for frontend dimensions, “behavioral quality” for Agent dimensions, and use a fixed case set for continuous regression. Managing model capabilities through engineering practices is the only path to turning “cool demos” into “production assets.”