A Scientific Approach to Evaluating LLMs: Frontend and Agent Capabilities

1. From “Watching Demos” to “Building Systems”: Why Evaluation Needs Methodology

Whenever a new Large Language Model (LLM) is released, the tech community’s first reaction is usually to watch the demos: a prompt generating a cool webpage, or an Agent autonomously completing multi-step tasks. While intuitive, these demos are unreliable for technical selection:

• Sample Bias: Demo use cases are typically cherry-picked, reflecting the model’s upper limits rather than its expected performance.
• Irreproducibility: Sampling parameters, prompt details, and session contexts significantly affect outputs; a single run lacks statistical significance.
• Scenario Mismatch: Demo highlights might not align with your business pain points. For instance, a demo might excel at copy refinement, while your scenario heavily relies on long-context code modification.

Therefore, a more engineering-oriented approach is to treat “evaluation” as a small software project: define evaluation goals, design test sets, fix the execution environment, quantify scores, and archive reports. Below, we unfold a practical evaluation framework using two capabilities developers care most about: frontend code generation and Agent task execution.

2. Evaluation Framework: Define Dimensions Before Scores

2.1 General Dimensions

Regardless of the capability being evaluated, the following four dimensions should be covered:

1. Correctness: Is the generated code runnable, and does the logic meet the instruction requirements?
2. Instruction Following: Does it satisfy all explicit constraints (tech stack, output format, disabled items)?
3. Code Quality: Structural organization, readability, maintainability, and potential security risks.
4. Stability: The variance across multiple runs of the same prompt, directly reflecting output reliability.

2.2 Scoring Method

A three-tier structure is recommended: “Rule-based Scoring + LLM-as-Judge + Manual Spot-checking”. Rule-based scores handle mechanically determinable hard metrics (e.g., compilation pass, assertion pass). The Judge model handles soft metrics like aesthetics and fluency, while manual spot-checking calibrates systematic biases of the Judge. The ratio can be adjusted based on available manpower, but the manual spot-checking step is not recommended to be omitted entirely.

3. Frontend Generation Evaluation: Four Probes

3.1 Visual Aesthetics and Fidelity

Test method: Provide a visual requirement described in natural language (e.g., “dark theme data dashboard, card-based layout, gradient accent colors”) or a design mockup screenshot, and ask the model to output a complete, runnable page. Scoring focuses on:

• Layout rationality (grid, whitespace, alignment relationships)
• Hierarchy of colors and typography
• Responsive adaptation implementation
• Detail completeness (hover states, empty states, boundary performance under extreme data)

3.2 Spatial Understanding and 3D

Ask the model to generate a WebGL-based 3D scene, such as “a draggable, rotatable cube array with light sources and shadows”. Scoring focuses on whether camera controls, lighting models, and geometric organization are correct, as well as the handling of coordinate systems and matrix transformations. This is an effective probe for testing spatial reasoning capabilities that ordinary text tasks cannot measure.

3.3 Lighting, Shadows, and Animation Effects

Examine CSS animations, Canvas drawing, and shader capabilities. Typical use cases include particle backgrounds, parallax scrolling, and ripple buttons. Beyond “does it look right visually”, performance literacy should also be evaluated: does the animation prioritize composite layer properties (transform/opacity)? Are there obvious reflows or repaints? Is the implementation frame-rate friendly?

3.4 Code Engineering Quality

For the same visual requirement, different models yield vastly different code structures. It is recommended to check:

• Is it componentized, or just piled into a single file?
• Does it use semantic tags and necessary accessibility attributes?
• Are there code smells like rampant inline styles or magic numbers?

A reusable test prompt template looks like this:

Please generate a single-file webpage with the following requirements:
1. Theme: A dark, tech-style product landing page
2. Tech constraints: Use only native HTML/CSS/JS, no external libraries
3. Must include: Navigation bar, Hero section (with gradient lighting effects), three-column feature cards, footer
4. Interactions: Card hover should have transition of position and shadow; navigation bar background should change on scroll
5. Output: Complete runnable code, do not omit any part

Incorporate such templates into version control and freeze them to ensure valid cross-model and cross-time comparisons.

4. Agent Capability Evaluation: Look at Results, but Also the Process

Agent evaluation is much more complex than single-turn generation, with the core lying in “process quality”. Four key points are recommended:

4.1 Task Decomposition and Planning

Give a multi-step task (e.g., “scrape a public data page, clean the data, and generate a chart page”). Observe whether the model first outputs a reasonable plan. The plan’s granularity should be moderate—too broad means it gets lost during execution, too detailed wastes steps and tokens.

4.2 Tool Calling and Error Recovery

Intentionally induce exceptions: make a specific tool return an error or empty result, and observe whether the model retries, switches approaches, or simply gives up. The error recovery rate is a critical metric for Agent deployment, often reflecting real-world usability better than first-time success rate.

4.3 Process Self-Awareness

An excellent Agent should clearly know “what it is doing, which step it is at, and what information it lacks”. During evaluation, check its process output: does it proactively report phase progress? Does it initiate clarification instead of guessing parameters when information is insufficient? Process transparency directly determines the cost of human intervention and fallback mechanisms.

4.4 Long-Horizon Tasks and State Management

In cross-multi-turn tasks, examine whether the model maintains goal consistency: does it forget early constraints, or repeat completed steps?

Pseudo-code for a minimal evaluation loop:

TASKS = load_tasks("agent_tasks.jsonl")   # Each contains goal, available tools, expected artifact
for task in TASKS:
    for run in range(N_RUNS):              # Multiple runs to eliminate sampling variance
        trace = agent.run(task.goal, tools=task.tools)
        score = {
            "success":    judge.final_artifact_ok(trace, task),
            "steps":      len(trace.actions),
            "recoveries": count_error_recoveries(trace),
            "drift":      judge.constraint_drift(trace, task),
        }
        save(trace, score)                 # Retain full trace for review

The key is retaining the full trace: looking only at the final pass/fail loses a lot of information. Frame-by-frame review is necessary to pinpoint whether failure occurred in planning, calling, or recovery.

5. Putting it into Action: Minimal Evaluation Pipeline

1. Build Test Sets: Stratify by difficulty (basic/advanced/tricky), with each layer covering different sub-capabilities; bring test sets into version control to avoid “evaluation set drift”.
2. Fix Execution Environment: Unify sampling parameters, system prompts, execution sandboxes, and timeout strategies. Any variable changes must be recorded.
3. Automate Execution and Artifact Retention: Save code artifacts, traces, and scores for every run. Engineering-wise, leverage general capabilities of cloud platforms to simplify setup. For instance, using object storage to archive artifacts, log services to aggregate traces, and monitoring alerts to track pipeline anomalies. Taking Baidu AI Cloud as an example, these general components can be combined directly without building storage and scheduling from scratch.
4. Scoring and Reporting: Rule-based scores are calculated automatically; Judge scores must include reasoning; manually spot-check 5%–10% for calibration. Finally, output a radar chart or itemized table, turning “which model is better for my scenario” into a discussable data problem rather than a debate of opinions.

6. Common Pitfalls Checklist

• Drawing conclusions from a single run: Sampling has randomness; run critical use cases at least 3–5 times to observe variance.
• Testing with demo-level prompts: Overly polished prompts will mask the model’s true performance under plain instructions.
• Ignoring cost and latency: At equivalent quality, token consumption and response time are often hidden variables in decision-making.
• Overlooking security: Generated frontend code should be checked for XSS risks (e.g., directly concatenating user data with innerHTML); Agents should have their tool permission boundaries checked for controllability.
• Overfitting the evaluation set: Public questions may have been mixed into training data; periodically swapping in private questions is necessary.

7. Conclusion

There is no one-size-fits-all leaderboard for LLM evaluation; there is only a measurement system continuously aligned with your own scenarios. By engineering your evaluation—fixing the environment, quantifying dimensions, retaining traces, and conducting regular regressions—you can answer that most critical question in half a day when a new model is released: For my business, where exactly does it excel?