How to Systematically Evaluate LLMs for Frontend Generation and Agent Capabilities

Some people compare large language models to the “philosopher’s stone”—as if you could throw requirements in and get gold out. Reality is often different: official demos look stunning, but once you plug the model into your own business workflows, things fall apart. With new models launching faster than ever, how do you judge in one or two hours, using a structured approach, whether a model can actually get work done? This article uses frontend generation and agent tasks—the two areas that best test a model’s overall capability—as entry points and lays out a practical evaluation methodology.

1. Why “Running a Demo” Is Far From Enough

A frontend page is a litmus test for a model’s overall capability: it tests aesthetics (color harmony, whitespace, typographic hierarchy) and engineering ability (structural decomposition, state management) at the same time. 3D scenes and effects further stress spatial reasoning and mathematical foundations (coordinate systems, matrix transformations, lighting models). Agent tasks, meanwhile, test planning, tool invocation, and self-restraint. Looking only at hand-picked official examples makes it easy to fall for survivorship bias. A structured evaluation is the reliable basis for selection decisions.

2. Frontend Generation: Five Must-Test Dimensions

1. Visual aesthetics and design fidelity. Give the model a natural-language requirement or a screenshot of a design mockup and ask it to produce a complete page. Score from four angles: color harmony, spacing rhythm, information hierarchy, and font pairing. Blind testing is recommended—strip identifiers from multiple models’ outputs, shuffle them together, and avoid preconceptions.

2. Layout and responsiveness. Focus on whether multi-breakpoint adaptation feels natural, whether flexbox and grid are used appropriately, and whether horizontal scrollbars appear on small screens. This dimension best separates “it runs” from “it’s usable.”

3. Interaction and state management. Ask the model to implement a form with validation, an asynchronously loaded list, or drag-and-drop sorting, then observe whether it covers three boundary states: loading, empty, and error. Many models “paint the skin but not the bones”—the page looks good, but event binding and state transitions are a mess.

4. Spatial understanding and 3D visualization. This is where models diverge most. A typical test case:

Implement an interactive 3D scene in a single HTML file. Requirements:
1. Randomly distribute 20 metallic cubes in the scene;
2. Support mouse-drag rotation and scroll-wheel zoom;
3. Include a directional light source, with cubes casting shadows on the ground;
4. Display the real-time frame rate in the bottom-right corner;
5. Do not depend on any third-party library that requires local installation.

Scoring points include: whether objects are distributed in a reasonable space (rather than clumped together), whether camera controls feel intuitive, and whether coordinate conversions are correct. Models weak in spatial understanding frequently produce clipping artifacts or cameras that fly out of the scene.

5. Lighting and effects. Examine the use of directional light, ambient light, and shadows, as well as the performance cost of effects like gradients, particles, and blur. A practical trick: explicitly require “display the real-time frame rate” in the prompt, so performance problems expose themselves.

3. Agent Capabilities: From “Can Chat” to “Can Do Things”

Agent evaluation is far more complex than single-turn conversation. Four things matter most:

1. Task decomposition and planning. A good model first probes the environment (confirming whether files exist, what the field structure looks like), then lays out a plan before acting, with a clear goal at each step—rather than improvising.

2. Tool-calling accuracy. Are parameters valid? Is the right tool selected? Can it switch paths and retry after a failure? You can design deliberate “must-fail” steps: make a file path intentionally nonexistent and observe whether the model gives up immediately or corrects course and continues.

3. Self-awareness and boundaries. A noteworthy phenomenon in hands-on testing: some models show “excessive restraint” in multi-step tasks—to compress step count and cost, they skip verification and wrap up hastily, claiming the task is complete when the output is unusable. Other models can clearly report the current stage, completed items, and next-step plan mid-task. Evaluations should score “process transparency” and “result reliability” separately, rather than looking only at the final result.

4. Error recovery and reflection. Track two metrics—“first-attempt success rate” and “recovery-within-three-steps rate”—to measure a model’s ability to localize and fix failures.

A multi-step task evaluation example:

Task: Read data/sales.csv, compute monthly sales by region,
generate a visualization page report.html, and write outliers to anomalies.log.
Available tools: read_file / write_file / run_python / list_dir.
Scoring points:
1. Does it first confirm the file and field structure?
2. Does it proceed step by step rather than dumping all code at once?
3. Do the computed results match the baseline answer?
4. Does it proactively verify that report.html renders correctly?
5. Does it clearly report progress and the next-step plan throughout?

4. Common Failure Patterns and a Checklist to Avoid Them

1. API hallucination. The model calls nonexistent endpoints or parameters—it looks fine statically but errors out the moment it runs. Countermeasure: every evaluation case must actually run end to end; never score by eyeballing.

2. Dependency version mismatch. Generated code references library versions that don’t match the runtime environment. Countermeasure: pin fixed versions and feed error messages back to the model to test its repair ability.

3. Long-context degradation. Once the task description grows long, the model starts dropping requirement points. Countermeasure: embed 8–10 “must-satisfy” items in the case and check them one by one.

4. Evaluation set contamination. Classic public problems may already be in training data, inflating scores. Countermeasure: rewrite problems based on your own real business requirements and rotate them regularly.

5. Overly conservative agents. Models “tamed” too aggressively by cost targets cut corners. Countermeasure: explicitly add a “completeness” veto item to the scoring rubric.

5. Turning Evaluation into a Sustainable Engineering Process

Scattered manual testing can’t support long-term decisions. Codify it into a pipeline:

1. Define task profiles. Clarify the model’s main responsibilities in your business (page generation, data visualization, or multi-step engineering tasks) and weight scoring toward core scenarios.

2. Build a private evaluation set. Extract 30–50 cases from real requirements, each with its own rubric (automated scoring items plus manual scoring items).

3. Automate execution. Batch-call models through a unified interface on a cloud model service. Cloud platforms typically provide a unified API access method, so the evaluation script only needs to swap the model identifier to compare models side by side, and can be integrated into continuous integration for one-click regression whenever a new model is released.

4. Visualize and archive results. Record score curves for each version to see whether capability is improving or regressing.

5. Manual spot checks. Automated metrics can’t cover subjective dimensions like “aesthetics.” Keep 10%–20% of samples for manual review each round.

A minimal viable evaluation script skeleton:

import json
CASES = [json.loads(line) for line in open("cases.jsonl", encoding="utf-8")]
def run_case(case):
    # Send a request via the model service's unified API (pseudocode)
    resp = model_client.chat(
        model=case["model"],
        messages=[{"role": "user", "content": case["prompt"]}],
        temperature=0.2,
    )
    return resp["content"]
def grade(output, rubric):
    scores = {}
    for item in rubric["auto"]:              # Automated rule-based scoring
        scores[item["name"]] = item["check"](output)
    scores["pending"] = rubric["manual"]     # Manual items enter the review queue
    return scores
for case in CASES:
    output = run_case(case)
    print(case["id"], grade(output, case["rubric"]))

6. Conclusion

The capability boundaries of large models are moving fast, but evaluation methodology is stable: break the vague question of “is it good” into measurable dimensions, write test cases based on real business scenarios, make the code actually run, and make the agent actually do things. Do this well, and no matter when the next new model drops, you can deliver a convincing verdict within an hour—instead of being led around by marketing copy.