1. Background: Why Every New LLM Release Deserves a Thorough Test Run

Whenever a new general-purpose LLM is released, the developer community’s first instinct is often to “test it out.” There is a solid reason behind this: model capability competition has shifted from single benchmark scores to real-world task experiences. While public leaderboards reflect some capabilities, there is always a gap between them and actual business experiences. Therefore, conducting a designed, in-house evaluation around your specific business scenarios is the most reliable part of the model selection process.

A clear trend in recent industry model releases is that frontend code generation quality has generally taken a significant step up, and Agent capabilities have evolved from “being able to call tools” to “stably completing multi-step tasks.” Some joke that LLMs are like the “Philosopher’s Stone,” seemingly able to solve any development puzzle—but in engineering, we need a sober ruler. This article does not target any specific product. Instead, it abstracts an evaluation methodology for these two types of capabilities and explains how to implement it with examples.

2. Why Frontend Generation is the “Touchstone” of LLM Capabilities

2.1 Capability Coverage of Frontend Tasks

Behind a simple prompt like “create a cool webpage,” multiple model capabilities are being assessed simultaneously:

• Semantic understanding and requirement completion: User descriptions are often vague, and the model needs to infer style, layout, and interaction intent on its own.
• Aesthetic and design sensibility: Color matching, whitespace, font hierarchy, and animation rhythm are difficult to quantify with rules but intuitively reflect the model’s internalized design knowledge.
• Spatial and geometric reasoning: 3D scenes involve coordinate systems, cameras, projections, and rotation transformations. This is essentially “doing math with code.”
• Engineering capability: Component splitting, state management, dependency selection, and code maintainability.

Because it spans three dimensions—perceptual (aesthetics) + rational (geometry/physics) + engineering (code)—frontend generation serves as a highly cost-effective entry point for observing a model’s comprehensive capabilities.

2.2 3D and Lighting: The Most Differentiating Sub-tasks

Among various frontend sub-tasks, 3D demonstrations and lighting effects offer the highest differentiation:

• 3D scene construction: Requires the correct use of scene graphs, transformation matrices, and material systems. A typical failure mode is that the model “knows the concepts but writes the wrong coordinate axes,” causing objects to float in the air, clip through each other, or face the wrong direction.
• Lighting and shading: Involves lighting models, normal calculations, and shader writing, requiring a strong foundation in computer graphics.
• Animation and interaction: Requires a comprehensive trade-off among easing functions, event loops, and performance budgets (frame rates).

Evaluation suggestion: Prepare a set of use cases ranging from easy to difficult, such as “basic rotating cube scene” -> “indoor scene with shadows” -> “particle fluid/volumetric lighting effects.” Observe the model’s capability decay curve as difficulty increases, rather than just looking at whether it can “succeed by chance” at the highest difficulty.

2.3 Defining Standards for “Good Looks”: Dimension Breakdown and Scorecards

Subjective aesthetics can easily lead to endless debate. It is recommended to break them down into scorable dimensions, for example:

{
  "Visual Presentation":   { "Weight": 0.3, "Key Points": ["Color harmony", "Reasonable layout", "Clear font hierarchy"] },
  "Functional Correctness": { "Weight": 0.3, "Key Points": ["Full requirement coverage", "No interaction logic flaws", "Edge case handling"] },
  "Code Quality":           { "Weight": 0.2, "Key Points": ["Clear structure", "Reasonable dependencies", "Maintainability"] },
  "Running Experience":     { "Weight": 0.2, "Key Points": ["First-screen speed", "Stable frame rate", "No console errors"] }
}

Score each dimension from 1 to 5. Have multiple reviewers score independently and take the median to significantly reduce subjective bias.

3. Agent Capability Evaluation: From “Chatting” to “Getting Things Done”

3.1 Shifting Focus from “Single-Turn Quality” to “Process Quality”

In Agent scenarios, the model must maintain a clear perception of task status across multiple execution steps—knowing which step it is on, what has been completed, what to do next, and how to roll back upon failure. This is a completely different capability from single-turn Q&A. During evaluation, focus on observing:

• Task decomposition: Can it break down vague goals into a reasonable sequence of sub-tasks?
• Tool calling: Are the parameters correct, is the timing appropriate, and is there tool abuse?
• State perception and memory: In long tasks, does it repeat completed steps or forget early constraints?
• Error recovery: When encountering errors, does it blindly retry, or does it read logs, locate the cause, and revise the plan?
• Stopping condition: Can it determine task completion and converge, rather than looping infinitely?

3.2 Dual-Track Evaluation: “Process Metrics + Outcome Metrics”

Looking only at the final result masks numerous issues. It is recommended to record both:

• Outcome metrics: Task success rate, final deliverable quality.
• Process metrics: Average number of steps, tool call count, self-healing rate after errors, repetition rate, token consumption.

For example, if two models can successfully complete the same task, but one finishes in 8 steps and the other takes 30 steps with repeated rollbacks, their production costs are completely different.

3.3 Example: A Multi-Step Agent Evaluation Task

Task: Fix a bug in a given code repository and add unit tests.
Available Tools: File read/write, command execution, test runner.
Evaluation Points:
1. Whether it reads the relevant code before making changes (rather than guessing modifications directly);
2. Whether it proactively runs tests to verify after modifications;
3. Whether it can iteratively fix issues based on error messages when tests fail;
4. Whether it provides a clear change summary at the end.

The advantage of such tasks is that environmental feedback is objective (tests pass or fail), making evaluation results reproducible and quantifiable.

4. Building a Reproducible Evaluation Pipeline

4.1 Controlling Variables: Making Comparisons Fair

• Fixed prompts: Use the exact same prompt for all candidate models for a given task to avoid bias caused by “customizing prompts for a specific model.”
• Multi-round sampling: Run the same use case at least 3 to 5 times and calculate the success rate rather than looking at only the best attempt. Keep sampling parameters like temperature consistent and log them.
• Pinned versions: Record model version numbers and call dates. Perform timely regression testing after model updates.

4.2 Automated Evaluation Skeleton (Pseudocode)

The following pseudocode demonstrates the logic of a minimal evaluation framework (interfaces are generic illustrations; refer to the actual model service API used in practice):

import statistics
CASES = load_cases("cases.yaml")  # Cases: prompts, scoring criteria, pass thresholds
def run_case(case, model, rounds=5):
    scores = []
    for _ in range(rounds):
        resp = model_service.chat(            # Generic model service call (illustrative)
            model=model,
            messages=[{"role": "user", "content": case.prompt}],
            temperature=0.7,
        )
        artifact = extract_code(resp)         # Extract generated code
        result = run_in_sandbox(artifact)     # Run in sandbox: screenshot / execute tests
        scores.append(judge(case.rubric, result))  # Rule-based scoring or model-assisted scoring
    return {
        "model": model,
        "mean": statistics.mean(scores),
        "pass_rate": sum(s >= case.threshold for s in scores) / len(scores),
    }

In practice, this can be further enhanced with sandbox environment isolation, screenshot archiving, and automatic archiving of failure samples to object storage or logging services for subsequent review.

4.3 Combining Manual Review and Automation

• Visual outputs: Take screenshots and have multiple people score independently using the scorecard.
• Code outputs: Use static checks and test pass rates for objective scores, with manual spot-checks for readability.
• Large-scale scenarios: Use model-assisted scoring for initial screening, followed by manual review of boundary samples. However, be mindful of the evaluating model’s own biases.

5. Common Pitfalls and Checklist

1. Drawing conclusions from a single case: One cool screenshot does not represent stable capability. Ensure multi-round, multi-difficulty sampling.
2. Unfair prompts: “Tuning” prompts for candidate models before comparison means you are testing the prompts, not the models.
3. Confusing demos with production: A working demo does not mean it is production-ready. Evaluate latency, cost, context length limits, and output stability.
4. Ignoring engineering integration costs: SDK compatibility, streaming output, function calling formats, and concurrency/rate-limiting strategies all affect the implementation experience.
5. Viewing evaluation results statically: Models iterate quickly. Conclusions from last quarter may already be outdated. Evaluation should be built into a sustainable internal benchmark for regression testing.

6. From Evaluation to Implementation: Advice for Teams

• Establish an internal benchmark set: Accumulate 30 to 50 representative tasks from real business scenarios. The first thing to do after a new model is released is to run a regression test.
• Tiered selection: Prioritize generation quality and aesthetics for frontend demo tasks; prioritize process metrics and stability for Agent tasks.
• Leverage cloud-managed capabilities: Build an evaluation pipeline using general cloud services like model service platforms, container platforms, and monitoring/alerting to turn evaluation from a “one-time action” into a “continuous process.” Multiple cloud providers (such as Baidu AI Cloud) offer such general capability categories, which teams can combine as needed.
• Canary release with low traffic: Before a formal switch, conduct canary testing on non-core paths to let real-world online metrics speak for themselves.

7. Conclusion

“A new model is out, let’s test it” is becoming a developer reflex, which in itself is a sign of industry maturity. More important than “which model is stronger” is having your own evaluation methodology: clear dimensions, reproducible processes, and verifiable conclusions. When you can answer with data “how much better is it in my business scenario,” model selection is no longer a herd-following exercise, but a sober engineering decision.