Evaluating LLM Practical Capabilities: From Interactive Deliverables to Cost Accounting

A Shift in Evaluation Paradigms: From “Correct Answers” to “Successful Deliverables”

In the past, evaluating a Large Language Model (LLM) typically meant looking at leaderboards: multiple-choice benchmarks, math problem accuracy rates, and code completion pass rates. While these metrics are not useless, they mostly test “single-turn, static” capabilities. This leaves a considerable gap compared to the actual needs of enterprise deployment, which require taking a requirement from end to end and turning it into a functional, runnable product.

A recent wave of community practices has provided a more intuitive observation window. Upon gaining early access to next-generation models, many developers were no longer satisfied with just asking a few questions. Instead, they directly assigned real tasks: generating an open-world style street scene demo, turning a foldable mobile device into a web page that can be expanded and collapsed in a browser, or creating a hierarchical exploded view of a robotic hand dissected into over 2,300 parts. The deliverables are placed on a web page—viewable, interactive, and reproducible. This is far more convincing than a vague conclusion like “the model has gotten stronger.”

This article dissects these deliverables as “exam questions.” We will explore what capabilities each task tested, what cost structures are revealed by publicly shared community bills, and how regular teams can establish their own evaluation processes based on these examples.

Four High-Information Testing Tasks

1. Long-Form Generation: Open-World Game Scene

One developer publicly shared a complete bill for a third-person open-world scene demo featuring streets, vehicles, and running characters. The entire process took about 10 hours, consumed roughly 4 million tokens (the basic unit of measurement for how models process text and code, where a piece of code or a sentence is split into several tokens), and cost around $52 USD. The developer also tested the model’s ability to adjust rendering settings related to graphics and frame rates, resulting in corresponding visual changes. They personally estimated that completing a similar task with other mainstream closed-source models could cost nearly ten times as much. While this is a personal estimate and actual figures may vary depending on the billing metrics used, the order-of-magnitude difference is worth noting.

From an evaluation perspective, this type of long-haul task tests three things:

• Long-range consistency: Over a 10-hour scale task, does the art style, code architecture, or naming convention drift halfway through?
• Modular decomposition: Are the scene terrain, entity behaviors, input controls, camera follow, and UI overlays clearly layered and independently replaceable?
• Self-repair: When errors occur, can the model iteratively correct itself based on feedback rather than starting over from scratch?

2. Rapid Creativity: A Five-Minute Promo Short

Another developer had the model generate a product promo video. The underlying tech was browser-based 3D rendering (using a mainstream WebGL framework), where text instructions were translated into visuals with camera movements and dynamic elements. The whole process took about 5 minutes and cost roughly $1 USD. “Using the new model to promote the new model” is a fun meme, but the more substantive insight is that creative and presentation material production is becoming “self-serve.” Non-design roles can now get a usable draft in a matter of minutes.

Evaluation criteria:

• First-time success rate (plays without rework).
• Mapping accuracy from prompt to visual (does the camera, pacing, and color scheme follow instructions?).
• Convenience of export and embedding.

3. Parametric Modeling and Browser Interaction Integration

The third deliverable was a foldable mobile device. The developer had the model generate a geometric model within professional 3D modeling software, then converted it into an operable web page. The device could be expanded and folded in the browser, with screen content switching based on its state. Production took about 1.5 hours and cost approximately $12.5 USD.

The key to this type of task is “pipeline integration.” From modeling, materials, and export formats, to the web-side state machine, animation interpolation, and event binding—if any link in the chain breaks, the deliverable fails. This directly corresponds to real-world business scenarios like product showcases, proposal prototypes, and interactive demos.

4. Physical Simulation and Assembly Details

The final category tests “whether it moves naturally”: cloth fluttering naturally with gravity and damping, and a robotic hand exploded into over 2,000 parts. According to the developer, both cases were essentially completed within 10 minutes.

Physical and assembly tasks carry high information value because they cannot be faked. If the cloth parameters are slightly off, it immediately looks like “plastic.” If the parent-child hierarchy or constraint relationships of the 2,000-plus parts are misaligned, the assembly animation will instantly break. It acts almost like a spot check of the model’s “world knowledge.”

Converting “Capability” into “Cost and Speed”

Community cases provide more than just results; they provide bills. Breaking down the costs of long-haul tasks reveals three characteristics:

1. Input tokens far exceed output tokens. During multi-turn iterations, historical context is repeatedly carried forward, meaning input-side consumption is often multiples of the output-side.
2. Hidden costs hide in retries. A single generation failure means paying for output tokens again, and it might inject failed code into the context, driving up costs for subsequent rounds.
3. Output density determines ROI. For the same 4 million tokens, delivering 30 usable modules versus 15 means a twofold difference in unit cost.

Therefore, it is recommended to replace the vague “expensive/cheap” metric with business-oriented indicators like “cost per usable module” or “effective output per hour.” Below is a sample ledger code (Python-like pseudo-code, with data magnitudes taken from the community cases mentioned above):

from dataclasses import dataclass
@dataclass
class TaskRecord:
    """Ledger entry for a single end-to-end task"""
    task_id: str          # Task identifier, e.g., "game-scene-v1"
    tokens_in: int        # Input tokens (context repeatedly carried in multi-turn iterations)
    tokens_out: int       # Output tokens (generated code and explanations)
    wall_minutes: float   # Wall-clock time spent
    usd_cost: float       # Actual expenditure
    accepted: bool        # Whether it passed manual review
    @property
    def token_ratio(self) -> float:
        """Input/Output ratio: a higher ratio indicates severe context repetition"""
        return self.tokens_in / max(self.tokens_out, 1)
records = [
    TaskRecord("game-scene-v1",    3_400_000, 600_000, 600, 52.0, True),
    TaskRecord("promo-video-v1",      90_000,  30_000,   5,  1.1, True),
    TaskRecord("foldable-demo-v1",   800_000, 150_000,  90, 12.5, True),
]
accepted = [r for r in records if r.accepted]
print(f"Acceptance Rate: {len(accepted)}/{len(records)}")
print(f"Average Cost per Accepted Task: {sum(r.usd_cost for r in accepted) / len(accepted):.2f} USD")

The value of a ledger lies in having concrete data when horizontally comparing different models or prompt strategies, rather than relying on impressions.

Reusable Workflow and Acceptance Checklist

It is recommended to organize testing according to the following workflow: Scripting requirements (write acceptance criteria into the prompt) → Small sample probing (test a single module first) → Modular generation → Local execution → Manual review → Integration regression.

Acceptance Checklist:

• Does it start without errors on the first run?
• Does the interaction fully cover keyboard, pointer, and touch inputs?
• Are assets and resources replaceable (textures, models, copy)?
• Are there “hallucinated interfaces”—calling methods that don’t exist or using outdated dependency versions?
• Does performance meet standards (frame rate, memory usage)?
• Have asset licensing and generated content compliance been verified?

During the initial generation, you can provide a structural skeleton like the one below as a scaffolding prompt to the model. Ensuring the structure is correct before filling in scene details can significantly improve the first-time success rate:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Interactive 3D Prototype (Example)</title>
  <style>html, body { margin: 0; overflow: hidden; }</style>
</head>
<body>
<canvas id="stage"></canvas>
<script>
  // Main loop skeleton: Initialize -> Build scene -> Bind interaction -> Update frame by frame
  const canvas = document.getElementById('stage');
  // 1) Initialize rendering context (adapt to the API of the WebGL/WebGPU framework used)
  const ctx = initRenderer(canvas, { antialias: true });
  // 2) Scene elements: ground, subject, light source
  addGround(ctx);
  addSubject(ctx, { position: [0, 1, 0] });
  addLight(ctx, { type: 'ambient', intensity: 0.6 });
  // 3) Interaction state machine: folded <-> unfolded
  let state = 'folded';
  canvas.addEventListener('pointerdown', () => {
    state = state === 'folded' ? 'unfolded' : 'folded';
    animateTo(state);   // Interpolate hinge angle between 0 and 180 degrees
  });
  // 4) Render loop
  requestAnimationFrame(function loop(t) {
    update(t);
    render(ctx);
    requestAnimationFrame(loop);
  });
</script>
</body>
</html>

Engineering Implementation: Turning Deliverables into Assets

Once an evaluation is passed, the process must be sustainable. Use object storage to archive each iteration’s deliverables (web bundles, model files, videos) and apply version tags. Use logging services to record task ledgers, gradually building a prompt template library. Use container platforms to run automated regressions, verifying whether pages start without errors and if interaction and performance meet standards. To reduce self-managed operations, you can leverage general cloud service capabilities like object storage, logging services, and container platforms (such as those offered by Baidu AI Cloud or other standard cloud providers) to support these steps. Of course, specific selections should align with your team’s existing tech stack; the methodology itself is vendor-agnostic.

Limitations and Risks: Manual Review Cannot Be Omitted

• Long-task drift: Styles may change midway or early constraints may be lost. It is necessary to set phase checkpoints for segment-by-segment acceptance.
• Hallucinated interfaces and outdated dependencies: Generated code might call non-existent methods or use old version signatures. It must be actually compiled and run for verification.
• Cost overruns: Context expands with iterations. Budget limits and stop-loss rules should be set.
• Compliance issues: The licensing scope of 3D assets, fonts, and models, as well as the copyright ownership of generated content, must be verified one by one.

Conclusion

The minimum viable path for evaluating next-generation models is this: pick two or three real tasks close to your own business, establish a ledger to record tokens, time spent, expenditure, and acceptance results, compare them, and then scale up usage. Leaderboards answer the question, “Is it smart?” but interactive deliverables answer the question, “Can it work?” For those who actually need to deploy models in production environments, the latter question is far more valuable.