Beyond Benchmarks: Four Key Perspectives for Evaluating Real-World LLM Capabilities

I. A Case Study: High Benchmarks, Crashed Reputation

A recently released flagship model provided a textbook example. Judging solely by public marketing data, its hard capabilities were nearly flawless:

• Parameter scale reached the trillion level (~2.8T);
• Context window nominally hit the million-token mark;
• Terminal operation coding benchmark scores jumped from 13% in the previous generation to 65.7%, surpassing several competing flagship models and trailing the industry-recognized leader by only 0.7 percentage points;
• A third-party aggregate evaluation intelligence index scored 55, ranking fourth globally;
• API pricing was reduced by a third, with a clear timeline for open-weight availability.

However, less than a week after release, negative feedback from community testing poured in: the official 3D demo was questioned for relying on external services rather than native model capabilities; a developer failed to replicate an interactive component using reference images across three iterations; and long-running tasks hit usage quota limits. The benchmarks didn’t drop, but the reputation tanked first.

The value of this case isn’t to judge a specific vendor, but to expose the systemic flaws in the current LLM evaluation system. Below, we abstract these into four technical perspectives for step-by-step analysis and provide actionable evaluation practices.

II. Perspective 1: Four Gaps Between Benchmarks and Real Capabilities

2.1 Data Contamination and Task Distribution Bias

Questions in public benchmarks have likely entered the training corpora of various models, making the “teaching to the test” effect difficult to eradicate. Additionally, benchmark task distributions are highly concentrated, whereas real-world business tasks follow a long-tail distribution. It is precisely those long-tail tasks not covered by benchmarks that determine actual user experience. Benchmarks measure “performance on known questions,” while real capability is “performance on unknown questions”—they are inherently not the same thing.

2.2 The “Average Trap” of Aggregate Scores

Third-party aggregate evaluations weight multiple capabilities like reasoning, coding, and agent execution into a total score. The average masks shortcomings, but for specific businesses, the shortcoming is often the decisive factor: a model scoring 90 in coding but only 60 in tool-calling is a failure for a team building AI agents. When reviewing leaderboards, always break down the sub-items and align them with your own business weights.

2.3 Nominal Context ≠ Effective Context

A million tokens is the window limit, not the effective workspace. The industry has repeatedly observed the “lost in the middle” phenomenon: a model’s retrieval and reasoning accuracy for mid-segment information in ultra-long contexts drops significantly. Marketing materials almost never include an “effective length vs. accuracy curve.” During selection, you must self-test using probe tasks.

2.4 Parameter Scale and Capability Are Not Linearly Correlated

Total parameter count, activated parameters (for sparse architectures), training data quality, and post-training strategies jointly determine capability. If trillion-level total parameters are largely sparsely activated, the actual capacity may not significantly exceed that of a smaller but more thoroughly trained model. Parameter scale should only be one reference dimension, not the sole conclusion of capability.

III. Perspective 2: “Skill Plugins” — Component Analysis of Demo Effects

3.1 Phenomenon Review

A user developing a 3D game purchased a subscription to replicate an official, stunning 3D demo, but repeatedly failed. Upon investigation, the issue was found: the model environment had a built-in 3D generation skill, where the actual execution path ran a script to call an external generation service API. The script’s supported version numbers exactly matched those of a specific 3D generation service provider’s API versions (the whistleblower noted this was not yet finally confirmed). The more critical contrast was that when the model was forced to generate using only its own code, the quality degraded significantly.

3.2 Technical Essence: Agent Capability is a “Model + Toolchain” Composite

The typical architecture of a modern Agent system can be abstracted as:

User Request
  └─ Model (Planning / Reasoning / Generation)
       ├─ Inherent Capability: Directly produces text and code
       └─ Skill / Tool Layer
            ├─ Basic tools: code interpreter, file system, etc.
            └─ Skill Packs: orchestration scripts + external service calls

A skill pack is essentially a wrapper of “an orchestration script + an external call”:

# Pseudo-code: Typical skill wrapper structure
@skill(name="generate_3d_model", desc="Generate 3D model")
def generate_3d_model(prompt, version="v2"):
    # The actual generation happens in the remote service, not inherent model reasoning
    resp = http_post(EXTERNAL_ENDPOINT, {
        "prompt": prompt, "version": version
    })
    return save_asset(resp["asset_url"])

This is completely legitimate in engineering—encapsulating external capabilities is the original design intent of an Agent. The real issue lies in the granularity of capability disclosure: if an official demo is implicitly understood by the audience as “native model capability” but is actually an external plugin, it creates a systemic misalignment in capability perception.

3.3 Ablation Testing: Distinguishing What the “Model Knows” from What the “Plugins Know”

For users, the standard method to determine capability attribution is ablation testing:

1. Export the environment’s built-in skill and tool inventory;
2. Disable all tools, retaining only conversation and code generation capabilities, and repeat the same task;
3. Compare the quality difference between the two outputs;
4. If possible, inspect the skill scripts’ dependency declarations and network egress to confirm if external services were called.

# Ablation test illustration
results = {}
for mode in ["with_tools", "without_tools"]:
    tools = default_tools if mode == "with_tools" else []
    results[mode] = run_task(task, tools=tools)
# with_tools significantly outperforms without_tools => capability is mainly from external toolchain

3.4 Implications for the Industry

Capability disclosure should distinguish between two metrics: “inherent model capability” and “system-level capability (including toolchain).” Third-party evaluations should also track these separately. Before metrics are unified, benchmarks, demos, and hands-on tests will each tell a different story, making a reputation crash almost inevitable.

IV. Perspective 3: Coding Agent Hands-on — Why “Replicating an Interactive Component” is a Good Touchstone

4.1 Case Abstraction

A developer used five reference images to ask a model to replicate an interactive gamepad component using a mainstream frontend 3D rendering framework. The results over three iterations were: the first version was evaluated as rough and lazy, “like a downgraded lightweight model”; the second version still had incorrect labels and erroneous spatial relationships between parts; the third version, after receiving a list of annotated screenshot issues, the model acknowledged the errors and began fixing them, seemingly about to succeed, but hit the five-hour usage limit.

Notably, the best-performing model in the same test group exhibited a significant behavioral difference: it would proactively measure positions and spatial relationships, rather than placing parts by “feel.”

4.2 Why This Task is Scientific

• Multimodal Input + Structured Output + Executable Verification: Reference images go in, runnable code comes out, and the result is immediately visible in the browser;
• Objective Criteria: Geometric proportions, occlusion relationships, and interaction events have clear right or wrong answers;
• Naturally Multi-turn: Generate → Feedback → Fix. This tests the model’s repair capability rather than one-shot generation—which is the primary mode of real-world R&D work.

4.3 Five Typical Failure Modes

1. Capability Downgrade Laziness: Using simplified implementations to bypass difficulties, outputting something that “looks right” but fails under interaction;
2. Detail Hallucination: Guessing labels, text, and textures based on training memory;
3. Weak Spatial Reasoning: Guessing relative positions of components, lacking measurement actions;
4. Acknowledging but Not Fixing: Verbally accepting feedback but making no behavioral changes;
5. Resource Interruption: Quotas and rate limits interrupting long tasks, wasting all prior effort.

4.4 Setting “Proactive Measurement” as an Advanced Metric

The dividing line between top-tier and average models is often not one-shot generation quality, but whether they proactively acquire deterministic information: measuring coordinates, reading dimensions, running validation scripts, and turning “guessing” into “verification.” It is recommended that teams set a high weight for this behavior in internal evaluations. It predicts real-world R&D performance far better than any public benchmark.

V. Perspective 4: Quotas and Rate Limits — The Overlooked Evaluation Variable

A five-hour usage limit interrupting the third round of fixes is not a model capability issue, but a product strategy issue. However, it directly determines the completion rate of long tasks. Selection and evaluation must take this into account:

• What are the token and request quotas per unit time, and are they viable for peak scenarios;
• Can sessions be persisted, and can they resume from a breakpoint after interruption;
• Context management strategies for long tasks: incremental summarization, externalizing key states.

The universal engineering response is to break large tasks into recoverable sub-tasks, writing key states to external storage (object storage or databases), so any interruption can be resumed at a low cost. If evaluation does not consider quotas, the conclusions will systematically favor product forms that are “beautiful for short tasks, broken for long tasks.”

VI. Actionable Evaluation Checklist

Consolidating the above four perspectives into an operational checklist:

1. Ablation First: Distinguish model capabilities from tool capabilities before discussing benchmarks;
2. Real Task Replay: Build an internal benchmark using representative tasks from your own business; use public benchmarks only for initial screening;
3. Multi-turn Iteration Design: Evaluate “repair capabilities,” recording diffs and screenshot evidence round by round;
4. Multi-model Blind Testing: Conduct blind tests on identical prompts with hidden brand names to avoid confirmation bias;
5. Self-test Effective Context: Use “needle in a haystack” probe tasks to test the actual long-context accuracy curve;
6. Audit Disclosure Transparency: Check if skill inventories, external dependencies, and quota strategies are publicly available;
7. Total Cost Modeling: Factor the costs of quotas, retries, and manual corrections into the total bill.

Whether calling a model API directly or using hosted models and accompanying toolchains on cloud platforms like Baidu AI Cloud, this checklist applies equally—cloud solutions also have the distinction between “platform toolchain capability” and “inherent model capability,” and equally require ablation verification.

VII. Conclusion

Benchmarks are a necessary condition, not a sufficient one. A stunning demo might be the composite effect of the model and toolchain; a hands-on crash might just be the fault of quotas and engineering details. For developers, the only trustworthy path is: on their own task distribution, using ablation, multi-turn, and reproducible methods, build their own evaluation chain of evidence. The model not dropping behind does not mean the product hasn’t; reputation is ultimately determined by performance on real tasks, not by press conferences.