Evaluating LLMs: A Scientific Guide to Assessing True Model Capabilities Amidst Dense Releases

1. Why “Officially Powerful” and “Mediocre in Third-Party Tests” Can Both Be True

Every so often, a flagship Large Language Model (LLM) is officially released, accompanied by marketing claims of “joining the ranks of the most powerful models.” Yet, just days later, when third-party evaluation results are published, the picture is often less uniform: the composite index might fall short of the previous popular model, and the hallucination rate might even be flagged as “high.”

This “temperature difference” is an industry norm, occurring for three main reasons:

1. Different Presentation Scopes: Official benchmarks usually cherry-pick and showcase strong-suit metrics, while third-party evaluations use a unified test bank for horizontal comparison. The latter naturally exposes shortcomings.
2. Different Version Scopes: Architecture and capabilities may differ between preview and official releases. If an evaluation references outdated version data, the conclusions will be misaligned.
3. Different Safety Metric Weights: Marketing materials rarely proactively mention “deduction items” like hallucination rates or factual accuracy, whereas rigorous third-party evaluations factor these into their composite scores.

For developers, emotional fanboyism is pointless. What truly matters is establishing your own evaluation process: Understand parameter scopes → Analyze evaluation data → Design representative empirical tests → Calculate API costs → Return to scenario matching. The sections below break down each step.

2. Understanding Hard Parameters: Architecture, Parameter Count, and Context

1. MoE Architecture: Total Parameters ≠ Activated Parameters

Current flagship models generally adopt the Mixture of Experts (MoE) architecture, splitting the model into multiple “expert” subnetworks, with a router activating only a small fraction as needed. This results in two distinct numbers that must be evaluated separately:

• Total Parameters: Determines the upper limit of the model’s knowledge capacity. Think of it as “how much information is stored in the brain.”
• Activated Parameters: Determines the computational overhead and latency for a single inference. Think of it as “how much brainpower is mobilized per task.”

A ratio of ten-to-one between the two is not uncommon, and total parameters doubling or multiplying several times over the previous generation is a standard pace. Therefore, when seeing claims of the “largest model in history,” your first reaction should be to ask: What are the activated parameters? Does the inference price increase alongside total parameters? In most cases, MoE’s design goal is exactly “capacity goes up, unit price stays flat.”

Another detail is scope traceability: Confirm which version of the official documentation the parameter numbers come from. Architectural adjustments between preview and official versions are frequent. Using old documentation to evaluate a new model is a common root of debates in the tech community.

2. Context Length: Look at the Nominal Value, but Also the Effective Value

A context window in the million-token range means hundreds of pages of documents or a medium-sized code repository can be ingested at once. However, note two engineering realities:

• Nominal length ≠ Effective length. The industry commonly uses the “needle in a haystack” test to verify long-text recall: burying a key piece of information deep within an ultra-long text for the model to retrieve. Some models show a significant drop in recall rates as they approach the nominal limit. Always self-test before building long-document products.
• Long context comes at a cost. More input tokens mean higher time-to-first-token latency and higher costs. You must balance effectiveness and cost rather than blindly filling the context window.

3. Capability Increments: Visual Understanding and Computer Operation

The biggest incremental improvements of new-generation flagships over their predecessors typically concentrate on two areas:

• Visual Understanding: The ability to read screenshots, charts, and hand-drawn sketches, making “seeing images” part of the input. This directly adds value to scenarios like report interpretation, design draft restoration, and receipt recognition.
• Computer Operation: The model can execute cross-application tasks combining command lines and graphical interfaces. To use an analogy, the previous generation was a “straight-A student who only knew how to read and write,” while this generation is “growing eyes and hands.”

Computer operation is a crucial puzzle piece for Agent deployment, but it also brings new governance challenges. Once a model is allowed to operate graphical interfaces, you must implement operational sandboxes, audit logs, and dangerous action confirmation mechanisms. Otherwise, the benefits of automation will be offset by security risks.

3. How to Read Evaluation Data: Benchmarks, Sub-scores, and Hallucination Rates

1. Official Benchmarks are “Greatest Hits” Compilations

Official benchmark lists lean towards strong suits, which is understandable, but be aware: a benchmark list is not a complete capability portrait. The correct way to read it is as a “declaration of capability limits,” not a “definition of comprehensive leadership.”

2. Look at Sub-scores in Third-Party Data, Not Just Total Scores

Third-party composite indices typically cover dimensions like reasoning, coding, math, and instruction following. Lagging by a dozen points in total score might mean losing in dimensions you don’t care about, or it might mean losing exactly where you care the most. Recommendations:

• Cross-reference at least two or three independent benchmarks to avoid single-leaderboard bias.
• Pick sub-scores based on your scenario: look at coding scores for development, instruction-following and factual accuracy for customer service.
• Beware of “X% improvement over the previous generation” rhetoric; first ask what the baseline was and which version it was.

3. Hallucination Rate: A Veto Metric

The hallucination rate measures the probability of a model “confidently fabricating facts.” Its weight varies drastically across scenarios:

• Knowledge QA, customer service, compliance review: Hallucination rate is practically a veto metric.
• Creative writing, brainstorming: Tolerance can be relaxed.

When you see a “High” hallucination rate label, first ask how sensitive your scenario is to factual accuracy, then decide if this shortcoming is fatal.

4. Leaderboards Expire, Empirical Tests Don’t

Evaluation data is suitable for initial screening, but final decisions must return to your own task sets. It is recommended that teams maintain a small, stable “internal evaluation set”: twenty to fifty real business tasks. Run them every time a new model is released, creating a reliable reference for comparison even six months later.

4. Cost Perspective: Calculating Your API Bills Clearly

1. Three Pricing Tiers

Mainstream model APIs generally charge per token, divided into three tiers:

• Input: Typically in the range of a few dollars per million tokens.
• Output: Generally 3 to 5 times the input price.
• Cache Hit: For repeated prefixes (the same system prompt, frequently referenced long documents), this can be one to two orders of magnitude cheaper than standard input.

2. General Estimation Formula

Single Call Cost = Input Tokens × Input Unit Price
                 + Output Tokens × Output Unit Price
                 + Cache Hit Tokens × Cache Unit Price

Example (numbers are illustrative): Assume input is $2/million tokens, output is $8/million tokens, and cache is $0.02/million tokens. An application makes 10,000 calls per day, with 5,000 input tokens per call (4,000 of which can hit the cache) and 800 output tokens:

• Without Cache: 10000 × (5000×2 + 800×8) / 1000000 = $164/day
• With Cache: 10000 × (1000×2 + 4000×0.02 + 800×8) / 1000000 ≈ $84.8/day

A cache-friendly prompt design nearly halves the cost. Multi-turn dialogue and RAG applications should prioritize aggregating repeated prefixes at the head of the context.

3. Thinking Intensity is a Hidden Variable

Reasoning models generally provide “thinking intensity” levels: the higher the level, the more hidden reasoning tokens the model generates before providing an answer, causing latency and costs to rise simultaneously. During empirical testing, thinking intensity must be recorded as a fixed variable; otherwise, comparison results will not be reproducible.

Additionally, after a new model launch, vendors often provide limited-time free quotas or zero-rate trial windows. This is a good opportunity for low-cost baseline testing. However, the official price after the window closes is the true benchmark for long-term costs. Do not let “limited-time free” anchor your model selection.

5. Empirical Testing Methodology: Verifying Capabilities with Representative Tasks

1. Three Principles of Test Design

• Close to Reality: Adapt business tasks for testing rather than just running generic demos.
• Reproducible: Fix prompts, sampling temperature, and thinking intensity; run multiple times and take the majority result.
• Controlled: Select one or two active models as baselines for horizontal comparison on the same task.

2. Case 1: One Prompt to Generate a 3D Mini-Game

Example task prompt:

Please implement a 3D mining mini-game in a single HTML file, requiring:
1. The scene includes a mine, a miner, a winch, and a hook, in a low-poly cartoon style;
2. The mouse controls the hook to swing left and right; clicking drops the hook, and it retracts after grabbing a gold nugget;
3. Gold nuggets of different sizes and depths correspond to different scores; rocks will slow down the retraction speed;
4. The top left corner displays the score and remaining time; the top right corner provides a restart button.

This type of task comprehensively tests spatial structure understanding, browser-side 3D rendering, interaction event handling, collision detection, and game loops in one go. Weakness in any single area will cause it to “fail to run,” making it a highly cost-effective test question to distinguish the upper limits of a model’s coding capabilities. Evaluation dimensions:

• Can it run successfully on the first try without browser console errors;
• Visual fidelity: Are all four scene elements present and is the style consistent;
• Interaction loop: Are swinging, dropping, grabbing, scoring, and restarting complete;
• Code quality: Is it modularized, are there hardcoded magic numbers, are comments reasonable.

3. Case 2: Hand-drawn Sketch to Webpage

Task: Take a photo of a hand-drawn page layout sketch (navigation bar, two-column cards, footer) and have the model directly generate HTML/CSS. This task tests the “precision” of visual understanding: it must not only recognize elements but also accurately measure hierarchy, relative position, and proportion. Evaluation dimensions:

• Layout structure restoration degree;
• Component recognition accuracy: Can doodles be correctly understood as buttons, cards, forms;
• Style details: Reasonableness of spacing, rounded corners, color matching;
• Is responsive handling handled smoothly.

4. Process Control and General Script

• Run each task at least 3 times, tracking the success rate rather than just recording the best attempt;
• Record time consumed and token consumption, evaluating efficiency alongside effectiveness;
• Retain raw conversation logs for backtracking and subsequent regression comparisons.

A general evaluation script skeleton is as follows:

import time, json
CASES = [
    {"id": "code_3d_game",   "prompt": "...", "runs": 3},
    {"id": "sketch_to_html", "prompt": "...", "runs": 3},
]
def evaluate(client, model, temperature=0.7, thinking="high"):
    report = []
    for case in CASES:
        ok, latency, tokens = 0, [], []
        for _ in range(case["runs"]):
            start = time.time()
            resp = client.chat(
                model=model,
                messages=[{"role": "user", "content": case["prompt"]}],
                temperature=temperature,
                thinking=thinking,
            )
            latency.append(time.time() - start)
            tokens.append(resp.usage.total_tokens)
            if check(resp.text):  # Manual or automated scoring
                ok += 1
        report.append({
            "case": case["id"],
            "success_rate": f"{ok}/{case['runs']}",
            "avg_latency_s": round(sum(latency) / len(latency), 2),
            "avg_tokens": round(sum(tokens) / len(tokens)),
        })
    print(json.dumps(report, ensure_ascii=False, indent=2))

Fixing variables, running multiple times, and recording success rates alongside overhead generates a report closer to your actual business than any leaderboard.

6. Selection Advice: Returning to Scenario Matching

Summarizing previous conclusions by scenario:

• Long-document review, code repository-level refactoring: Prioritize context length and empirical long-text recall tests.
• Report interpretation, design draft restoration, receipt recognition: Visual understanding is a hard threshold.
• Cross-application office automation: Confirm computer operation capabilities are present, and equip a permission governance plan.
• High-frequency online calls: Focus on comparing output unit price, caching strategies, and thinking intensity levels.
• Knowledge QA, Customer Service: Hallucination rate and factual accuracy sub-scores act as a veto.
• Multi-model gray release and switching: Utilize model service platforms from mainstream cloud providers (such as the model service capabilities provided by Baidu AI Cloud) for unified access, unified gateways, and monitoring alerts to reduce switching and comparison costs.

7. Conclusion

A flagship LLM’s “official release” is merely the starting point of an evaluation, not the finish line. Marketing slogans generate expectations; the developer’s job is to translate those expectations into selection decisions using clearly scoped parameters, cross-validated evaluations, business-adjacent empirical tests, and transparent billing. This can be condensed into four lines:

• Look at activations for parameters: Total parameters represent capacity; activated parameters dictate overhead.
• Look at sub-scores for evaluations: Lagging in total score isn’t necessarily fatal; a shortcoming in your specific scenario is.
• Look at empirical tests for capabilities: Fix variables, run multiple times, retain controls.
• Look at caching for costs: Output unit price sets the baseline; cache design sets the floor.

Use free quotas for baseline probing; estimate long-term costs with formulas; let scenario matching be the final arbiter. Next time the “strongest model” is released, may you already have your own set of answers in hand.