How to Evaluate LLMs: A Practical Five-Step Assessment Methodology

Whenever a new flagship Large Language Model (LLM) is released, the narrative is often the same: official marketing claims it ranks among the best in the industry. A few days later, independent benchmarking agencies release their test results, and the picture changes significantly. The comprehensive intelligence index shows a double-digit gap compared to recognized leading models, and the hallucination rate is tagged as “high.”

This disparity is not surprising. It is the result of three compounding factors:

• Benchmark Discrepancy: Official scores usually come from public benchmark datasets selected by the vendor. Independent agencies, to avoid training data contamination, use new or dynamic question banks. The score difference for the same model on “old questions” versus “new questions” can sometimes be larger than the performance gap between two different models.
• Dimensional Discrepancy: Benchmarks measure peak performance on single-turn, short tasks. Real-world production cares more about long-task degradation, multi-turn consistency, hallucination rates, and instruction-following stability. A model can easily shine on a leaderboard but fail in engineering practice.
• Conflict of Interest: Release cadence is strongly tied to market buzz, so promotional messaging naturally highlights favorable perspectives. This is a widespread industry phenomenon, not limited to a single vendor.

Rather than choosing between trusting official claims or third-party reports, it is better to take evaluation into your own hands. Below is a five-step methodology, guiding you from parameter specifications all the way to hands-on testing and cost accounting.

Step 1: Understand Parameter Specifications to Avoid “Size Illusions”

1. Total Parameters ≠ Activated Parameters

Current flagship models generally adopt the MoE (Mixture of Experts) architecture, where two metrics are most easily blurred in marketing:

• Total Parameters: The overall size of the weight files, determining memory consumption and deployment barriers.
• Activated Parameters per Token: The actual number of expert parameters involved in the forward pass for each token, determining the computational cost and theoretical latency of a single inference pass.

Taking a common flagship specification as an example: total parameters often reach the hundreds of billions (generally more than double the previous generation), while single-pass activation is often only tens of billions, with an activation ratio of less than 10%. This means the model “stores” knowledge capacity at a large-model scale but runs with computational overhead closer to a small or medium model. This is the core value of MoE, and also why comparing total parameters directly against dense models is inherently misleading.

2. Beyond Parameters: Verify Three Key Specifications

• Context Length: Distinguish between “claimed length” and “effective length.” The millions of tokens listed in API documentation usually refer to the maximum input limit. What truly determines the long-document experience is distant recall and mid-section information utilization, which require self-testing to verify.
• Capability Increments: The most valuable aspect of a stable release over a preview version is often not a parameter change, but new capabilities—such as visual understanding (reading images, parsing charts) and computer operation (executing cross-application tasks via command line and graphical interfaces). The former gives the model “eyes,” and the latter gives it “hands.” Both are qualitative leaps.
• Reasoning Mode: Whether the thinking intensity is adjustable and whether the thinking process is visible directly impacts the reproducibility of hands-on tests.

In short: Parameters determine the baseline and cost structure, but not the experience ceiling; the ceiling can only be determined through hands-on testing.

Step 2: Deconstruct Four Core Capability Dimensions

Evaluation is not just chatting with the model and feeling good about it; it requires breaking down capabilities into measurable dimensions. It is recommended to cover at least these four:

1. Long-context Recall: “Needle in a haystack” is the baseline test—embedding key information at specific depths in a long document and asking the model to answer. Stricter variants include “multi-needle” and “unordered needles,” designed to catch models that only retrieve information from the beginning and end.
2. Visual Understanding: Do not stop at “recognizing what the image is.” An engineering-oriented hierarchy is: chart data extraction → UI screenshot structural restoration → hand-drawn sketch layout semantic parsing. The last layer is the hardest because sketches have high noise and distorted proportions, exposing the true level of multimodal capability.
3. Code Generation: “Can it run” is only the passing line. Professional scoring should cover: whether dependencies are self-consistent, whether scenario elements are complete (subjects, interactive objects, and feedback systems must all be present), and whether the code structure is maintainable.
4. Computer Operation / Agent Capability: Have the model combine command-line and graphical interfaces to complete cross-application tasks. Evaluate task completion rate, misoperation rate, exception recovery capability, and the token cost of each operation step.

Step 3: Read Scores Correctly—Three Traps in Benchmark References

1. Data Contamination: If a benchmark dataset was made public before the model’s training cutoff date, the score’s credibility must be discounted. Prioritize third-party evaluations using dynamic question banks.
2. The Obscuring Effect of Composite Indices: A composite score is a weighted average of sub-items, which smooths out strengths and weaknesses. Two models with similar composite scores might differ by a whole tier in coding ability. When selecting a model, you should expand the sub-scores and map them against your high-frequency tasks.
3. Hallucination Rate Must Be Listed Separately: Hallucination and capability are two quasi-independent dimensions; a high composite score does not mean a low hallucination rate. In knowledge Q&A and document sorting scenarios, the weight of the hallucination rate should be higher than the benchmark score.

Additionally, it is recommended to incorporate cost-performance into the score-reading framework: divide the score by the output token price to get “intelligence bought per dollar.” This often reflects engineering value better than absolute scores.

Step 4: Design Reproducible Hands-on Test Cases

The key to hands-on testing is controlling variables: using the same prompt, decoding parameters, and thinking intensity settings across models is the only way to make a meaningful comparison. Here are two battle-tested case templates.

Case 1: One Prompt for a 3D Mini-Game (Testing Code and Spatial Understanding)

Prompt template:

Implement a low-poly, cartoon-style 3D mining game in a single HTML file: the scene must include a mine shaft, a miner, a winch, and a hook. The player controls the hook’s swinging and releasing to grab minerals and score points. Pay attention to camera angles and operational feel.

Scoring rubric (0–5 points each):

Dimension	Observation Points
Runnability	Can it open directly in a browser with zero modifications
Element Completeness	Are the mine shaft, miner, winch, and hook all present
Interaction Loop	Is there a closed loop of swinging, releasing, grabbing, and scoring
Visual Style	Does it meet the low-poly and cartoon shading standards
Code Structure	Are the state machine and rendering logic decoupled

This type of case is a “composite capability exam”: 3D spatial modeling, physical intuition, interaction design, and engineering structure are all tested at once. The differences between models in element completeness and interaction loops are more intuitive than any leaderboard.

Case 2: Hand-drawn Sketch to Webpage (Testing Visual Understanding)

Take a picture of a hand-drawn page layout sketch (a few blocks, a navigation row, a button), upload it to the model, and ask it to output the corresponding responsive webpage. Scoring key points: layout restoration (relative position and proportion of blocks), semantic completion (reasonable inference of undrawn details), and styling implementation (whether it is genuinely responsive). This specifically tests the level of the model’s “eyes”: sketches lack precise coordinates, so the model must rely on structural understanding rather than pixel matching to restore intent.

Appendix: Reusable Testing Skeleton

import time, statistics
def run_case(call_fn, prompt, runs=3):
    """call_fn: unified model conversation interface, returns (text, output_token_count)"""
    rec = []
    for _ in range(runs):
        t0 = time.perf_counter()
        text, n_out = call_fn(prompt)
        dt = time.perf_counter() - t0
        rec.append((text, n_out / dt))
    tps = [r[1] for r in rec]
    return {
        "Output Text": rec[-1][0],   # Take the last output for manual scoring
        "Throughput Median (token/s)": round(statistics.median(tps), 1),
        "Throughput Fluctuation": (round(min(tps), 1), round(max(tps), 1)),
    }

Run it 3 times to check stability before doing manual scoring. Stability itself is an evaluation dimension: a model with high variance in output quality across three runs of the same prompt should be used cautiously in automated pipelines.

Step 5: Calculate the Cost and Engineering Books

1. Token Billing Structure

Mainstream API billing is generally divided into three tiers: input, output, and cache hits. The output price is usually 2–3 times the input price, and the cache hit price might be less than one-twentieth of the input price. This means the cache hit rate is the primary lever for production costs. For applications with long system prompts and many conversation turns, implementing hierarchical caching can reduce costs by an order of magnitude. A rough formula:

Cost per call ≈ Input tokens × Input unit price + Output tokens × Output unit price
             − Cache hit tokens × (Input unit price − Cache unit price)

2. Latency and Throughput

Beyond end-to-end latency, you should also record Time To First Token (TTFT) and decoding throughput. In interactive applications, TTFT determines perceived speed; in batch processing tasks, throughput determines the bill and timeline.

3. Engineering Integration

Model evaluation should not end at the raw API layer. Enterprise deployment also requires a unified gateway and quota management, call logging and monitoring alerts, as well as multi-model grayscale rollout and rollback capabilities. These can be built in-house, or you can leverage platforms like Baidu AI Cloud, which provide model services and supporting observability features to turn “multi-model A/B testing” into a team asset.

Conclusion: Let the Right People Choose the Right Model

Distilling the five steps into a decision checklist:

• Heavy Long-Document Analysis: Prioritize verifying the actual recall curve of large contexts before looking at benchmark scores.
• Heavy Code Production: Use a real repository as a private evaluation set, and maximize the weight of code sub-scores.
• Heavy Knowledge Q&A: Hallucination rate acts as a veto; composite scores take a backseat.
• Cost-Sensitive Scenarios: Focus on cache hit rate and output unit price, calculating “intelligence per dollar.”
• Heavy Agent Scenarios: Task completion rate and exception recovery in computer operations are hard thresholds.

Returning to the initial question: when a new model is released, is it a genuine breakthrough or just marketing? The answer lies neither in the press release nor entirely in third-party leaderboards, but in the ten hands-on test cases you design for your own business. Official benchmarks determine “whether to spend time testing it,” but your hands-on tests determine “whether to put it into the production environment.” Solidifying your evaluation process is the most composed stance you can take during the busy release season.