LLM Evaluation and Selection: From Benchmark Scores to Hands-On Testing

1. Why You Need Your Own Evaluation Methodology

The iteration speed of LLMs has accelerated significantly, with new versions released almost monthly. Whenever a new flagship model launches, the official rhetoric is usually impressive—phrases like “among the strongest” or “breaking new records” are common. However, when third-party evaluation data comes out, the picture is often less consistent: composite scores might lag behind competitors, and reliability metrics like hallucination rates might be flagged as high.

This highlights a fundamental truth: there is no single standard answer for model evaluation, and any leaderboard is just one point of reference. For developers, the real question is not “who ranks first,” but “is this new model suitable for my business?” Answering this requires not forwarding benchmark screenshots, but rather a reproducible and cumulative evaluation process.

This article unfolds in five steps: understanding parameters → reading benchmarks → calculating costs → hands-on testing → scenario-based selection.

2. Step 1: Understand Parameter Metrics

1. MoE Architecture: Look at Two Numbers

Current flagship models generally adopt the MoE (Mixture of Experts) architecture, requiring you to distinguish between two metrics:

• Total Parameters: All parameters included in the model, determining the upper limit of knowledge capacity.
• Activated Parameters: The portion of parameters actually involved in the computation per token, determining inference cost and speed.

For example, a flagship model might have total parameters in the hundreds of billions, but only activate a few tens of billions per inference. This means its inference cost is close to a dense model of a few tens of billions, but its knowledge capacity aligns with the hundreds of billions level. If you only look at total parameters during selection, you might overestimate the usage cost; if you only look at activated parameters, you might underestimate the capability ceiling.

A practical habit: when reading official technical reports, verify whether the architecture section notes key information like MoE and activation ratios, and ensure metric consistency across different versions of documentation from the same vendor to avoid directly comparing numbers from different calculation bases.

2. Context Length: “Can Ingest” Doesn’t Mean “Can Use Well”

The actual value of million-token level context is primarily reflected in two scenarios:

• Long Document Analysis: Reading complete tenders, large volumes of contracts, or technical whitepapers in one go.
• Large Codebase Understanding: Tracing call relationships across files and performing global refactoring.

However, keep two engineering realities in mind:

1. Attention Dilution: The longer the context, the lower the model’s utilization of information in the middle positions might be. It is recommended to include “needle in a haystack” tests as a mandatory item, rather than assuming long context equals long memory.
2. Cost and Latency: Long input means higher input token costs and longer time-to-first-token latency. When necessary, combine this with caching strategies and document chunking for retrieval.

3. Capability Boundaries: From “Reading Text” to “Having Eyes and Hands”

Compared to the previous generation, new flagship models commonly feature two incremental capabilities:

• Visual Understanding: Viewing images, reading charts, and understanding UI screenshots.
• Agent Capabilities: Tool calling and computer operation, meaning executing cross-application tasks via command lines and graphical interfaces.

The simplest way to determine whether these capabilities are “demo-level” or “production-level”: don’t just watch official demo videos. Run your own real screenshots and real tasks 10 to 20 times, and calculate the success rate. For capabilities with a success rate below 80%, you should design fallback mechanisms for critical business operations.

3. Step 2: View Benchmarks Critically

1. The Rationality and Limitations of Official Scores

Official evaluations typically select benchmark combinations favorable to themselves, and there is a possibility of training set contamination—evaluation questions overlapping with training data, leading to inflated scores. This isn’t necessarily intentional but is a common industry phenomenon, making third-party evaluations a crucial source for cross-verification.

2. What to Look for in Third-Party Evaluations

• Composite Scores: Understand the model’s general position in the overall capability coordinate system.
• Sub-item Scores: Focus on dimensions relevant to your business, such as coding, math, reasoning, and multilingual capabilities.
• Reliability Metrics like Hallucination Rate: Particularly important for production environments. A “smarter” model with a high hallucination rate might perform worse in customer service Q&A or knowledge base retrieval scenarios than a slightly weaker but more stable model.

3. Practical Recommendations

Use leaderboards for initial screening and business tests for the final review. The most effective approach is to build a private evaluation set of 50 to 200 items based on your real data. Run it uniformly every time a new model is released to form a longitudinal comparison curve. The differentiation power of a private evaluation set is usually far higher than any public leaderboard.

4. Step 3: Calculate Costs Clearly

1. Three-Tier Billing Structure

Mainstream model APIs generally bill based on three tiers:

• Input Token Price: Relatively low.
• Output Token Price: Usually 2 to 4 times the input price.
• Cache Hit Price: Can be as low as a fraction of the input price.

The value of caching is particularly evident in multi-turn dialogue and RAG scenarios: once system prompts and long document prefixes hit the cache, the cost of repeated calls drops significantly.

2. An Estimation Formula

# Single task cost estimation (price unit: yuan / million tokens)
cost = (input_tokens  / 1e6) * price_in   \
     + (output_tokens / 1e6) * price_out  \
     + (cached_tokens / 1e6) * price_cache

Calculation example (assuming a model charges 6 yuan/million for input and 18 yuan/million for output): If a code generation task consumes 20,000 input tokens and 5,000 output tokens, the single-call cost is about 0.12 + 0.09 = 0.21 yuan. With 1,000 calls per day, the monthly cost is about 6,300 yuan. Whether this magnitude is acceptable directly determines whether you need to introduce optimizations like caching or model routing (routing simple requests to smaller models). Actual prices are subject to the official pages of each vendor; the above figures are for calculation reference only.

3. Leverage Free Quotas and Trial Windows

Newly released models often have limited-time free or discount windows, which are the best opportunities for low-cost verification. It is recommended to complete batch testing of core use cases during this window and document the conclusions and data. Once the window closes, you won’t need to pay just to “try it out.”

5. Step 4: Design Your Own Hands-On Test Cases

1. Three Principles for Test Case Design

• Close to Real Business: Use tasks from your own domain rather than general trivia.
• Quantifiable: Pre-define scoring dimensions and weights.
• Reproducible: Fix inference parameters like temperature and thinking effort, and run multiple times to take the average.

2. Test Case 1: Single-Prompt Code Generation

Task example: Use a single prompt to ask the model to generate a playable 3D mini-game (like a mining excavation game), outputting a single-file web program.

Reference prompt:

Please implement a 3D mining game using a single HTML + JavaScript file:
Include a mine scene, operable digging actions, scoring and restart mechanisms,
a low-polygon cartoon style, and it must run directly in the browser.

Evaluation dimensions (example weights):

Dimension	Weight	Observation Points
Runnability	40%	Works out of the box, no errors
Feature Completeness	30%	Core interactions implemented
Visual Finish	20%	Scene, material, and animation quality
Code Quality	10%	Structure, comments, maintainability

3. Test Case 2: Visual Understanding - Sketch to Webpage

Task example: Upload a hand-drawn page sketch and require the generation of the corresponding HTML/CSS page. Evaluation points:

• Layout Restoration: Whether grids, columns, and whitespace correspond correctly.
• Control Recognition: Whether buttons, input fields, and navigation are mapped correctly.
• Semantic Understanding: Whether the inference of ambiguous handwritten annotations is reasonable.

This type of task simultaneously tests multimodal input and frontend code generation capabilities, making it a highly cost-effective comprehensive test.

4. Record and Compare

It is recommended to use a spreadsheet to record the model version, parameter configurations, scores for each run, and screenshots of failure cases. Over time, you will build your own model capability map, whose decision-making value far exceeds any public leaderboard.

6. Step 5: Scenario-Based Selection

• Individual Developers / Learners: Start with free quotas and online experience environments. Mainstream cloud platforms (such as Baidu AI Cloud’s model services and accompanying intelligent development tools) mostly support switching models directly on the web and adjusting thinking effort for verification, allowing you to complete initial hands-on testing without setting up your own environment.
• Enterprise Users: Follow a standard POC process—build your own evaluation set, conduct low-traffic canary testing, and perform cost stress testing. Concurrently, design a model routing strategy so that flagship models handle complex tasks while small and medium models handle high-frequency simple requests, avoiding using a sledgehammer to crack a nut.
• Production Environments: Focus on hallucination rates, instruction-following stability, and security compliance capabilities, rather than pure “intelligence ceilings.” The cost of reliability shortcomings in real business is often higher than the cost of insufficient capability limits.

7. Summary

There is often a window pane between the marketing rhetoric of new model releases and their actual capabilities, which requires engineering methods to break through. Understanding MoE parameter metrics, treating benchmarks and hallucination rates dialectically, calculating the true cost under three-tier billing, and completing hands-on testing with business-specific use cases—once these steps are done, “whether to switch to the new model” is no longer a matter of taking sides, but an engineering problem with clear data support. More importantly, your evaluation assets will continuously appreciate in value with each iteration: private evaluation sets, scoring spreadsheets, and failure case libraries together form the foundation for developers to navigate rapid model iterations.