How to Scientifically Evaluate the True Capabilities of a Large Language Model
This article explains how to evaluate large language models beyond parameter counts and leaderboard scores. It covers MoE architecture, scaling efficiency, long-context usability, multimodal capabilities, evaluation methodology, coding and agentic performance, and deployment cost, with practical guidance for selecting models for real business scenarios.
Introduction: Why Evaluating LLMs Is Hard
As large language models enter a flagship competition stage, marketing around parameter counts, context lengths, multimodal abilities, and leaderboards is widespread. For developers, the real pain is not a lack of evaluation data, but too much data, confusing dimensions, and mismatch with business scenarios.
A model that ranks highly on a general leaderboard may still be unreliable for your codebase refactoring, long-document analysis, or frontend generation tasks. Instead of memorizing one model’s score, it is better to master an evaluation method from low-level architecture to engineering deployment.
1. Start with Architecture: MoE and the Real Meaning of Scaling Efficiency
Flagship models today commonly use Mixture of Experts (MoE) sparse architectures. Understanding this is the first step in evaluation.
Dense Models vs. Sparse Models
- Dense models: all parameters participate in each forward pass. Parameter count is strongly tied to compute consumption.
- Sparse models (MoE): the network is split into many expert subnetworks, and only a small subset is activated per inference.
For example, a flagship MoE model may have around 2.8 trillion (2800B) total parameters, contain hundreds to thousands of experts (for example, 896), but activate only a dozen or so experts (for example, 16) per token. This means:
Total parameter scale (knowledge capacity): ~2800BActivated parameters per inference (compute cost): only a small fraction of total
Why Scaling Efficiency Matters
More important than absolute parameter count is scaling efficiency: how much usable capability a model can turn into per unit of compute budget. Claims such as “several times higher scaling efficiency than the previous generation” generally refer to improved training and inference resource utilization, not simply more parameters.
Evaluation advice: Do not focus only on total parameters. Combine activated parameters, inference cost, and measured performance on target tasks. More parameters do not automatically mean higher per-call cost or better results.
2. Context Length: From Fitting More Text to Using It Effectively
Long context is a standard selling point for flagship models, with advertised windows ranging from hundreds of thousands to millions of tokens.
What a Million-Tokens Context Means
With a 1 million-token context, a single request can accommodate a large code repository, hundreds of pages of technical documents, or hours of meeting transcripts. This removes the hard constraint of truncation beyond the window.
Fitting More Does Not Mean Using It Well
When evaluating long-context capability, effective utilization matters more than the advertised length. Common challenges include:
- Attention decay: When information is in the middle of a very long context, models may effectively ignore it. This is known as the lost-in-the-middle phenomenon.
- Retrieval precision: Whether the model can locate and cite specific facts in long documents.
- Positional extrapolation: Whether inference remains stable beyond the training length.
To mitigate these issues, the industry introduces techniques such as improved attention mechanisms, attention residual connections, and positional encoding improvements so information flows more smoothly through deep networks and very long sequences.
Evaluation advice: Use Needle-in-a-Haystack-style tests. Place key information in long text and check whether the model can recall it accurately, including differences when the information is at the beginning, middle, or end.
3. Multimodal Understanding: Beyond Text
Native multimodal capability is another differentiator for flagship models. Native support means images and videos are encoded and understood inside the model, rather than being stitched together through external vision modules. Such bolt-on approaches often perform worse on fine-grained understanding.
Typical value includes:
- Interface-level coding: Reading design files, sketches, or screenshots and directly generating or modifying frontend code.
- Document and chart understanding: Extracting structured information from charts, scanned documents, and video frames.
- Multimodal reasoning: Performing question answering, summarization, and code generation over multimodal inputs.
Evaluation advice: Distinguish perception from reasoning. A model should not only identify what is in an image, but also complete complex tasks based on visual information, such as “reconstruct runnable page code from this UI screenshot.”
4. Evaluation Methodology: Avoid Being Misled by Leaderboards
This is the most overlooked but critical part of evaluation.
Common Evaluation Methods and Their Limitations
| Evaluation method | Description | Limitation |
|---|---|---|
| Composite intelligence index | Weighted score from third-party organizations across benchmarks | Weights are subjective and may not match your scenario |
| Arena-style blind tests | Users blind-vote; ELO-style rankings, such as topping WebDev leaderboards | Influenced by voter preferences; reflects subjective experience |
| Single benchmark tests | Specialized math, coding, and reasoning benchmarks | Risk of leaderboard gaming and overfitting |
| Custom business evaluation set | Validation with your own real data | Costly, but closest to real needs |
Core Principle: Custom Evaluation Sets Matter Most
Any third-party leaderboard should be used only as an initial screening reference. The final decision should be based on a small evaluation set built from your business data, covering typical tasks and edge cases and comparing candidate models side by side.
Recommended evaluation process:1. Sample 50-200 representative real business cases (including hard cases)2. Define quantitative scoring criteria (accuracy, pass rate, human preference)3. Run all candidate models with the same prompts and parameters4. Record stability, latency, and cost before making a decision
5. Coding and Agent Capabilities: Can It Complete Work With Less Supervision?
A major improvement in flagship models over previous generations is long-horizon task autonomy, also known as agentic ability.
When evaluating coding ability, consider three levels:
- Code generation correctness: Whether it produces compilable and runnable code from requirements.
- Long-horizon task execution: Whether it can work through large repositories, call terminal tools, and debug across multiple steps, rather than only outputting code snippets.
- Visual closed-loop capability: Whether it can “look at the interface and modify the interface,” read screenshots, run tests, find bugs, and fix them.
From technical demos to community tests, models with agent capabilities often show good aesthetic sense and high one-shot frontend success rates. This reflects mature planning, execution, and verification loops.
Evaluation advice: Design end-to-end tasks instead of single-point Q&A. For example, ask: “Generate a playable small game from this sketch, then test and fix issues yourself,” and observe how completely the model can proceed autonomously.
6. Cost and Engineering Deployment: Turning Capability into Productivity
Even the strongest model must fit into an affordable and maintainable engineering system.
Understanding API Pricing Models
Mainstream LLM APIs usually charge per token and separate several dimensions:
- Input token price (cache miss)
- Cache-hit price: Significantly lower than cache miss, used for repeated prefixes
- Output token price: Usually the highest
A typical price structure might be: cached input tokens around 2, cache-miss input tokens around 20, and output tokens around 100 per million tokens. Prompt Caching is key to reducing costs. By caching stable system prompts and long document prefixes, repeated calls can receive substantial discounts.
Deployment and Access
Mature model services usually provide multiple access forms: web, mobile, desktop clients, command-line tools (CLI), and standardized APIs. For engineering integration, models can be called through a unified inference interface without worrying about underlying scheduling.
Cloud Platform Support
In practice, models rarely run in isolation. They are embedded in a complete cloud-native system. Developers can use cloud platforms, such as Baidu AI Cloud’s general capability categories including model inference services, container platforms, object storage, logging, and monitoring and alerting, to handle model hosting, elastic scaling, call-chain monitoring, and cost accounting. This helps turn model capability into stable online services.
Evaluation advice: Include cost in the decision matrix. Compare models by the total cost of completing one business task, including token cost, retry cost, and latency cost, rather than unit price alone.
Conclusion: Return to Scenarios and Choose Rationally
A comprehensive evaluation of a large language model should not stop at surface numbers such as parameter counts and leaderboard scores. It should form a complete chain: architecture understanding, capability verification, methodology calibration, and engineering deployment.
Key conclusions:
- Architecture: Focus on MoE sparse activation and scaling efficiency, not just total parameters.
- Capability: Measure effective utilization for long context; distinguish perception from reasoning for multimodality.
- Methodology: Third-party leaderboards are only for initial screening; custom business evaluation sets are the final judge.
- Engineering: Use prompt caching to reduce costs, and use cloud platforms for hosting, monitoring, and operations.
When flagship models are close to the ceiling in overall capability while prices are more friendly, the real question is no longer “which model is the strongest,” but “which model best fits my scenario and can be stably supported by my engineering system.” With this framework, you can make verifiable technical choices in a fast-moving model ecosystem.
