How to Evaluate and Select Large Language Models: Beyond Leaderboard Scores
As top Chinese large language models converge to near-identical scores on general capability leaderboards, this article explains why leaderboard rankings alone are no longer a reliable basis for model selection. It covers how general benchmarks are structured, how to interpret small score gaps, how to build a private evaluation set tailored to your business, and which non-accuracy factors—cost, latency, integration, security, and compliance—belong in a complete evaluation framework. Includes a practical batch evaluation script example and guidance on leveraging cloud platform infrastructure.
Why Leaderboard Scores Alone No Longer Decide Model Selection
A Leaderboard That Signals a New Phase of Competition
A third-party evaluation organization recently published a new round of benchmark results for general-purpose Chinese language models. The headline finding: several flagship domestic models scored nearly identically on overall capability, with gaps between top models within a fraction of a point—a clear pattern of “tied leadership.”
This carries an important signal: competition among Chinese large language models has moved out of the early “generational gap” phase and into a new stage of tightly contested, same-generation competition.
What does this mean for developers? Simply put, the old selection logic of “just pick the strongest model” is breaking down. When the differences between top models on general benchmarks shrink to the level of statistical noise, choosing a model based on total leaderboard scores alone actually increases risk. What truly determines real-world success is how a model performs in your specific business scenarios—along with inference cost, response latency, context length, tool-calling capability, and how easily it integrates with your existing engineering stack.
This article does not focus on any single model or vendor. Instead, it addresses a general technical question: how to evaluate and select large language models in a scientifically sound way, covering both evaluation methodology and engineering practice.
Understanding General Capability Benchmarks: The Methodology Behind the Scores
What Dimensions Do General Benchmarks Cover?
Mainstream Chinese LLM evaluation frameworks typically assess models across the following dimensions:
- Language understanding and generation: reading comprehension, semantic reasoning, summarization, and text continuation or rewriting—basic tasks that test how well a model handles the characteristics of the Chinese language.
- Knowledge and reasoning: commonsense Q&A, academic knowledge, mathematical reasoning, and logical reasoning, testing knowledge breadth and multi-step reasoning ability.
- Coding: code generation, code completion, bug fixing, and code explanation—one of the dimensions developers care about most.
- Instruction following and alignment: whether the model accurately interprets complex instructions, respects output format constraints, and appropriately refuses improper requests.
- Long-context handling: long-document Q&A and consistency across multi-turn conversations, evaluating stability over large context windows.
How to Interpret “Fraction-of-a-Point” Gaps
When multiple models differ by less than one point overall, keep the following statistical realities in mind:
- Evaluation sets are limited in size. Most public benchmarks contain thousands to tens of thousands of samples. A fraction-of-a-point gap often falls within the margin of error and cannot establish a meaningful capability difference.
- Per-dimension variance is larger. Two models with nearly identical total scores may show clear strengths and weaknesses in specific dimensions. One model might excel at math while being mediocre at coding; another could be the reverse.
- Benchmark contamination. If benchmark questions or their variants appear in a model’s training data, scores will be inflated. The industry mitigates this by regularly refreshing questions and using private evaluation sets.
Therefore, the proper use of a leaderboard is to narrow down your candidate list, not to make the final decision.
Enterprise Model Selection: Build Your Own Evaluation System
Why Private Evaluation Is Necessary
Public leaderboards use general-purpose questions, but enterprise workloads have strong domain characteristics: financial applications care about compliance-aware phrasing and numerical accuracy; customer service cares about context consistency across multi-turn dialogue; engineering teams care about code generation and repository-level understanding. General benchmarks cannot cover these differentiated needs, so building an evaluation set that reflects your own business is a critical step in model selection.
Best Practices for Building an Evaluation Set
A practical business evaluation set should meet the following criteria:
- Realistic data: sample from desensitized production logs rather than fabricating questions;
- Coverage of common and edge cases: include high-frequency routine queries as well as boundary and error-prone cases;
- Verifiable answers: favor questions with clear ground-truth answers or programmatic verification to reduce subjective scoring overhead;
- Version control: keep updating the evaluation set as the business evolves to prevent models from effectively “memorizing” the questions.
An Illustrative Evaluation Script Framework
Below is a simplified batch evaluation script framework showing how to organize the evaluation pipeline (the interface portion is illustrative pseudocode):
import jsondef load_eval_dataset(path):with open(path, "r", encoding="utf-8") as f:return [json.loads(line) for line in f]def call_model(prompt, model_config):# Illustrative interface: connect to your model inference service in practice# Note: control the temperature parameter; use a low value in evaluation# scenarios to ensure reproducibilityresponse = inference_service.generate(prompt=prompt,temperature=0.1,max_tokens=1024,)return response.textdef score_answer(prediction, reference):# Choose based on task type: exact match / keyword hit /# semantic similarity / manual spot checksreturn 1.0 if prediction.strip() == reference.strip() else 0.0def run_evaluation(dataset_path, model_config):dataset = load_eval_dataset(dataset_path)results = []for item in dataset:pred = call_model(item["question"], model_config)score = score_answer(pred, item["answer"])results.append({"id": item["id"], "score": score})avg = sum(r["score"] for r in results) / len(results)return avg, results
In real-world engineering, you will also need concurrent request handling, retry logic on failures, result persistence, and per-dimension statistics. For open-ended questions, introduce manual spot checks or a hybrid mechanism combining model-as-judge scoring with human review.
Beyond Capability Scores: What Else to Evaluate
Model selection is a multi-objective decision problem, and general capability is only one variable. A complete evaluation framework should also include:
Cost and Performance Metrics
- Inference cost: under token-based pricing, estimate the cost per request using your business’s average input and output lengths, then multiply by projected call volume for a monthly budget;
- First-token latency and overall throughput: for real-time interactive scenarios such as customer service chatbots or code completion, first-token latency directly affects user experience;
- Concurrency capacity: whether the service remains stable under peak request volume, and what its rate-limiting policies look like.
Engineering Integration
- API compatibility and stability: whether standard invocation protocols are provided, how complete the SDK is, and how API versions evolve;
- Context window: whether it meets the needs of long-document and multi-turn conversation scenarios;
- Function calling and structured output: whether the model reliably produces structured results such as JSON, which is essential for building agent-style applications;
- Private and hybrid deployment options: for data-sensitive industries, whether the model can be deployed in your own environment is often a hard requirement.
Security and Compliance
This includes content safety filtering, guarantees that data stays within your domain, and logging and auditing capabilities. In highly regulated industries such as finance, healthcare, and government, this criterion can be an outright disqualifier.
Leverage Cloud Platform Capabilities to Reduce Evaluation and Integration Costs
For most teams, building model evaluation and inference infrastructure from scratch is expensive. A pragmatic path is to rely on mature cloud platforms for general-purpose capabilities—for example, using object storage to manage evaluation datasets and result files, container platforms to deploy evaluation jobs and inference services, message queues to decouple batch evaluation tasks, and log services with monitoring and alerting to track the quality of production model calls. Baidu AI Cloud, for instance, offers LLM platform capabilities covering model invocation, application orchestration, and effectiveness observability, allowing teams to quickly build their own evaluation pipelines and focus their effort on building business-specific evaluation sets rather than reinventing infrastructure.
One point deserves emphasis: regardless of which platform you use, ownership of the evaluation set and the design of evaluation criteria should remain in the hands of your own team—this is the key to keeping your selection decisions free from vendor lock-in.
Conclusion: Leaderboards Are the Starting Point, Scenarios Are the Destination
The “tied for first” landscape among Chinese LLMs fundamentally reflects rising technical maturity, which is good news for users: more excellent models to choose from, more room for negotiation, and greater flexibility to switch vendors. At the same time, it raises the bar for technical decision-makers—shifting from “picking a model by its score” to “evaluating a model with a system.”
To summarize the core recommendations of this article:
- Use public leaderboards to narrow your candidate pool, not as the final decision basis;
- Build a private evaluation set from real business data and maintain it with continuous versioning;
- Incorporate cost, latency, integration capability, and security compliance into a unified multi-objective evaluation framework;
- Make good use of cloud platform infrastructure to reduce the engineering cost of evaluation and integration.
Only by following this approach can you retain control over model selection amid the rapid iteration of large language models.
