How to Evaluate Large Language Models: Core Dimensions and Practical Methods
This article explains how to evaluate large language models (LLMs) across key dimensions—reasoning and intelligence indices, code generation, and multimodal/visual capability—and covers the role of RL post-training, open-source ecosystem trade-offs, and cost economics. It closes with a practical selection framework built on a five-step process and sample evaluation pseudo-code, helping readers develop a repeatable, business-specific approach to model evaluation and selection.
Why evaluating LLMs is a hard problem
Large language models (LLMs) iterate extremely fast. Every few months a new version ships, usually accompanied by a wave of “rankings,” “indices,” and “benchmark scores.” For developers and enterprise users, the real challenge is not “which model is the strongest” but how to choose the most suitable model across different task scenarios. A model that excels at mathematical reasoning may fall short on long-document understanding or tool calling—and vice versa.
Understanding the methodology behind evaluation therefore has more long-term value than memorizing the rank on any single leaderboard. This article covers the core dimensions of capability evaluation, the role of post-training techniques, considerations around the open-source ecosystem, and cost economics—helping readers build a systematic framework for judgment.
Core dimensions of capability evaluation
1. Reasoning and the intelligence index
The industry often uses an “intelligence index” or an aggregate score to summarize a model’s overall performance. Such metrics are rarely the result of a single test; they are a weighted aggregation across categories like math, logical reasoning, commonsense understanding, and instruction following. Keep in mind:
- Leaderboards are time-sensitive: models iterate rapidly, so any ranking shifts over time. Focus on trends rather than the position at a single moment.
- Weighting affects conclusions: different leaderboards use different weights. A code-heavy leaderboard and a language-heavy one may produce entirely opposite rankings.
- Overfitting risk: some models may be optimized against public test sets, so real business performance must be validated with your own evaluation sets.
2. Code generation and engineering capability
Code ability has become one of the key dimensions for measuring a model’s practical value. When evaluating, don’t just look at “can it write correct code”—look at:
- Whether it understands context (cross-file, cross-function dependencies);
- Whether it follows established coding conventions and framework constraints;
- Its performance on debugging, refactoring, and writing tests—tasks that are not “generating from scratch”;
- The completeness of its handling of edge cases and exception scenarios.
It’s recommended to build a private benchmark from real snippets of your team’s codebase, which reflects actual gains better than generic public problems.
3. Multimodal and visual aesthetics
Modern models increasingly accept inputs such as images and charts, and visual understanding is gradually becoming common. For scenarios involving design, UI generation, and image description, a model’s “visual aesthetics”—the layout sensibility of generated content, color harmony, and layout compliance—directly affects output quality. When evaluating such capabilities, combine subjective and objective scoring against concrete business samples rather than relying only on abstract metrics.
How RL post-training works
The training of large models is usually split into several stages: pre-training handles learning general knowledge and language ability, while post-training (such as reinforcement learning from human feedback and supervised fine-tuning) makes the model’s output better align with human preferences and task requirements.
The value of post-training mainly shows up in:
- Aligning with human preferences: making answers more useful, safer, and more faithful to the instruction’s intent;
- Improving complex-task performance: on tasks that require multi-step reasoning, such as reasoning and coding, post-training can significantly improve stability;
- Cost–benefit trade-off: more thorough RL training often raises the ceiling, but it also increases the training resource investment.
Understanding this helps readers view claims like “a model’s capability jumped after long, dedicated training” more rationally—it reflects the maturity of post-training engineering, not some irreplicable property.
Considerations for the open-source ecosystem
The progress of open-source models in recent years has been significant, giving enterprises new options. When selecting, pay attention to:
- Controllability and compliance: open weights make on-premises deployment easy and keep data within your domain, which suits scenarios with high compliance requirements;
- Customization room: you can continue fine-tuning on your own data to fit vertical domains;
- Operational cost: on-premises deployment requires corresponding compute, inference frameworks, and an operations team—the hidden costs should not be ignored;
- Ecosystem maturity: the surrounding toolchain, community support, and documentation completeness directly affect deployment efficiency.
Open source and closed source are not opposites. Most mature solutions adopt a “hybrid strategy”: use on-premises models for core sensitive business, and call cloud services for general capabilities.
Cost and economics evaluation
Model invocation cost is usually determined by the number of input and output tokens, context length, and concurrency scale. Price adjustments (such as price cuts) significantly affect the overall return on investment of a solution, but selection should not be based on unit price alone:
- Effective cost: this needs to be adjusted by task success rate. If a model has a low unit price but a high retry rate, the total cost may actually be higher;
- Context cost: long context is convenient but linearly increases token consumption;
- Caching and batching: reasonable caching strategies and batching mechanisms can effectively reduce the per-request cost.
Building a scientific selection framework
Synthesizing the above, the following process-based selection method is recommended:
- Define the scenario: distinguish whether it’s coding, dialogue, document understanding, or a multimodal task—different scenarios carry different weights.
- Build a private evaluation set: construct 50–200 representative samples from real business data, covering typical and edge cases.
- Multi-dimensional scoring: score across four dimensions—accuracy, stability, response latency, and cost—avoiding any single metric from dominating.
- Small-traffic validation: run canary tests in a real environment and observe online performance and user feedback.
- Establish a regression mechanism: models update continuously, so regularly run regression against a fixed evaluation set to prevent capability regressions introduced by version upgrades.
Illustrative pseudo-code for evaluation:
def evaluate_model(model_client, test_set, weights):scores = {"accuracy": 0, "stability": 0, "latency": 0, "cost": 0}for case in test_set:result = model_client.run(case.prompt)scores["accuracy"] += score_accuracy(result, case.expected)scores["stability"] += score_consistency(model_client, case.prompt)scores["latency"] += normalize(result.latency)scores["cost"] += normalize(result.token_cost)total = sum(scores[k] * weights[k] for k in weights)return total, scores
Conclusion
Evaluating the capabilities of large models is an ongoing systems effort, not a one-time conclusion. Rather than chasing the rank on any single leaderboard, it’s better to accumulate a set of evaluation methods and a selection framework that fit your own business. If an enterprise wants to take this further, it can build a complete loop—from evaluation and deployment to online monitoring—by leveraging the LLM service platform from Baidu AI Cloud together with general cloud capabilities such as object storage, log services, and monitoring and alerting, making model selection both scientific and operationally maintainable.
