How to Evaluate Large Language Models: Performance, Cost, and Selection in Practice
Every new model release splits the community into two camps: one that reads parameter counts and leaderboard rankings as the whole story, and one that reads nothing but the API price list. This article lays out a more rigorous approach. It covers the five dimensions of a practical evaluation framework, why asymmetric input and output token pricing makes "cheaper" misleading, how total cost of ownership should be calculated, how to build a private evaluation set instead of trusting public benchmarks, how tiered model routing and semantic caching cut spend, and how observability plus regression evaluation keep a model choice honest after launch. If you need to choose a model on evidence rather than hype, this is the decision structure to work from.
Why Parameter Count and Price Alone Tell You Nothing
Every time a new generation of large models ships, the community splits into two camps fast. One watches parameter counts and leaderboard positions and treats scale as destiny. The other watches API prices and treats the lowest rate as the right answer. Both judgments are too narrow.
Parameter count describes theoretical capacity. Real performance in production also depends on training data quality, post-training alignment, inference optimization, and a long list of other factors. A larger model with no clear advantage on the tasks you actually run is dead weight. Low unit price does not mean low total cost either: if you pick a model that misses your quality bar to save money, the retries and manual rework that follow usually cost far more than the API fees you saved.
Evaluating a model therefore comes back to one question: can it complete your specific tasks reliably, at a cost you can accept?
A Five-Dimension Evaluation Framework
The first step in a scientific selection process is a multi-dimensional framework, rather than letting a single metric lead you around. Consider these five dimensions together:
- Output quality: accuracy and consistency on your actual tasks, whether that is question answering, summarization, code generation, or structured extraction.
- Inference performance: time to first token, generation throughput, and response behavior on long inputs.
- Cost structure: not just unit price, but the input/output split and how context length drives spend.
- Capability boundaries: supported context window, multimodality, function calling, structured output support, and so on.
- Engineering readiness: API stability, rate limiting policy, observability support, and the completeness of ecosystem and documentation.
Together these five dimensions form a three-dimensional coordinate system. A weakness in any one of them tends to get amplified once you move to production scale.
Reading the Token Billing Model: Why a Price Cut Is Not Automatically Savings
Most large model APIs bill by token, pricing input (prompt) and output (completion) tokens separately. A few details are easy to miss:
- Input and output pricing are usually asymmetric. Generating output tokens costs more than reading input tokens, so a workload that produces long text has a completely different cost profile from one that mostly reads long context.
- Long context carries hidden cost. Stuff a large system prompt or the full conversation history into every request, and input tokens grow linearly. Even a low unit price can add up to a substantial bill.
- Caching and batching matter. Some services offer context caching or asynchronous batch calls that significantly reduce cost for repeated prefixes.
- Retries and failed requests count. In unstable conditions, repeated retries push actual consumption above theoretical consumption.
Cost evaluation should therefore take a total cost of ownership (TCO) view: token consumption, retry rate, operations, and human effort belong in the same calculation, not just the number on the price sheet. A claim that “this generation is cheaper than the last” only holds up when you recompute it against your real call pattern.
Running a Sound Performance Evaluation: From Public Benchmarks to a Private Eval Set
Public benchmarks are a reasonable starting point for understanding model capability, but they have clear limits: test data may already be in the training set, the task distribution may not match your business, and the scoring methodology varies from one benchmark to the next. A reliable evaluation has to be built on a private evaluation set.
A pragmatic workflow looks like this:
# Minimal viable private evaluation workflowimport jsondef build_eval_set(samples):# Each sample carries: input prompt, expected output, scoring criteriareturn [{"input": s["prompt"], "expected": s["label"]} for s in samples]def run_eval(model_client, eval_set, scorer):results = []for item in eval_set:resp = model_client.generate(item["input"])score = scorer(resp, item["expected"]) # exact match / semantic similarity / LLM-as-judgeresults.append({"input": item["input"], "output": resp, "score": score})avg = sum(r["score"] for r in results) / len(results)return avg, results# Key point: the eval set should cover real traffic distribution and edge cases
While evaluating, record quality metrics (accuracy, semantic similarity) and performance metrics (latency, throughput) at the same time, then put cost into the same comparison table as a third dimension. That gives you an effectiveness–performance–cost basis for the decision.
Cost Optimization Architecture: Tiered Model Routing and Caching
In a real system, no single model is optimal for every scenario. The mature approach is a tiered dispatch architecture:
- Tiered model routing: send simple, high-volume, fault-tolerant tasks (intent classification, format conversion) to a smaller model, and reserve the stronger model for complex reasoning. This balances cost and quality.
- Semantic caching: cache results for identical or highly similar requests instead of calling the model again.
- Context trimming and summary compression: summarize long conversation history and keep only the relevant information, which lowers input tokens.
- Batching and asynchrony: run offline, non-real-time work as asynchronous batches to spread unit cost.
# Route by task complexitydef route_task(task):if task["complexity"] == "simple":return small_model_client # low latency, low costreturn large_model_client # high capability, called on demand
The core idea is to use the cheapest model for everything it can handle, and spend money on expensive capability only where a request genuinely needs it.
Engineering Practices: Observability and Continuous Evaluation
Model selection is not a one-time decision but an ongoing process. Once you ship, build a proper observability stack:
- Monitoring and alerting: dashboards and threshold alerts for call volume, latency, error rate, and token consumption.
- Logging and tracing: record inputs, outputs, and timing per request so you can locate problems and attribute cost.
- A/B testing and canary releases: put a new model behind a small slice of traffic first, validate it against real data, then roll it out fully.
- Regression evaluation: after a model or prompt update, rerun a fixed evaluation set to catch quality regressions.
On cloud platforms, these capabilities are typically assembled from general-purpose building blocks such as monitoring and alerting, logging services, message queues, object storage, and model-serving products, which closes the loop from evaluation through deployment to operations in environments such as Baidu AI Cloud.
Conclusion
Judging a new-generation model by parameter count alone is wrong, and so is judging it by price alone. The professional approach is a combined effectiveness–performance–cost framework, a private evaluation set instead of blind trust in public leaderboards, total cost of ownership instead of a narrow reading of unit price, and then tiered routing plus observability to make the choice stick in production. Answer those questions clearly, and whether a new model genuinely earns its reputation or is simply expensive becomes obvious.
