LLM Model Selection: A Practical Methodology for Evaluating Accuracy, Latency, and Token Cost
Choosing a large language model (LLM) is rarely as simple as picking the highest leaderboard score. This article lays out a reproducible methodology for evaluating LLMs across four measurable dimensions—accuracy, average latency, average token consumption, and cost per 1K calls—while also capturing three easily overlooked factors: latency distribution, stability, and reproducibility. It explains how to build a stratified, traffic-weighted test set, how to compute cost from token-level pricing, and how to run reproducible evaluation scripts. Drawing on anonymized data from a typical model version upgrade, it shows why ranking jumps, headline enhancements, and raw speed gains can be misleading, then offers a five-step path from leaderboard scores to real business decisions, plus practical cost-optimization techniques such as multi-model routing, result caching, context compression, and streaming output. The takeaway: build your own evaluation capability instead of chasing the strongest cheap model.
1. Background: Capabilities Keep Improving, Budgets Keep Tightening
Over the past two years, the capability curve of large language models has shown a clear generational leap. This has created a practical contradiction: the unit price of top-tier models has stayed stubbornly high, while the actual call volume of most businesses grows linearly. A mid-sized intelligent Q&A service—averaging a million requests a day and consuming two to three thousand tokens per request—can rack up an annual cost that exceeds the entire model budget of many teams. Add multi-turn, long-context scenarios like code completion and agent tool calls, and the cost is amplified further.
This raises a question developers keep coming back to: if you can’t afford the top-tier closed-source models, are more affordable general-purpose LLMs actually good enough? The question itself contains three sub-questions—does capability clear the business threshold, can speed meet the interaction experience, and does the unit cost fall within an acceptable range. Miss any one of them and the selection conclusion falls apart.
The problem with most “benchmark” posts online is that they give conclusions without methods, report a single overall score without variance, and compare a single generation without calculating total cost. This kind of information has very limited value for decision-making. This article tries to turn the process into something reproducible and engineering-grade.
2. Breaking Down the Evaluation Dimensions: Four Core Metrics and Three Hidden Costs
A solid model evaluation should cover at least four directly measurable core metrics:
- Accuracy: the correctness rate on a given test set, reflecting the capability ceiling;
- Average latency: the end-to-end latency of a single call, reflecting interaction experience;
- Average token consumption: the number of tokens consumed per call, reflecting “verbosity” and context efficiency;
- Cost per 1K calls: determined jointly by unit price and token consumption, reflecting cost-effectiveness.
But these four are not complete. Three hidden costs also need to be collected during evaluation design:
| Hidden dimension | What it means | Consequence of not measuring it |
|---|---|---|
| Latency distribution | Time to first token (TTFT) and P95/P99 tail latency | Averages look good, but users perceive lag |
| Stability | Timeout rate, rate-limiting rate, format-parsing failure rate | You need retry logic, which effectively doubles the real cost |
| Reproducibility | Variance of results across multiple samples of the same question | Single-run comparisons are unreliable |
A common mistake is to treat “average latency” as the only performance metric. In streaming-output scenarios, what users perceive is the time to first token; in non-streaming batch scenarios, throughput is determined by the distribution of complete-response latency. The two place completely different demands on the model service.
3. Building the Test Set: Stratified Sampling and Scenario Weighting
The credibility of an evaluation conclusion depends roughly 80% on the test set. Build it according to the following principles:
- Stratify by scenario: group real business traffic by intent—for example, “law and administration,” “education,” “finance,” “healthcare and mental health,” “reasoning and math,” “language and instruction following,” and “agents and tool calls.” Each category is an independent scoring dimension; you cannot rely on an overall score alone.
- Weight by business share: the overall score should be a weighted sum of each dimension’s score, with weights coming from real traffic share rather than being distributed evenly. If legal documents account for 3% of traffic, the weight of the legal score should not be 14%.
- Ensure sufficient scale: a ten-thousand-scale question set (for example, 15,000 items) lets the error of each single dimension converge into a statistically acceptable range; results below a thousand items fluctuate too much to support version-to-version comparison.
- Prevent contamination: test questions must not come verbatim from public leaderboards, otherwise you are testing memorization rather than reasoning. Keep a portion of de-identified real online requests as a “private set.”
- Fix the judging criteria: use exact match for multiple-choice questions, and rule-based validation or double-blind human scoring for generation questions. The scoring rubric must be versioned, otherwise cross-version comparison is invalid.
4. Metric Calculation and Reproducible Evaluation Scripts
Cost estimation should not be guesswork; it should be calculated by billing input and output tokens separately and then summing them:
# Example cost estimation for a general-purpose LLM call# (unit prices must follow the service provider's published rates)def estimate_cost(avg_input_tokens, avg_output_tokens,in_price_per_1k, out_price_per_1k, calls_per_1k=1000):input_cost = avg_input_tokens * in_price_per_1k / 1000output_cost = avg_output_tokens * out_price_per_1k / 1000return round((input_cost + output_cost) * calls_per_1k, 2)# Cost per 1K callsprint(estimate_cost(2100, 750, 0.002, 0.008)) # illustrative: input/output priced separately
On the evaluation execution side, always record the raw request, response, token usage, and latency of every call, and follow three engineering conventions:
import time, statistics, hashlib, jsondef run_once(client, prompt, model_params, temperature=0.0):t0 = time.perf_counter()first_token_at = Nonefor chunk in client.stream(prompt, **model_params):if first_token_at is None:first_token_at = time.perf_counter()text = chunktotal = time.perf_counter() - t0return {"prompt_hash": hashlib.md5(prompt.encode()).hexdigest(),"ttft_ms": round((first_token_at - t0) * 1000, 1),"total_ms": round(total * 1000, 1),"usage": {"prompt_tokens": 0, "completion_tokens": 0},"text": text,"params": {**model_params, "temperature": temperature},}# When summarizing, output mean + P95 + standard deviation, not a single meandef summarize(samples):ordered = sorted(s["total_ms"] for s in samples)p95 = ordered[int(len(ordered) * 0.95)]return {"avg_ms": round(statistics.mean(ordered), 1),"p95_ms": p95,"stdev_ms": round(statistics.stdev(ordered), 1),}
Three key points: keep the temperature parameter fixed (0 is recommended, to ensure reproducibility), keep concurrency fixed (otherwise latency data is not comparable), and run each test at least three rounds and take the mean.
5. Case Data: Four Kinds of Change in a Version Upgrade
Take the measured data from a typical version upgrade of a general-purpose LLM as an example (the data is de-identified and used only to illustrate the analysis method). The old version is labeled A and the new version B:
| Metric | Version A | Version B | Change |
|---|---|---|---|
| Overall accuracy | 66.6% | 68.7% | +2.1pp |
| Overall ranking | 20th | 10th | ↑10 places |
| Average latency | 98s | 59s | -39.8% |
| Average token consumption | 2,944 | 2,851 | -3.2% |
| Cost per 1K calls | 38.8 CNY | 37.6 CNY | -3.1% |
Breaking it down by dimension, the changes show clear divergence:
- Law and administration: 72.7% → 80.3%, up 7.6 percentage points, the largest gain across all dimensions;
- Education: 41.2% → 46.9%, up 5.7 percentage points, but the absolute score is still a weak spot;
- Healthcare and mental health: up 3.7 percentage points to 77.6%;
- Reasoning and math: up 3.6 percentage points to 65.8%;
- Finance: up 0.8 percentage points to 77.7%, essentially flat;
- Language and instruction following: 66.2% → 64.6%, down 1.6 percentage points;
- Agents and tool calls: 72.7% → 69.9%, down 2.8 percentage points.
This data yields three conclusions with direct implications for selection.
6. Three Conclusions That Are Easy to Misread
First, a jump in ranking does not mean a broad capability improvement. Moving from 20th to 10th looks like a huge leap, but if overall accuracy rose only 2.1 percentage points, the more likely explanation is that competitors in the same period either stagnated or regressed. Ranking is a relative quantity that drifts with the comparison group, so treating it as a measure of capability growth is a mistake. Selection decisions should anchor on the gap between absolute accuracy and the business threshold, not on leaderboard position.
Second, the improvement directions emphasized by the vendor can conflict with per-dimension measured results. The most alarming finding in the case is that “agents and tool calls” dropped 2.8 percentage points. Improvements in agent capability often come from longer context windows, clearer tool-calling format conventions, and stronger planning for search-type tasks—changes that look impressive in demos but do not necessarily score higher on standardized evaluation sets, and can even lower tool-selection accuracy because the model emits longer reasoning processes and introduces more intermediate steps. This is exactly why a spec like “context window expanded from 128K to 200K” must be validated on a self-built test set before it can be trusted in production.
Third, speed optimization is the biggest real payoff, and it propagates into cost. Dropping average latency from 98s to 59s—an improvement of nearly 40%—is severely underrated. For interactive products, the difference between 60s and 100s is the difference between “usable” and “unusable.” At the same time, lower latency reduces the number of timeout retries, and every timeout retry is a wasted token charge. This is why, even though tokens dropped only 3.2%, cost per 1K calls fell 3.1% alongside improved stability—real savings on the bill are usually larger than what you would calculate directly from unit prices.
7. From Leaderboard Scores to Business Selection: A Five-Step Method
- Define the business threshold: set the minimum acceptable accuracy for each dimension and the latency ceiling (for example, TTFT ≤ 2s, complete-response P95 ≤ 30s). Anything that fails is out and does not enter cost comparison.
- Compute a traffic-weighted score: calculate the overall score using the stratified weighting method from Section 3, rather than citing a third-party leaderboard’s overall score.
- Build a cost model: fold in token unit prices, average consumption, timeout-retry rate, and cache hit rate to compute the combined cost of “1,000 successful business calls.”
- Validate with a small-traffic canary: run a 5%–10% canary and observe accuracy and stability under real distributions, focusing on the long tail and format-error rate.
- Establish a regression baseline: rerun the same private test set after every model upgrade or prompt change to prevent “upgrade-as-regression.”
8. Engineering Practices for Cost Optimization
As long as capability clears the bar, there is usually more room to optimize cost than to switch models.
Multi-model routing: split requests by difficulty—send simple intents to a low-cost model and complex reasoning to a high-capability model.
def route(request):if request.token_count > 8000 or request.intent in ("code_review", "math_proof"):return "high_capability_model"if request.intent in ("faq", "rewrite", "classify"):return "low_cost_model"return "default_model"# Escalate based on confidence: call the high-capability model when the low-cost model has low confidencedef call_with_escalation(req, predict, low, high, threshold=0.75):pred, conf, usage = predict(req, low)if conf >= threshold:return pred, usagepred2, usage2 = predict(req, high)return pred2, usage + usage2
Result caching: apply semantic-level caching to high-frequency, idempotent requests; a cache hit is zero cost. The cache key should include hashes of the model version and prompt version to avoid returning stale results after an upgrade.
Context compression: a long context window does not mean stuffing all history into the request. Effective practice is to keep a summary plus retrieval of key snippets plus structured trimming of tool return values, which can significantly cut input tokens.
Streaming output: this both improves the first-token experience and makes it easy to set server-side timeout circuit breakers, avoiding long-tail requests that drag down overall latency.
Observability: turn token consumption, latency, failure rate, and cache hit rate into monitoring and alerting metrics, with drill-down by business line and model version. Evaluation data and analytics reports can be hosted on object storage, evaluation jobs can run on a container platform, and metrics and logs can feed into logging and monitoring/alerting services, forming an “offline evaluation + online observation” closed loop. These capabilities map to standard categories in general-purpose cloud products such as those offered by Baidu AI Cloud, and can be combined as needed.
9. Conclusion and a Selection Checklist
Faced with a budget constraint that rules out the top-tier closed-source models, the rational path is not to hunt for “the strongest cheap model” but to build your own evaluation capability. The case data shows that this version delivered a passing result on both accuracy and speed, with a modest cost reduction, so the improvement in cost-effectiveness holds. But the regression in agents and tool calls is a reminder that behind an overall leaderboard score, capabilities on a business-critical dimension can quietly degrade.
When putting a selection into practice, check the following items one by one:
- Do you have a self-built test set weighted by business traffic, with more than ten thousand items and none taken verbatim from public leaderboards?
- Do you measure accuracy, average latency, P95 tail latency, token consumption, and cost per 1,000 successful calls at the same time?
- Have you fixed temperature, concurrency, and prompt version to ensure reproducible cross-version comparison?
- Have you set a separate accuracy regression threshold for the agent/tool-call scenario?
- Have you implemented the three cost-control measures—routing, caching, and context compression?
- Have you established an automated regression and canary mechanism after model upgrades?
Get these six right, and the discussion of model selection shifts from an unanswerable question like “which model is stronger” to a quantifiable, verifiable engineering question: “how does my business clear the threshold within a given budget?”
