## LLM API Selection: A Six-Dimension Evaluation Framework and a Practical Decision Guide
Choosing the right LLM API is a constrained optimization problem, not a leaderboard lookup. This article breaks model capability into six measurable dimensions — price, context window, reasoning, coding, Chinese quality, and latency — and explains how to turn each one into reproducible metrics. It covers cost-per-task calculation with real token math, how to measure effective context length with a multi-needle probe, why pass@1 and regression rates matter more than raw coding benchmarks, how to build an offline evaluation set with automated and LLM-based grading, and how an adapter layer plus tiered routing lets you avoid the false choice between model families. Includes weighting examples, evaluation code, caching design, and a final selection checklist.
Before We Start: Selection Is About Fit, Not Rankings
When you build an AI agent, a RAG system, an AI coding tool, a customer service bot, or a document analysis platform, you eventually face the same question: which model API should you use? Many teams answer it by consulting public leaderboards, picking the “strongest” model, and then finding that the results or the bill do not match expectations. The problem is not that leaderboards are unreliable — it is that answering a business question with a generic ranking is a category error.
Looking at capability structure, mainstream LLMs fall into two broad groups. One is stronger at code generation and complex reasoning, with deep adoption in the overseas developer community. The other has natural advantages in local language understanding, Chinese-language knowledge, and the domestic network environment. Both are iterating quickly, and their capability boundaries keep moving. At the same time, models are billed separately for input and output, and unit prices can differ by several times or even tens of times. For the same requirement, choosing a different model can change your quarterly bill by an order of magnitude.
So the question this article addresses is not just “how large is the gap,” but rather how to answer that question with a method: how to convert “the gap” into measurable metrics, and how to make engineering decisions based on business requirements. Below, the two groups are referred to as “leading international models” and “China-based models.” All prices in this article are illustrative; always verify against each provider’s latest price list.
1. Clarify the Problem First: Selection Is Constrained Optimization, Not Ranking
From an engineering perspective, model selection is essentially a constrained multi-objective optimization problem:
Given a task distribution, concurrency level, latency budget, cost budget, and compliance constraints, choose a model or a combination of models that maximizes business value.
Three corollaries follow:
- There is no business-independent “best model.” The gap between your business task distribution and a public evaluation set will either amplify or cancel out the capability difference on paper.
- Capability can be substituted; hard constraints cannot. Capability gaps can be closed with prompt engineering, fine-tuning, tool calling, and task decomposition. Data residency, content compliance, and access stability cannot easily be engineered around.
- It is not a single choice, it is a combination. Real systems often use several models together, alongside caching, retrieval, and tool calls. A single model’s ranking cannot reflect how the whole system performs.
2. The Six-Dimension Framework: From Intuition to Measurable Metrics
Model capability can be decomposed into six dimensions, each with metrics that are quantifiable and reproducible.
| Dimension | Core question | Key metrics |
|---|---|---|
| Price | What does it cost to complete the same batch of tasks | Cost per task, cost per 1,000 requests, cache hit rate |
| Context window | How much material fits in one call, and how well it is used | Nominal window, effective length, long-context recall |
| Reasoning | Can it correctly decompose and execute multi-step problems | Multi-hop QA accuracy, constraint satisfaction rate |
| Coding | Can it write, fix, and run code | Task pass rate, first-attempt pass rate, rate of unhelpful edits |
| Chinese quality | Is the output natural and appropriate for the business register | Human ratings, business sampling pass rate |
| Latency | Does the user get an answer in time | Time to first token, output rate, P95/P99 latency |
These six are not equally important and should be weighted per business. A customer service scenario is more sensitive to Chinese quality and latency; an AI coding tool cares more about coding and reasoning; offline document analysis can relax latency in exchange for lower cost. The next sections show how to configure those weights.
3. Price: Unit Price Is Not Effective Cost
1. Unbundle the bill
LLM APIs are typically billed separately for input and output tokens, and reasoning modes generate additional thinking tokens. An easily overlooked fact: RAG and multi-turn dialogue are input-heavy scenarios, since every request carries retrieved chunks and conversation history; reasoning models and coding agents are output-heavy, since the thinking trace and code edits both inflate output volume. Because output unit prices are usually higher than input unit prices, a high input share does not necessarily mean cheaper, and a high output share does not necessarily mean more expensive — you have to compute with the actual prices.
2. Use “cost per task” instead of unit price
Unit price is just a parameter. The comparable metric is the total cost of completing one unit of work. Take a customer service Q&A workload of 1 million requests per month (illustrative calculation):
- Each request has about 1,500 input tokens (system prompt + retrieved chunks + multi-turn history) and about 300 output tokens.
- Model A costs $0.5 per million input tokens and $2.0 per million output tokens.
- Output tokens are only 1/6 of the total volume, but contribute about 44% of the cost.
- Monthly token cost is 750 + 600 = $1,350.
If 80% of requests are simple Q&A with clear intent and are routed to the cheaper Model B ($0.1 input / $0.4 output per million tokens), the monthly cost drops to about $490, a reduction of more than 60%. The precondition is that evaluation shows B’s first-resolution rate on those requests is on par with A.
3. Account for hidden costs
- Retry cost: failed retries and timeouts multiply call volume. If poor output makes 10% of requests fall back to a human agent, the labor cost of tens of yuan per request quickly exceeds the model price difference.
- Token bloat cost: long system prompts, redundant tool definitions, and untrimmed conversation history can quietly double your input volume.
- Caching gains: many APIs offer caching or discounts for repeated prefixes such as a fixed system prompt or tool definitions. Placing stable content at the front of the message array and frequently changing content at the end is the general practice for improving hit rates.
Conclusion: Do not ask “which model is cheaper.” Ask “which combination of models minimizes the cost per task.”
4. Context Window: Nominal Length Is Not Effective Memory
Claiming support for hundreds of thousands or even a million tokens does not mean the model uses the whole window uniformly. In practice, very long contexts tend to bring these problems:
- Recall drops noticeably when key information sits in the middle.
- Effective lengths vary widely between models; some advertise a very long window but degrade sharply near the limit.
- Long contexts significantly increase latency and cost, with time to first token potentially reaching several seconds.
So a context window should be evaluated at three levels.
1. Window capacity: does your material fit?
You need the input length distribution per request, including P95 and P99, not just the average. A typical composition:
[System prompt + tool definitions] [Conversation history] [Retrieved chunks 1..N] [User question]↑ Fixed part ↑ Variable part ↑ Fixed part
The length of each part determines cache hit rates and your cost structure.
2. Effective length: once the material is in, is it actually used?
Use a multi-needle retrieval test: bury several facts (IDs, dates, amounts) in a long document, then ask about them at the start, 1/4, 1/2, 3/4, and end positions, recording recall accuracy.
def long_context_probe(fill_tokens, needles):"""Bury information at different positions and measure recall accuracy."""results = []for size in [8_000, 32_000, 128_000, 512_000]:for pos in ["head", "quarter", "middle", "three_quarter", "tail"]:text = build_document(size, fill_tokens, needles[pos])resp = ask_model(f"{text}\n\nQuestion: {needles[pos]['question']}")results.append({"size": size,"position": pos,"hit": needles[pos]["answer"] in resp,"cost": estimate_cost(size, len(resp)),})return summarize_recall(results) # Hit rate by length and position
3. Long context vs. RAG
| Criterion | Long context | RAG |
|---|---|---|
| Information completeness | Full material is sent, cross-document relations are more natural | Constrained by retrieval quality, key chunks may be missed |
| Cost and latency | Grows linearly with input length, high latency | Retrieval stage is controllable, less content sent to the model |
| Knowledge updates | Re-supplied on each request, no retraining needed | Updating the index takes effect immediately |
| Typical use cases | Analyzing an entire contract or a whole code repository in one pass | Knowledge base Q&A, retrieval over massive document collections |
In practice the two are often combined: narrow the scope with retrieval first, then place the top-k chunks in the context in full, balancing cost and quality.
5. Reasoning: Look at Where It Actually Fails
Public math and code benchmarks give a rough sense of a model’s reasoning ceiling, but what matters for business is usually a different set of steps:
- Multi-hop reasoning: reaching a conclusion requires chaining several conditions, e.g., “using the payment milestones in the contract appendix and the delay clauses, compute the actual amount payable.”
- Constraint satisfaction: meeting word count, format, role, and citation requirements simultaneously.
- Self-correction: after a tool returns an error, adjusting the approach based on the error message.
When evaluating, pay attention to error types, not just accuracy. If errors concentrate in numeric calculation, introduce a calculator tool into the pipeline; if they concentrate in broken logic chains, consider switching models or splitting the task. A practical signal is stability: sample the same question multiple times and check whether the answers agree. Frequent swings mean the model’s understanding of the problem is not solid.
Reasoning modes improve performance on hard problems, but the thinking trace is billed too, so on simple tasks it is not worth it. Select dynamically by difficulty: let rules or a classification model answer what they can handle, and send only genuinely difficult cases to reasoning mode.
6. Coding: From “Code Completion” to “Correct End to End”
For AI coding tools and data analysis agents, the coding requirements on a model break into four layers:
- Code generation: produce runnable code from a natural language description or comment.
- Code comprehension: read across files, locate defects, explain a legacy system.
- Code modification: refactor and fix bugs without breaking existing behavior, which demands precise instruction following.
- Execution loop: run tests, read errors, iterate until the task is complete.
Beyond task pass rate, evaluate these metrics:
- First-attempt pass rate: the share of outputs that pass tests on the first try.
- Unhelpful edit rate: the share of edits that break previously passing cases.
- Average repair rounds: iterations from first generation to final pass.
- Cost: average tokens consumed per completed task.
First-attempt pass rate and unhelpful edit rate matter most for user experience: a model that frequently breaks working code is hard to trust day to day, even if it occasionally solves hard problems.
Coding capability also has a spillover effect. Coding tasks force a model to follow syntax, naming, and interface constraints strictly, so models that are robust on coding tasks often perform well on structured output and instruction following too. For non-coding scenarios such as JSON extraction or form filling, coding evaluation scores can serve as one reference metric.
7. Chinese Quality: From “Speaks Chinese” to “Says the Right Thing”
Chinese quality cannot be judged by fluency alone. Evaluate it at three levels:
- Foundational: correct grammar, standard terminology, no translationese.
- Register: official documents, contracts, support scripts, and marketing copy each have their own style, and output must match the specific scenario.
- Local: understanding of policies and regulations, industry conventions, and cultural context — for example, standard phrasing in government Q&A, or polite forms in customer support.
Generic leaderboards have limited reference value here. More effective approaches:
- Build a domain evaluation set: sample from real business traffic and have domain experts annotate reference answers.
- Double-blind human scoring: hide the model source and score accuracy, formality, and tone.
- Watch token efficiency: compare how many tokens different models consume on the same Chinese text. This affects both output quality and cost directly.
In scenarios with strict compliance requirements, model output must also pass content safety review (such as text moderation or sensitive content detection) before being returned to the user. That capability belongs in your selection criteria.
8. Latency: Time to First Token Matters More Than Total Time
In interactive applications, what users perceive is the time until the answer starts, not the time until generation finishes. Measure these separately:
- TTFT (time to first token): from request dispatch to the first token received.
- Output rate: token generation speed afterwards.
- P95/P99 latency: peak-hour behavior is more informative than the average.
These metrics are jointly affected by input length, output length, concurrency pressure, and deployment location. For streaming scenarios, prioritize TTFT; for offline batch processing, throughput matters more than latency. Also check service availability, rate limiting policies, and cross-region access stability, and where necessary design a multi-provider failover scheme so a single provider outage does not interrupt the business.
9. Weighted Scoring: Turning Six Dimensions into a Ranking
Weights differ a lot between businesses:
from dataclasses import dataclass@dataclassclass ScenarioWeights:price: float = 0.15context: float = 0.10reasoning: float = 0.20coding: float = 0.15chinese: float = 0.25latency: float = 0.15# Chinese-language customer service bot for consumers: latency and Chinese quality firstSERVICE_W = ScenarioWeights(price=0.20, context=0.05, reasoning=0.10,coding=0.00, chinese=0.35, latency=0.30)def weighted_score(model_report: dict, w: ScenarioWeights) -> float:"""Dimension scores are pre-normalized to 0~1; price and latency use reciprocal or rank mapping."""return sum(getattr(w, dim) * model_report[dim] for dim in w.__dataclass_fields__)def rank_models(reports: list[dict], w: ScenarioWeights):return sorted(reports, key=lambda r: weighted_score(r, w), reverse=True)
Two things to watch when using this:
- Drop dimensions with weight 0 from evaluation. A customer service scenario does not need coding tests, which saves evaluation cost.
- Keep the normalization method consistent. Every model must be measured on the same evaluation set with the same parameter configuration, otherwise the ranking is meaningless.
10. Build a Reproducible Evaluation: Offline and Online in Parallel
1. Offline evaluation set: close to reality, stratified sampling
- Sample from historical logs and desensitize them to form the evaluation set.
- Stratify by task type, difficulty, input length, and language so every stratum has enough samples.
- Have humans annotate reference answers or grading points, and put the dataset under version control.
2. Automated evaluation: rule checks plus model grading
import json, reRUBRIC = """You are a strict reviewer. Evaluate the answer against these points:{points}Output JSON only: {{"pass": true/false, "reason": "..."}}"""def grade(sample, response):# 1) Rule check: structured output can be validated directlyif sample.get("expect_json"):try:obj = json.loads(extract_json(response))if set(obj) != set(sample["expect_keys"]):return 0.0except (json.JSONDecodeError, KeyError):return 0.0# 2) Model grading: return JSON so it is easy to parseverdict = call_model([{"role": "user","content": RUBRIC.format(points=sample["rubric"])}],json_mode=True)return 1.0 if verdict.get("pass") else 0.0def run_eval(model_client, dataset):total = passed = cost = 0.0for s in dataset:resp = model_client.chat(s["messages"])cost += estimate_cost(resp.usage)passed += grade(s, resp.text)total += 1return {"pass_rate": passed / total, "cost_per_task": cost / total}
When using model grading, note:
- If the judge model comes from the same family as the model under test, it may score generously.
- Models tend to prefer longer answers, so the grading prompt should explicitly instruct the judge to assess content quality only.
- Periodically review a sample of results by hand to calibrate the grading standard.
3. Online validation: small-traffic canary
Once offline metrics pass, release 5%–10% of real traffic and track first-resolution rate, human escalation rate, user feedback, and P95 latency. Model quality is highly sensitive to prompts, so online validation must use the exact same prompts and tool definitions as production.
11. Engineering Architecture: Escape the Either/Or with an Abstraction Layer and Routing
1. A unified interface abstraction layer
Provider APIs differ in message format, parameter names, and error codes. Adding an adapter between business code and model APIs minimizes the cost of switching:
class LLMClient:def __init__(self, providers: dict):self.providers = providersdef chat(self, messages: list[dict], model: str = "default", **opts) -> LLMResponse:client = self.providers[model]for attempt in range(2): # Simple retry + fallback channel switchtry:return client.chat(messages, **opts)except (TimeoutError, RateLimitError):model = "fallback"raise RuntimeError("All available channels failed")
The adapter also handles request logging, token accounting, and rate limiting or circuit breaking. With a unified logging service and monitoring alerts, you can analyze call volume, success rate, and cost by model, business line, and scenario.
2. Tiered routing: the right model for the right job
A typical three-tier cascade:
Simple requests (intent classification, FAQ, templated extraction)→ Lightweight model + exact/semantic cacheLow-confidence or high-value requests (complex reasoning, code generation, complaint handling)→ Stronger model or reasoning modeFailures, timeouts, risk-control triggers→ Fallback model, log and alert
3. Cache design
- Exact caching suits high-frequency questions such as FAQs.
- Semantic caching suits scenarios where phrasing varies but the answer is stable, though you need a similarity threshold to prevent false hits.
- The cache key should include the prompt version so entries invalidate automatically after a prompt update.
With this architecture in place, “are China-based models or leading international models stronger” stops being an either/or question and becomes a routing decision per request.
4. Deployment location and cloud services
For businesses sensitive to data export, model APIs can be deployed in a domestic cloud environment; for scenarios with data compliance requirements, common cloud products such as content moderation can review inputs and outputs. On Baidu AI Cloud, model services can work with common cloud products such as logging, monitoring and alerting, object storage, and container platforms to support evaluation data version management and end-to-end observability of the call path. Refer to the official documentation for current availability and configuration details.
12. Back to the “Gap”: Look at Layers, Not Totals
Using this framework to observe the 2026 model landscape yields three conclusions:
- The gap is layered. On general language tasks, routine text generation, and standard information extraction, the gap for China-based models is no longer obvious. On frontier complex reasoning and very long-horizon tasks, the gap remains, and varies by model.
- The gap is scenario-specific. China-based models have structural advantages in Chinese understanding, local compliance, and access stability; leading international models still hold a head start in overseas developer community adoption and parts of the coding toolchain ecosystem.
- The gap keeps narrowing. The iteration pace of China-based models has clearly accelerated, with reasoning models, coding models, and open-source ecosystems emerging continuously while prices keep falling. Selection conclusions have a shorter and shorter shelf life, so re-run the same evaluation set quarterly.
So the more useful question is not “how large is the gap,” but “in my business, does the gap affect results, and how much cost does it affect.” Answering that with data is worth more than any generic ranking.
Conclusion: The Selection Checklist
- Build a stratified evaluation set from real business data and put it under version control;
- Measure cost per task with separate input/output pricing, including retries and human fallback;
- Test effective context length, not just the nominal window;
- Track business-sensitive metrics such as first-attempt pass rate, unhelpful edit rate, and P95 latency;
- Preserve multi-model flexibility through an adapter layer and tiered routing;
- Re-test every quarter and adjust your model strategy accordingly.
Models will keep iterating; the methodology is the long-term asset. Build a reproducible evaluation system so every selection decision is backed by data, and you can find the best trade-off among performance, cost, and compliance.
