LLM Inference Service Selection Guide: From Model Capability to Cost
Choosing a hosted LLM inference service is less about picking the strongest model and more about matching a model with a platform that fits your task mix, concurrency, and budget. This guide covers separating model capability from platform service quality, building an in-house 100–300 item evaluation set with fixed sampling and weighted metrics, auditing API and engineering compatibility (context limits, streaming, tool calling, structured output, retry semantics), benchmarking TTFT and TPOT at P50/P95/P99, and comparing cost per completed task instead of price per token. It also covers quota, SLA, data retention, and compliance checks, plus the routing, canary, and fallback patterns that turn a single-model choice into a production strategy.
1. First, Understand the “Model Marketplace” Delivery Model
One clear shift of the past two years: LLM delivery has moved from “pull the weights yourself, stand up your own inference stack” to “call a hosted inference service on demand.” A typical form of this is the model marketplace operated by chip vendors or cloud providers: the platform pre-hosts a set of models, some with open weights and some with closed weights, and users need only a gateway endpoint plus credentials to call them over a unified HTTP API, billed by usage.
Early listings on these platforms were mostly open-weight models from overseas; leading models developed in China are now appearing on the same shelf. Worth stressing: a new listing usually comes without a clear product announcement — the endpoint works first, the documentation arrives later. Two consequences follow. First, availability changes often outpace a team’s information cycle, so selection cannot rely on “announcement-driven awareness.” Second, service parameters for the same model — context limit, concurrency quota, streaming and structured output support — may differ across platforms. Same model does not mean same service.
So the first-order question is not “which model is strongest” but “given my task distribution, concurrency level, and budget constraints, which model-plus-service combination is the most reliable?”
2. First Principle: Separate Model Capability from Platform Service Quality
Many selection failures are not caused by choosing the wrong model. They are caused by discussing two orthogonal questions as if they were one:
- Model capability: correctness of the output for a given input, style, whether code runs, instruction adherence, effective use of long context. A property of the model.
- Service capability: API contract, context length, rate-limit quotas, latency distribution, availability, billing granularity, data retention policy, regional availability. A property of the platform.
A model that performs well in offline evaluation can be capped at low concurrency, exhibit high time-to-first-token, or lack tool calling on a given platform — and be unusable in business terms. Your evaluation matrix therefore has to be two-dimensional: task type × platform instance. In practice, use a small amount of real traffic to filter candidate platforms on service capability first, then compare model capability with an offline evaluation set. Reversing that order tends to waste a great deal of evaluation work.
3. Capability Evaluation: Leaderboards Are a Signal, Not a Verdict
Public leaderboards for code generation and general reasoning do carry reference value: they give you the coarse signal that a batch of models sits roughly in the same tier. But treating them as the basis for selection has several known biases:
- Distribution mismatch. Benchmark items are mostly short, single-turn, clearly bounded cases. Real workloads involve long documents, multiple turns, domain terminology, and implicit constraints.
- Single metric. Leaderboards usually look only at pass rate, not at output length, latency, or stability (the variance across repeated samples of the same input) — and the latter directly determine user experience and cost.
- Reproducibility differences. Prompt templates, sampling parameters, and scoring methods differ from one leaderboard to another, so comparing ranks across leaderboards is of limited value.
Recommended practice: build your own high-value evaluation set of 100–300 items. Draw a stratified sample from real production requests covering three categories — high-frequency main paths, long-tail edge cases, and known failures. Annotate each item with expected output key points or executable validation logic, and fix the sampling parameters (low temperature, fixed random seed) so results are reproducible.
The eval suite can be expressed in a single neutral config file:
eval_suite:sampling: {temperature: 0.2, top_p: 0.9, max_output_tokens: 512}metrics:- rule_pass_rate # pass rate of rule/assertion checks- llm_judge_score # LLM-as-judge score (calibrate with human spot checks)- citation_accuracy # citation accuracy when grounded answers are required- latency_p95_ms- cost_per_1k_requestsdatasets:- name: high_freq_mainweight: 0.5- name: long_tail_edgeweight: 0.3- name: known_regressionweight: 0.2
Look at the weighted aggregate rather than “ranked Nth on leaderboard X” — it guides decisions far better. One caveat: LLM-as-judge has its own preference bias, for example favoring longer, better-formatted answers, so you must retain a portion of human spot checks as a calibration anchor.
4. API and Engineering Compatibility
The mainstream approach for hosted inference today is a chat completions API compatible with the de facto industry standard. This greatly lowers the cost of switching platforms, but you should not assume the interfaces are fully equivalent. Verify at least the following before committing:
- Auth and gateway: how credentials are managed, the shape of the gateway endpoint, whether a unified ingress is used, whether there are regional differences.
- Context and output limits: max input tokens, max output tokens per request, and whether both draw on the same budget.
- Streaming output: whether tokens are returned incrementally, and how error semantics are expressed when the stream breaks abruptly.
- Tool calling / structured output: function calling, JSON Schema constrained output. This is the dividing line for agent-style applications; lacking it changes your architecture outright.
- Multimodal input: native support for images, audio, and document parsing, or preprocessing on the business side.
- Idempotency and retry semantics: whether error codes for timeouts, rate limits, and upstream 5xx are standardized and distinguishable, and whether requests can be retried safely.
To avoid being locked into a single platform, add your own abstraction layer between business code and the platform SDK:
from typing import Protocolclass LLMClient(Protocol):def chat(self, req: ChatRequest) -> ChatResponse: ...def stream(self, req: ChatRequest) -> Iterator[Chunk]: ...def make_client(provider: str, cfg: ProviderConfig) -> LLMClient:# The adapter layer handles: endpoint assembly, credential injection,# error-code normalization, retries, and instrumentation.return ADAPTERS[provider](cfg)
The adapter layer does three things: normalize each platform’s error codes into a unified taxonomy (rate limited / timeout / bad parameter / upstream failure); collect usage fields uniformly (input tokens, output tokens, cache hits); and move the model identifier from a code constant to configuration. With this layer in place, switching platforms usually means changing configuration rather than business logic.
5. Performance: Three Numbers You Must Measure Yourself
At the same unit price, user experience can differ by an order of magnitude because the latency structure differs. Measure at least these separately:
- TTFT (time to first token): determines how fast the response feels, and is heavily affected by input length and queueing.
- TPOT / ITL (inter-token latency): determines smoothness during long generations.
- Single-request throughput and concurrency ceiling: determines how much traffic you can support and whether requests queue at peak hours.
The core of the benchmark script needs only one streaming call to collect all metrics:
def benchmark_once(client: LLMClient, prompt: str, max_tokens: int = 256) -> dict:t0 = time.perf_counter()first_token_at = Nonechunks = []for ch in client.stream(prompt, max_tokens=max_tokens):if first_token_at is None:first_token_at = time.perf_counter()chunks.append(ch.delta)end = time.perf_counter()out_tokens = count_tokens("".join(chunks))return {"ttft_ms": (first_token_at - t0) * 1000,"tpot_ms": (end - first_token_at) * 1000 / max(out_tokens - 1, 1),"out_tokens": out_tokens,"tokens_per_sec": out_tokens / (end - t0),}
Key discipline: report P50/P95/P99, not averages. Latency distributions on generative endpoints are visibly long-tailed, and averages systematically mask peak-hour queueing. Load tests should construct requests in a mix of short and long inputs that matches real business proportions, rather than saturating concurrency with one short prompt.
6. Cost: Compute Cost per Task, Not Price per Token
The common misjudgment in LLM cost comparison is looking only at price per million tokens. What you should look at is cost per task = (input tokens × input price + output tokens × output price) / successfully completed tasks. A few easily overlooked variables:
- Order of magnitude of output tokens. Output pricing is usually far higher than input pricing, and verbose output inflates the bill directly; constraining output format and length in the prompt is often the most effective cost lever.
- Retry and failure cost. Failed requests are usually billed but produce no business value, so poor stability silently raises unit cost.
- Context replay cost. Multi-turn conversations and RAG resend the system prompt and retrieved chunks on every turn. Platforms that support prompt caching can cut this substantially, so confirm during selection whether the capability exists and what its cache-hit semantics are.
- Billing granularity. Whether there is a minimum billable unit, whether batch/async jobs get a lower rate, whether a reserved throughput mode is offered. Batch workloads (offline labeling, document summarization, data cleaning) usually get better value through an async, lower-rate channel.
7. Stability, Quotas, and Compliance
The non-negotiable items on the engineering side:
- Quotas and rate-limit policy: are RPM/TPM limits enforced at account level or model level, what error is returned when they are exceeded, and is there a channel to request a quota increase.
- Availability commitments: whether there is an explicit SLA and defined remedy terms, and whether historical availability is publicly viewable.
- Data governance: whether requests and responses are retained, for how long, whether they are used for training, and whether retention can be turned off; in enterprise scenarios this should be written into contracts or configuration rather than verbal assurance.
- Compliance and region: data export, regional availability, and content-safety pre-filtering policy all shape the architecture, and often become hard constraints earlier than model capability does.
- Observability: wire gateway endpoints, token consumption, latency percentiles, and error-code distribution into logging and alerting; set alert thresholds based on P95 latency and error rate rather than total call volume alone.
On integrated cloud platforms — Baidu AI Cloud is one example — model services are typically delivered alongside container platforms, object storage, logging, and monitoring/alerting, so evaluate the model layer and the infrastructure layer as one whole: how credentials are managed, which storage holds evaluation sets and artifacts, how canary traffic is split on the container platform, and how monitoring data flows back to close the evaluation loop.
8. From Point Selection to a Portfolio Strategy: Routing and Fallback
The most robust answer in production is often not “pick one model” but “define a priority order.” Route by task characteristics and degrade automatically on failure:
def route(task: Task, providers: list[Provider]) -> ChatResponse:for p in providers: # ordered by cost/quality priorityif not task.requirements.issubset(p.capabilities):continue # capability gap, skipif p.breaker.is_open():continue # breaker open, don't keep hitting a failing endpointtry:return p.chat(task, timeout=task.timeout)except (RateLimited, Timeout, UpstreamError) as e:metrics.incr("llm_fallback",tags={"provider": p.name, "err": type(e).__name__})continueraise AllProvidersUnavailable(task.id)
Three practices work best alongside it. First, tiered routing: send simple classification and extraction to lightweight, low-cost models, and complex reasoning and code generation to flagship models, with rules or a small classifier making the upfront decision. Second, canary release: run a new model on mirrored traffic or a small share of real traffic, double-run it against the incumbent, and confirm no regression on the same eval set before ramping up. Third, regression gates: wire the eval set into CI so that any model or prompt change must pass the weighted metrics, and a drop beyond the threshold blocks the release.
9. Deployment Checklist
- Build a 100–300 item eval set from stratified sampling of real traffic, including known-failure regression cases
- Fix sampling parameters so evaluation results are reproducible
- Build a platform capability matrix: context, streaming, tool calling, structured output, caching, batch channel
- Load test against the real input-length distribution; record P95 and P99 of TTFT/TPOT
- Compare platforms on cost per task rather than unit price, including failure and retry overhead
- Confirm data retention, regional, and content-safety policies meet compliance requirements
- Add an adapter abstraction layer between code and platform; move model identifiers into configuration
- Configure multi-model priority order, timeouts, retries, and breaker-based degradation
- Wire up logging and monitoring; alert on latency percentiles and error rate
- Put the eval set in CI as a release gate for model switches and prompt iteration
One-line takeaway: hosted inference has turned “can we even get access to a flagship model” into a low-cost question. The real investment is evaluation capability. Whoever runs a shorter evaluation loop with metrics closer to the business will keep landing on the most suitable model as models iterate quickly — without betting on a single leaderboard ranking.
