A Guide to LLM Selection: From Benchmark Shortlisting to Business Evaluation

1. Why LLM Selection is Harder Now: From “Availability” to “Which One”

A year or two ago, the main question for developers was whether they could access an LLM at all. Today, the question is which one to use. Open-source communities and commercial vendors are releasing new models at a rapid pace, with flagship versions iterating every few months and public benchmark rankings refreshing almost weekly. Concurrently, mainstream cloud providers and inference platforms have widely adopted a “model supermarket” approach to hosted inference services. Platforms uniformly wrap models from different sources into standardized APIs, allowing developers to call them on-demand and use them instantly.

The shelves are getting longer, and the trial-and-error cost is lower, but the difficulty of deciding “which one” is actually rising. The fundamental reason is that high benchmark scores do not equate to usability in production. Public evaluation benchmarks assess general capability distributions, whereas each business has its own unique prompt distribution, latency requirements, and cost budgets. Model selection is fundamentally an engineering decision that requires methodology, not just chasing the latest trends.

2. Understanding the “Model Supermarket”: How Hosted Inference Platforms Work

A hosted inference platform can be understood as a “model inference microservice.” The platform provider deploys model weights onto inference clusters, handles quantization, batching, and elastic scaling, and then exposes them externally via unified API protocols. For developers, this brings three direct benefits:

1. Deployment-Free: No need to prepare your own GPUs, build inference frameworks, or perform quantization and compression, eliminating a whole set of operations work.
2. Low Switching Cost: Most mainstream platforms are compatible with industry-standard chat interface protocols. Switching models often only requires changing the model name parameter.
3. Pay-As-You-Go: Billed at the token level, lowering the barrier to entry for POC phases to nearly zero.

It is worth noting that the speed at which platforms “stock new items” often outpaces their documentation updates—a new model might already be callable via the API before the official documentation is finalized. Therefore, always perform a real call validation before integration to confirm the model version number, context length limits, and billing metrics. A minimal validation code snippet is as follows:

import os
import httpx
BASE_URL = os.environ["LLM_BASE_URL"]   # Access address of the hosted inference platform
API_KEY  = os.environ["LLM_API_KEY"]
def chat(prompt: str, model: str, temperature: float = 0.2) -> str:
    """Call the standard chat interface of the hosted inference platform"""
    resp = httpx.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": temperature,
        },
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

This code is not bound to any specific vendor and will run as long as the platform is compatible with the standard chat protocol. It is recommended to configure candidate model names in environment variables or a configuration center for easy batch comparison later.

3. Establishing an Evaluation Framework Across Five Dimensions

1. Capability Dimension: Benchmarks are for Shortlisting, Not Final Decisions

Capability evaluation can be split into two layers:

• General Capabilities: Knowledge QA, logical reasoning, mathematical calculation, and instruction following, which correspond to what most public benchmarks assess.
• Task-Specific Capabilities: Code generation and repair, long-text understanding, tool calling (for Agent scenarios), and structured output stability. These are often the actual lifelines of a business.

The correct way to use public benchmarks is to narrow down the candidate set, not to make the final decision. Benchmarks have three known limitations: first, data contamination risk, where test questions might have entered the training corpus; second, task distribution variance, where benchmark weights may not align with your business prompt distribution; third, sensitivity to “exam-taking ability” in some benchmarks, meaning scores might disconnect from actual user experience. A safer approach is to use benchmarks to lock in three to five first-tier candidates, and then proceed to the business evaluation phase.

2. Performance Dimension: Latency and Throughput Determine the Experience Ceiling

With equal capabilities, performance differences directly impact product experience. Focus on three metrics:

• Time To First Token (TTFT): The most noticeable metric for users in streaming output scenarios.
• Throughput (tokens/s): Determines the generation time for long answers.
• Concurrency Stability: Whether p95/p99 latency can maintain an acceptable water level during peak hours. This is more meaningful than the average latency of a single call.

Interactive applications (customer service, coding assistants) are extremely sensitive to TTFT, while offline batch processing tasks care more about overall throughput and success rates. Selection should be weighted according to the scenario.

3. Cost Dimension: Hidden Bills Beyond Unit Price

Many teams only compare the unit price per million tokens, which can be misleading. A more reasonable calculation is the cost per task:

Cost per task ≈ Average input tokens × Input unit price
              + Average output tokens × Output unit price
              + Failed retry amortization × Call count

Additionally, pay attention to: tiered billing rules where longer contexts cost more, whether context caching and batch discounts are supported, and whether output length is controllable (a verbose model will implicitly inflate costs). Sometimes, a model with a slightly higher unit price but more concise outputs and correct answers on the first try actually results in a lower overall cost.

4. Engineering Dimension: API Capabilities Determine the Application Ceiling

Engineering characteristics beyond capability benchmarks are often the real dividing line during implementation:

• Context Window Length: Whether it can fit your documents or full conversation history.
• Function Calling and Structured Output: When building Agents or extracting data, the stability of JSON output is more important than how “smart” the model is.
• Streaming Output, Concurrency Quotas, Rate Limiting: Directly dictate how your server-side architecture should be written.
• Ecosystem Compatibility: Models compatible with standard protocols allow your evaluation scripts, gateways, and monitoring systems to be smoothly reused.

5. Delivery Dimension: What to Look for Beyond APIs

Beyond public cloud APIs, you should also evaluate the possibility of private deployment and dedicated cloud delivery. Cloud providers like Baidu AI Cloud typically offer multiple delivery formats simultaneously, including online APIs and private deployment. Teams with strict data compliance requirements (finance, healthcare, government) should treat “whether data leaves the domain and whether the model can be deployed on an intranet” as a veto factor to confirm in advance.

4. Building a Business Evaluation Set: More Reliable Evidence Than Benchmarks

To stop “selecting models based on benchmarks,” the core action is to build your own evaluation set using real business data. The method involves four steps:

1. Sampling: Desensitize and sample from real prompts in online logs or manually curated ones. Cover high-frequency scenarios, edge cases, and difficult scenarios. Typically, 100 to 300 cases are sufficient to effectively differentiate candidate models.
2. Define Scoring: Prioritize rule-based scores (keyword coverage, format validation, running unit tests for code questions), then use an LLM-as-a-Judge to supplement subjective dimensions, and finally manually spot-check to calibrate the judge’s bias.
3. Automated Batching: Run each candidate model under identical conditions, recording both scores and latency.
4. Lock Version, Multiple Runs: Model APIs might be silently upgraded; lock the version number during evaluation. Generation has randomness, so run multiple times to get the mean and monitor the variance. Models with high variance carry a higher risk in production.

A lightweight evaluation framework example:

import json, time, statistics
# Business evaluation set: each entry contains prompt, expected key points, format constraints
cases = json.load(open("biz_eval_set.json", encoding="utf-8"))
def rule_score(answer: str, case: dict) -> float:
    """Rule score: key point coverage × format compliance coefficient"""
    hits = sum(1 for k in case["expect_keys"] if k in answer)
    coverage = hits / len(case["expect_keys"])
    fmt_ok = True
    if case.get("format_hint"):
        fmt_ok = case["format_hint"] in answer
    return coverage * (1.0 if fmt_ok else 0.5)
def evaluate(model: str, runs: int = 3) -> dict:
    scores, latencies = [], []
    for _ in range(runs):
        for case in cases:
            t0 = time.time()
            answer = chat(case["prompt"], model)
            latencies.append(time.time() - t0)
            scores.append(rule_score(answer, case))
    return {
        "model": model,
        "avg_score": round(statistics.mean(scores), 4),
        "score_std": round(statistics.stdev(scores), 4),
        "p95_latency": round(sorted(latencies)[int(len(latencies) * 0.95)], 3),
    }

Run all candidate models and output a comparison table. This “business evaluation report” is the most persuasive material for a selection meeting.

5. Balancing Cost and Performance: Tiered Routing and Continuous Observation

Model selection does not have to be an “either-or” decision. Mature teams typically adopt a tiered routing strategy: assigning simple tasks (rewriting, classification, extraction) to lightweight models, and complex tasks (reasoning, code generation, Agent planning) to flagship models, striking a balance between cost and performance:

def route(prompt: str) -> str:
    """Tiered routing based on task complexity"""
    if estimate_complexity(prompt) < COMPLEXITY_THRESHOLD:
        return "model-lite"       # Lightweight model: low cost, low latency
    return "model-flagship"       # Flagship model: complex reasoning and code tasks

This is best paired with two cost-reduction methods: first, caching, which can drastically cut call volumes for high-frequency repetitive questions; second, batch processing, which leverages batch interfaces for offline tasks to secure lower unit prices.

After going live, establish continuous observation: success rates, latency percentiles, token consumption and cost alerts, and routing bad cases back to the evaluation set. Models iterate quickly. Re-evaluating with the same evaluation set each quarter can determine if it is worth switching to a new model—selection is not a one-time decision, but a sustainable mechanism.

6. Implementation Decision Checklist

Finally, here is an actionable POC checklist:

• Use benchmarks for initial shortlisting to lock in 3 to 5 first-tier candidates;
• Verify API availability one by one: model version, context limit, rate limit quota;
• Build a business evaluation set of 100+ cases: rule scores + LLM-as-a-Judge + manual spot-checks;
• Run batches under identical conditions, recording mean scores, variance, and p95 latency;
• Calculate the cost per task, including retry and output length factors;
• Confirm data compliance and delivery format (API / private deployment);
• Configure tiered routing and monitoring alerts, routing bad cases back to the evaluation set;
• Solidify a quarterly re-evaluation mechanism to maintain the ability to switch at any time.

Conclusion

The LLM market is highly prosperous and iterating rapidly. “Model supermarket” hosted inference platforms have lowered the trial threshold to an absolute minimum. What is truly scarce is not the models themselves, but a reproducible selection mechanism: benchmarks handle shortlisting, business evaluation sets handle the final verdict, cost and compliance act as constraints, and monitoring with re-evaluation keeps things fresh. By turning selection from a “guessing game” into “running data,” you can make fast, robust, and correct decisions no matter how the models on the shelves change.