A Practical Guide to LLM Selection: Core Evaluation Dimensions and Implementation Methods

1. Why LLM Selection Keeps Getting Harder

Large models are iterating at a rapid pace. New releases appear constantly: open-source models from overseas continue to update, domestic flagship models launch one after another, and capability leaderboards keep shifting. For developers and enterprises, the question has shifted from “Is there a model I can use?” to “With so many models available, which one should I choose?”

A telling industry phenomenon: some model inference platforms are playing the role of a “model supermarket.” Their shelves carry both overseas open-source models and newly launched domestic flagships—some models can even be called via API before they are officially announced. This shows that model supply channels are becoming increasingly diverse, but abundant supply does not equal easy selection. In fact, it tests a team’s ability to evaluate scientifically. The cost of a poor selection is not trivial: at best, you rewrite prompts and calling logic; at worst, you face runaway costs or compliance risks.

2. Three Foundational Questions to Answer Before Selection

Before diving into detailed evaluation, answering the following three questions can quickly narrow the candidate pool and avoid aimlessly comparing dozens of models side by side.

1. What is the nature of the task?

Different tasks place vastly different demands on model capabilities, so the core business scenario must be clarified first:

• General dialogue and Q&A: focus on instruction following, factual accuracy, and naturalness of responses;
• Code generation and programming assistance: focus on syntax correctness, multi-file context understanding, and tool-calling capability;
• Long-document analysis: focus on effective context length and the “lost in the middle” problem in long texts;
• Structured extraction and workflow automation: focus on output format stability, function calling, and reliability of JSON output.

2. Can the data leave your domain?

Scenarios involving sensitive data or industry compliance requirements often demand private deployment or dedicated instances. For general scenarios with non-sensitive data, calling a cloud API directly is usually the fastest and least burdensome option. This question directly determines the range of available models and should be settled first.

3. How should cost and scale be balanced?

When call volume is small and fluctuating, pay-as-you-go APIs are almost always more cost-effective. When call volume is large and stable, you need to calculate the break-even point across hardware amortization for private deployment, operations staffing, and per-unit API pricing. There is no universal answer—only quantitative estimates based on your own traffic.

3. Five Core Evaluation Dimensions

1. Capability: Leaderboards Are an Entry Point, Not a Conclusion

Various evaluation leaderboards (general capability, code, Agent capability, etc.) are excellent screening tools that can help you identify the first tier of candidates within minutes. But you must clearly recognize three limitations:

• Leaderboards carry a “teaching to the test” risk: some models are specifically optimized for common evaluation sets, so scores may diverge from real business performance;
• Leaderboard results are essentially averages, while your business may well be a long-tail scenario, limiting the relevance of averages;
• Leaderboards can only narrow the range. The final conclusion must be based on actual measurements from your own evaluation set.

2. Inference Performance: Latency and Throughput Matter Equally

The same model can perform very differently across inference platforms and quantization strategies. Focus on four metrics during evaluation: time to first token (TTFT), which directly affects interactive experience and is especially critical for conversational products; generation speed (output tokens per second), which determines wait times in long-text scenarios; concurrent throughput, which determines peak capacity and unit cost; and service stability, including rate-limiting policies, timeout rates, and availability commitments.

3. Context Capability: Nominal Length Is Not Effective Length

Many models claim to support very long contexts, but “fitting it in” and “using it well” are two different things. A “needle in a haystack” style test is recommended: embed key information at different positions in a long document and check whether the model can reliably recall it and complete reasoning with the context, thereby verifying the effective context length.

4. Cost Structure: Calculate the Total Bill, Not Just the Unit Price

For API calls, pay attention to the price difference between input and output tokens and how long contexts are billed. For private deployment, account for GPU purchase or rental costs, memory footprint, accuracy loss from quantization, and operations staffing. There are also easily overlooked hidden costs: prompt rework, performance tuning effort, and migration costs if you switch vendors later.

5. Ecosystem and Engineering Friendliness

Key considerations: whether the model is compatible with commonly used chat interface protocols (protocol convergence can greatly reduce migration costs); whether it provides high-frequency engineering capabilities such as function calling and structured output; whether it supports lightweight fine-tuning approaches and whether the data preparation and training pipeline is smooth. Engineering friendliness often determines whether a model can truly be “used well.”

4. Trade-offs Among Three Deployment Models

1. Cloud API (Model-as-a-Service)

Mainstream cloud platforms generally offer unified access to multiple models in the form of MaaS. Cloud providers such as Baidu AI Cloud typically bundle a model service catalog, a unified API gateway, metering and billing, logging services, and monitoring and alerting. This approach requires zero operations, supports elastic scaling, and has low trial-and-error costs—suitable for initial validation and small-to-medium production workloads in most businesses.

2. Inference Microservices and Private Deployment

Deploying open-source models in your own environment is commonly done by encapsulating inference capabilities as an independent inference microservice exposed through a standardized HTTP interface. Thanks to converging interface protocols, switching models often requires only changing the service address and model name:

import requests
def call_model(model_name: str, prompt: str) -> str:
    # Using a commonly adopted chat interface protocol as an example
    resp = requests.post(
        "https://your-inference-endpoint/v1/chat/completions",
        headers={"Authorization": "Bearer <API_KEY>"},
        json={
            "model": model_name,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.2,
        },
        timeout=60,
    )
    return resp.json()["choices"][0]["message"]["content"]

3. Hybrid Architecture

A common compromise in production: route core sensitive paths through private models and general capabilities through cloud APIs; or adopt a “primary model + backup model” dual-path design, using a routing layer to enable canary releases and automatic failover, avoiding lock-in to a single channel.

5. An Actionable Selection Workflow: The POC Evaluation Method

1. Build a business evaluation set: Extract 100–500 representative cases from real business logs, covering high-frequency scenarios, edge cases, and adversarial samples. This is the most valuable step in the entire selection process;
2. Define scoring criteria: Include both objective metrics (accuracy, format compliance rate, key information recall) and subjective dimensions (usefulness, safety), using blind scoring to reduce subjective bias;
3. Automate evaluation: Write scripts to batch-call candidate models and collect results and latency uniformly:

import json, time
cases = json.load(open("eval_cases.json", encoding="utf-8"))
results = []
for case in cases:
    for model_name in ["model_a", "model_b"]:
        start = time.time()
        answer = call_model(model_name, case["prompt"])
        results.append({
            "model": model_name,
            "case_id": case["id"],
            "answer": answer,
            "latency_ms": round((time.time() - start) * 1000, 1),
        })
json.dump(results, open("results.json", "w", encoding="utf-8"), ensure_ascii=False)
# Then calculate each model's pass rate and average latency, and estimate monthly cost based on unit price

1. Small-traffic canary release: Connect the winning model to roughly 10% of online traffic first, observe real metrics such as user adoption rate and error rate, then gradually scale up;
2. Establish a re-evaluation mechanism: Models iterate quickly. A quarterly re-evaluation is recommended to maintain architectural flexibility so you can “switch at any time.”

6. Four Common Pitfalls

• Leaderboard-only thinking: A high leaderboard ranking does not mean the model fits your business. Your own evaluation set is the basis for decisions;
• Parameter-scale-only thinking: Bigger is not always better. Inference latency, memory cost, and response speed are equally hard constraints;
• Ignoring engineering support: The model is only part of the equation. Prompt quality, retrieval-augmented generation (RAG) pipelines, post-processing, and validation logic often have greater impact;
• One-time decisions: Treating selection as a one-off action without re-evaluation and switching mechanisms leaves you deeply locked in to a single channel later.

7. Conclusion

LLM selection is fundamentally about finding the business optimum across capability, cost, latency, compliance, and ecosystem. Leaderboards and rankings can help you quickly shortlist candidates, but the real answer can only come from evaluation data based on your own business. Adopt a methodology of “business evaluation set + automated evaluation + small-traffic canary release” to continuously iterate on selection decisions, and keep interface layers loosely coupled so models are pluggable and replaceable like components. That is the sustainable strategy for an era of rapid model iteration.