LLM Selection Guide: Evaluation Dimensions and Practical Pitfalls

Start with the conclusion: LLM selection is not about finding the strongest model. It is about matching the most suitable model to a specific scenario, and replacing subjective experience with reproducible model evaluation. The actionable path has five steps: build an evaluation set, define metrics, run blind tests, run regression checks, and use gateway routing. This article does not focus on specific products; it focuses on engineering and architecture methods.

1. Why LLM selection is a technical decision, not a ranking contest

The difficulty is that model capability is multidimensional and strongly scenario-dependent. A model that performs well in general conversation may not perform equally well in long-document understanding, structured output, or code generation. Likewise, a model that is strong at a particular vertical task may still be average at open-domain conversation. Such judgments are only meaningful when the task type and data distribution are similar. Without that context, cross-model comparisons have limited reference value.

The correct approach is therefore: first establish evaluation dimensions, then validate them quantitatively, and finally combine the results with engineering constraints such as latency, cost, and compliance.

2. Six core evaluation dimensions

Dimension	Key focus points	Core metrics	Typical evaluation method
General capability	Open-domain stability, common-sense coverage, proactive clarification when information is insufficient, and stating the basis when uncertain instead of fabricating facts	Pass rate for accuracy, relevance, and completeness	Sample several tasks such as weekly reports, plans, copywriting, and planning; manually score them against a rubric
Chinese-language and contextual understanding	Implicit intent recovery, colloquial expressions, and robustness to irregular input order	Intent recovery accuracy	Construct samples with vague requirement descriptions and check whether the model answers off-topic
Code and logic reasoning	Functional correctness and rigor of the reasoning chain	Unit test pass rate and multi-step reasoning accuracy	Use test cases with standard answers and automated execution for scoring
Multimodal capability	Layout parsing precision and fine-grained information recognition	Field-level extraction accuracy	Use charts and documents with real business layouts for end-to-end validation
Performance and stability	Time to first token (TTFT), output speed, concurrency throughput, long-context stability, and service availability	p50/p95 latency, throughput, and error rate	Run load tests with fixed parameters and repeated sampling
Cost and controllability	Separate input/output billing, impact of context length, rate limits, deployment form, and compliance	Cost per call and effectiveness-to-cost ratio	Estimate based on actual token usage; see Section 4

For multimodal evaluation, pay special attention to boundary conditions. Some models support image input only, while support for audio and video varies significantly. Before selection, verify each capability against the official documentation; do not carry over conclusions from similar models.

3. How to evaluate models scientifically: from subjective experience to reproducible methods

The biggest problem with subjective experience is that it is not reproducible. Model output is also stochastic: when sampling parameters are not fixed, the same prompt can produce different results across runs.

Build a scenario-based evaluation set

Extract samples from real business. A practical range is 20–50 samples per task category, with a total of 50–200 samples. Cover typical, boundary, and adversarial samples, and use stratified sampling, for example with a 61 ratio. This range is based on experience. Its purpose is to prevent variation in a single dimension from outweighing differences between models. If samples are truly limited, for example fewer than 30, do not report absolute scores. Use paired blind tests with majority adjudication, and output only relative conclusions: win, lose, or tie.

Annotation rules

For each sample, clearly write the scoring points. For important dimensions, have two annotators work independently. If disagreements exceed the agreed ratio, refer them to a third adjudicator and write the adjudicated result back into the case library.

Pass thresholds and decision method

Before launch, set a minimum threshold for each dimension, for example structured-output format compliance must be at least 95%. If any dimension fails, eliminate that candidate. Then use total score plus a radar chart of all dimensions to decide among the remaining candidates. This prevents a high total score from hiding individual weaknesses. For tasks involving function calling or tool calling, additionally record end-to-end task completion rate, not only single-turn answer quality.

Blind testing and regression

Hide model names before scoring to avoid brand bias. Rerun the same evaluation set whenever switching models or versions to prevent silent regressions.

Below is an illustrative evaluation framework in pseudocode. The interfaces do not represent any real service.

# Illustrative evaluation framework (pseudocode)
def evaluate_model(model_client, test_cases, params=None):
    # Fix sampling parameters; otherwise neither latency nor quality is reproducible
    params = params or {"temperature": 0, "top_p": 1, "max_tokens": 1024}
    dims = {"accuracy": [], "relevance": [], "format": []}
    records = []
    for case in test_cases:
        messages = [
            {"role": "system", "content": case.get("system", "")},
            {"role": "user", "content": case["prompt"]},
        ]
        try:
            resp = model_client.chat(          # Chat-style API, rather than a single-shot complete
                messages,
                temperature=params["temperature"],
                top_p=params["top_p"],
                max_tokens=params["max_tokens"],
                timeout=30,
                retries=2,                     # Retry on timeout/rate limit with exponential backoff
            )
        except Exception as e:
            # Record a single failure only; do not interrupt the whole evaluation run
            records.append({"id": case["id"], "status": "failed", "error": str(e)})
            continue
        text = resp["content"]
        expected = case.get("expected")        # May be None
        # Scoring convention: return 1 for a hit and 0 for a miss;
        # if continuous scores are used, normalize them to 0-1 before summing
        dims["format"].append(check_format(text, case["schema"]))
        dims["relevance"].append(check_relevance(text, case["prompt"]))
        if expected is not None:
            dims["accuracy"].append(check_accuracy(text, expected))
        # Exclude samples without a reference answer from the accuracy denominator instead of scoring them as 0
        records.append({
            "id": case["id"], "status": "ok",
            "latency_ms": resp.get("latency_ms"),
            "usage": resp.get("usage"),        # Also accumulate token usage for cost estimation
        })
    summary = {k: (sum(v) / len(v) if v else None) for k, v in dims.items()}
    return {"summary": summary, "cases": records}   # Keep per-case details to help locate regressions

4. Cost estimation: turn effectiveness-to-cost ratio into numbers

Cost is not simply unit price; the key measure is effectiveness-to-cost ratio. Input and output tokens are usually billed separately, and output prices are often higher than input prices. Therefore, tasks that require long outputs are usually more expensive.

Cost per call ≈ input tokens × input price + output tokens × output price

Two additional adjustments should be considered:

• Multiply by (1 + retry rate) to account for wasted calls caused by failed retries.
• Requests that hit the cache should be converted at the cache price.

For example, if a call uses 2,000 input tokens, 500 output tokens, has a 10% retry rate, and no cache hit, the cost is approximately (2000 × input price + 500 × output price) × 1.1. Longer context and larger output increase costs. Long-document and long-context scenarios require dedicated budget controls.

5. Scenario-based selection strategies

• Daily office work and content creation: Prioritize general capability and Chinese-language understanding, while valuing stability and ease of use. The requirement for reasoning depth can be relatively relaxed.
• Programming assistance and engineering tasks: Focus on code correctness, long-context understanding, and structured output. It is recommended to use unit test pass rate as the core metric.
• Data analysis and logic reasoning: Focus on multi-step reasoning accuracy and numerical computation. Confirm whether the model supports function calling or tool calling to connect to external computation.
• Multimodal applications: Validate image, document, and chart parsing precision using real business layouts, rather than relying only on official demo examples.

A practical approach is to configure different models for different scenarios and use a unified access layer for routing, instead of trying to make one model handle everything.

6. Common pitfalls and considerations in LLM selection

1. Relying only on public leaderboards: Leaderboard task distributions may differ from your business. Data contamination and leaderboard optimization may also occur. Leaderboards can only be used for initial screening.
2. Replacing evaluation with chat experience: A pleasant chat experience is based on a small number of samples and subjective preference. Weaknesses will appear when the model is used in real tasks.
3. Ignoring context billing: Long context is billed by tokens, and costs may rise nonlinearly with context length.
4. Ignoring version iteration regressions: A model with the same name may be silently upgraded. Without a regression evaluation set, quality can degrade without being noticed.
5. Ignoring compliance and data transfer: For sensitive data scenarios, confirm the deployment form, such as public cloud API or private deployment, and confirm compliance requirements first.

7. Enterprise adoption: unified access and observability

1. Unified access gateway: Abstract a layer of interfaces so that business code depends only on a unified contract. Switching the underlying model does not require changing business logic, reducing lock-in risk. A self-developed gateway combined with open-source observability components is sufficient; a specific commercial platform is not required.
2. Observability: Record the input, output, latency, and cost of each call. Set alerts for latency spikes and rising error rates. For compliance, also desensitize or encrypt personal information and sensitive business data in inputs and outputs. Define log retention periods and access permissions, using least privilege and auditability. User data must be de-identified before being used in evaluation sets, and the authorization scope and compliance requirements must be confirmed.
3. Cost and capacity governance: Use rate limiting, caching, and batching to optimize call costs. Apply dedicated controls to high-consumption scenarios such as long context.
4. Performance comparison method: Latency and throughput are affected by sampling randomness. When comparing models, fix sampling parameters, measure multiple times under the same concurrency and input length, and report p50/p95 results rather than a single observation.

8. Five-action checklist before launch

1. Build the evaluation set: Stratified sampling of 50–200 samples from real business, with clear scoring points.
2. Define metrics: Determine the core metrics and pass thresholds for each dimension.
3. Run blind tests: Hide model names, have multiple people score independently, and adjudicate disagreements.
4. Run regression checks: Rerun the evaluation set after switching models or versions, and keep per-case details.
5. Use gateway routing: Route by scenario, and integrate logging, desensitization, and cost alerts.

9. Conclusion

Model selection is an iterative engineering process, not a one-time action. Once the evaluation set, metric system, and engineering pipeline are mature, you can use the same process to evaluate new models quickly and rationally, without getting stuck on the question of which model is the strongest.