LLM Selection Methodology: From Capability Evaluation to Cost Implementation

1. Why leaderboard-style selection fails

Almost every team asks the same question when making a technology decision: “Which general-purpose LLM is the strongest right now?” The question is difficult to answer not because information is scarce, but because the question itself contains three flawed assumptions.

First, it assumes capability is a single dimension. The real capability profile of a general-purpose LLM is a high-dimensional vector: natural-language reasoning, code generation and defect fixing, long-document understanding, multimodal perception, instruction following, naturalness of style, and stability of tool calling. These dimensions are not strongly correlated. A model that leads on graduate-level science reasoning benchmarks may perform poorly on long-context information extraction. A model with a strong reputation for coding may support only limited multimodal inputs. “Overall first place” is essentially an arbitrary weighting of different dimensions presented as an objective fact.

Second, it assumes public leaderboards match business distributions. Once a public benchmark becomes a selection criterion, it may enter the model’s training-data view, creating some degree of data contamination and benchmark-specific optimization. More importantly, distribution gaps matter: benchmark items are usually short prompts, single-turn questions with automatically verifiable answers, while real traffic includes many open-ended tasks with multi-turn context, domain terminology, very long tables, strict format constraints, and high failure costs. Using the former to predict the latter often produces errors larger than the true capability gap between models.

Third, it assumes capability is the only decision variable. In production engineering, non-capability factors often determine success or failure: cost per task, time to first token and throughput, concurrency quotas, service availability and rate-limiting policies, data residency and compliance requirements, stability of accounts and network environments, behavior drift caused by model version updates, and whether an abstraction layer can reduce vendor lock-in. A slightly less capable model with stable latency, controllable cost, and private-deployment support often wins in most production systems over a stronger model that cannot enter the critical path.

The correct question is therefore: Given my task distribution, cost budget, latency requirements, and compliance constraints, which model or combination of models is optimal? This is an engineering optimization problem, not a leaderboard lookup.

2. Build a multi-dimensional evaluation framework

The first step is to turn “it feels usable” into measurable and reproducible metrics. An evaluation table can be organized around four layers: capability, engineering, cost, and governance.

Layer	Dimension	Key metrics	Measurement
Capability	Logical reasoning and knowledge	Subject reasoning accuracy, multi-hop reasoning accuracy	Private question bank + automated scoring
Capability	Code engineering	Unit-test pass rate, defect localization hit rate, compilation pass rate	Execute generated code and patches in a sandbox
Capability	Long context	Long-document QA accuracy, key information recall, cross-paragraph coreference resolution	Construct long-document samples with “needles”
Capability	Multimodal	Chart-reading error, layout reconstruction fidelity, video event recognition accuracy	Use a proprietary material set for the domain
Capability	Generation quality and style	Factual error rate, format compliance rate, human preference win rate	Double-blind human review + model-based review
Engineering	Stability	Service availability, P95/P99 latency, rate-limit trigger rate	Load testing + production monitoring and alerts
Engineering	Throughput and concurrency	Output tokens per second, time to first token, per-account quota limit	Stepwise concurrency load testing
Engineering	Ecosystem and integration	Structured-output support, function-calling reliability, streaming and batch APIs	API consistency tests
Cost	Pricing structure	Separate input/output pricing, cache-hit price, batch discounts	Vendor documentation and invoice reconciliation
Cost	Total task cost	Cost per 1,000 tasks, cost per valid result	Measured invoice attribution
Governance	Compliance and security	Data residency, log-retention policy, content-safety filtering, auditability	Contract terms + configuration review
Governance	Portability	Prompt migration cost, abstraction-layer modification cost	Migration exercise

Three dimensions are especially easy to overlook but have the largest impact on total cost.

Context length is a cost amplifier, not only a capability metric. Output tokens are usually more expensive than input tokens, and long context means every request may pay the input cost for the entire history again. A model that claims to support very long context can become costly in multi-turn conversations if historical content is re-sent in full for every request.

Time to first token determines user experience, while throughput determines unit cost. In streaming scenarios, users perceive time to first token (TTFT); in batch scenarios, output tokens per second (TPS) matter more. These two metrics often trade off against each other and must be measured separately.

Version drift is a long-term risk. Cloud models may be updated without explicit notice, causing the same prompt to produce different results across months. During selection, confirm whether model version snapshots can be pinned, and prepare regression-evaluation gates for critical workflows.

3. Turn subjective impressions into data with private evaluation sets

Public leaderboards can only be used for initial screening. The real decision should be based on a self-built evaluation set. The construction method is as follows.

1. Source samples from real usage. Sample from de-identified historical tickets, real user-query logs, and internal documentation QA records rather than copying public question banks. The sample size does not need to be huge, but it must cover real long-tail cases.

2. Design stratified samples. Stratify by task type (QA, summarization, extraction, coding, multi-turn rewriting), difficulty, input length, language, and domain-term density. Keep enough samples in each stratum so that overall averages do not hide local collapse. In practice, two models may differ by only two or three percentage points in overall accuracy but differ by twenty percentage points on a specific stratum—and that stratum may be the main battleground of the business.

3. Combine scoring mechanisms.

   • Rule-based scoring: suitable for tasks with deterministic answers, such as extraction, classification, format constraints, and code executability. It is the cheapest and most stable method and should be the main scoring approach.
   • Double-blind human review: suitable for subjective dimensions such as naturalness, persuasiveness, and professionalism. A clear rubric is required, and inter-rater consistency should be checked.
   • Model-based review: useful for scaling evaluation, but known biases must be controlled—position bias (preferring the earlier answer), length bias (preferring longer answers), and self-preference bias (preferring outputs from the same model family). In practice, swap the order of candidates, score both orders, and take the average. Do not use length as a scoring rationale, and periodically calibrate with human-labeled samples.

4. Consider statistical significance. Do not make decisions based on a 2% gap from 50 samples. After fixing the evaluation set, report confidence intervals and run significance tests on paired samples. If the sample size is insufficient, mark the result as “indistinguishable” rather than manufacturing a false ranking.

5. Add regression gates to CI. Automatically rerun the evaluation after changing a model version, modifying system prompts, or adjusting retrieval strategies. Block release if any core metric falls below a threshold. This step is the foundation of long-term cost control.

Example evaluation harness:

# eval_harness.py — turn model selection into reproducible experiments
CASES = load_jsonl("eval_set_v3.jsonl")  # fields: id, task_type, difficulty, prompt, rubric
def rule_score(case, answer) -> float:
    """Deterministic tasks use rule-based scoring: most stable and cheapest"""
    if case.task_type == "code":
        return run_in_sandbox(answer, case.tests)      # unit-test pass rate
    if case.task_type == "extract":
        return field_f1(answer, case.gold)             # field-level F1
    return format_compliance(answer, case.schema)      # format compliance rate
def judge_score(case, answer, judge_model) -> float:
    """Use model-based review for subjective dimensions; swap order twice to suppress position bias"""
    a = judge_model(case.rubric, cand_a=answer, cand_b=PLACEHOLDER)
    b = judge_model(case.rubric, cand_a=PLACEHOLDER, cand_b=answer)
    return (a.prefer_first + (1 - b.prefer_first)) / 2
def run(model, temperature=0.0, repeats=3):
    rows = []
    for c in CASES:
        for _ in range(repeats):                        # multiple samples to quantify variance
            ans = model.generate(c.prompt, temperature=temperature)
            rows.append({"id": c.id, "layer": c.difficulty,
                         "score": rule_score(c, ans) if c.deterministic
                                  else judge_score(c, ans, JUDGE)})
    return aggregate_by_layer(rows)                     # output stratified metrics + confidence intervals

4. Cost model: low unit price does not mean low total cost

When selecting a model, convert the unit price into “cost per 1,000 effective tasks” using the following formulas:

Cost per request = (input tokens × input price + output tokens × output price) / 1,000,000
Effective cost = cost per request / first-pass success rate   # retries multiply the cost of tasks that need them

Example calculation (numbers are for method demonstration only; refer to the vendor’s current documentation for actual values): assume a summarization task averages 8,000 input tokens and 600 output tokens.

• Option A: input CNY 2 per million tokens, output CNY 12 per million tokens → (8000×2 + 600×12) / 1,000,000 ≈ CNY 0.023 per request
• Option B: input CNY 15 per million tokens, output CNY 75 per million tokens → (8000×15 + 600×75) / 1,000,000 ≈ CNY 0.165 per request

The unit-price gap is about 7x, and the total task-cost gap is also about 7x—enough to decide whether a feature is fully open or limited to a trial quota. If Option B has a 98% first-pass success rate and Option A has 88%, including retries narrows the gap to about 6x, which is still decisive.

Cost optimization should therefore follow this priority order:

1. Context slimming: inject only necessary history, summarize and compress instead of re-sending everything. For static instruction blocks, use APIs that support prefix caching; input billing for cached portions is usually significantly lower.
2. Tiered routing: use a smaller model for high-frequency simple requests first, and escalate only low-confidence or high-value requests to a larger model.
3. Prompt trimming: remove redundant examples and courtesy phrasing. Prompts are paid per request, and the cost accumulates over time.
4. Batch processing and asynchronous workflows: use batch APIs for offline tasks, which usually have discounts. This also smooths traffic peaks and reduces peak quota pressure.
5. Semantic caching: build a vector cache for frequently repeated semantics (different phrasings of the same intent). A cache hit means zero model cost and lower latency.

5. Scenario-based selection matrix

With evaluation data in hand, recommendations should be scenario-specific rather than a generic ranking. Note that the word “combination” in the matrix matters.

Scenario	Primary requirement	Selection preference	Key checks
Code generation and defect fixing	Unit-test pass rate, cross-file understanding	Models with strong code-specific capability	Sandbox execution results, context-injection strategy
Content creation and operations	Natural style, controllable format	Strong long-text writing model + image-generation model	Human preference win rate, factual error rate
Research and literature processing	Long-document reasoning, citation accuracy	Very-long-context model + strong reasoning model	Citation verifiability rate, hallucination rate
Investment and market analysis	Timeliness, multi-source aggregation	Model with real-time retrieval capability + deep-analysis model	Information freshness, source traceability
Daily QA and customer support	Response speed, cost, stability	Mid-sized model with strong cost-performance	P95 latency, cost per request, fallback strategy
Enterprise batch offline processing	Unit cost, throughput	Low-cost, high-throughput model + batch API	Effective cost, retry failure rate
High-compliance intranet scenarios	Data cannot leave the domain, auditable	Private or dedicated deployment option	Deployment mode, log auditing, model version locking