LLM Selection in Practice: Core Evaluation Dimensions and Methodology

The Selection Dilemma: Why “the Strongest Model” Is a False Premise

In real-world engineering projects, almost every team makes similar observations: some models deliver the strongest overall capability but also carry the highest inference cost and response latency; some excel at deep reasoning and produce highly complete answers, yet deliver them slowly; some are well-rounded but inconsistent in reasoning depth, making their performance unstable; and others are inexpensive and occasionally brilliant, but weak overall—prone to perfunctory answers when facing complex tasks.

These four profiles are not about any specific products—they are a snapshot of today’s LLM market: capability, speed, cost, and stability are inherently subject to trade-offs. The first step of model selection is to abandon the obsession with finding “the strongest model” and instead answer a different question: in my business scenario, which dimension carries the most weight?

Six Core Evaluation Dimensions

1. Overall Capability: Benchmark Scores Are Only the Starting Point

Public benchmarks (knowledge QA, code generation, mathematical reasoning, instruction following, etc.) can quickly narrow down your candidate pool, but two caveats apply:

• Training data contamination: Some benchmark questions may have made their way into a model’s training corpus, inflating its scores.
• Distribution mismatch: Benchmark questions don’t necessarily match the distribution of your business data—strong general capability doesn’t guarantee strong performance in your domain.

Practical advice: use public benchmarks for the initial screening, and build a private evaluation set from real business data for the final verdict.

2. Reasoning Depth and Thinking Modes

Many models offer a “deep thinking” or “extended reasoning” mode that generates longer chains of thought in exchange for higher accuracy on complex tasks. Typical characteristics of such modes: slower answers, more thorough answers, and high token consumption. They suit high-value tasks such as architecture design, tricky defect localization, and multi-step mathematical derivation; using them for high-frequency, simple Q&A is a clear waste of resources.

3. Response Speed and Throughput

Three metrics need to be distinguished:

• Time to first token (TTFT): The time from sending a request to seeing the first token—it directly determines how responsive the interaction feels.
• Output speed (TPS): Tokens generated per second, which drives how long users wait for long outputs.
• End-to-end latency: Total time including queuing, networking, and post-processing. In production, focus on P95/P99 figures rather than averages.

4. Output Completeness and Stability

Completeness means the model fully covers the task’s requirements—no skipped steps, no corner-cutting, and strict adherence to the output format. Stability means low variance in results when the same request is invoked multiple times. What practitioners often describe as a model “getting dumbed down” usually traces back to two causes: fluctuating server-side load or routing policies, and randomness introduced by unfixed sampling parameters and prompts. Engineering mitigations include fixing the temperature, spelling out acceptance criteria in the prompt, and adding output validation scripts.

5. Cost-Effectiveness: Judge the Total Bill, Not the Unit Price

Explicit costs are the token unit price and call volume. Hidden costs are often overlooked: failed retries, post-processing to repair formatting, manual spot-checking, and rework caused by low-quality output. A more sensible metric is cost per unit of quality—the total cost of obtaining each unit of acceptable output. A cheap model with a high rework rate is not necessarily cheaper on the total bill.

6. Ecosystem and Engineering Support

This dimension covers context window length, function calling, structured output, multimodal support, API style compatibility, private deployment options, and data compliance capabilities. It is frequently underestimated: no matter how capable a model is, without solid monitoring and alerting, logging services, and canary-release support, deployment costs climb significantly. When choosing a model service platform, look at the general capability categories mainstream cloud providers offer—model access, monitoring and alerting, logging services, and so on—and factor engineering support into your overall score.

Build Your Own Evaluation System

Rather than collecting leaderboards, spend two days writing a lightweight evaluation script. The core idea: extract real cases from your business logs, cover the high-frequency task types, run each case multiple times, and collect quality scores, latency, and token consumption together.

import time, statistics
from dataclasses import dataclass
from typing import Callable
@dataclass
class Case:
    task_type: str                    # Task type: code generation / long-text summarization / complex reasoning...
    prompt: str
    checker: Callable[[str], float]   # Validation function, returns a 0-1 score
def run_eval(client, cases: list[Case], runs: int = 3):
    report = {}
    for c in cases:
        scores, latencies, tokens = [], [], []
        for _ in range(runs):             # Run multiple times to observe stability
            t0 = time.time()
            resp = client.chat(
                messages=[{"role": "user", "content": c.prompt}],
                temperature=0.2,          # Fix sampling parameters to reduce randomness
            )
            latencies.append(time.time() - t0)
            scores.append(c.checker(resp.text))
            tokens.append(resp.usage.total_tokens)
        report[c.task_type] = {
            "avg_score":   round(statistics.mean(scores), 3),
            "score_std":   round(statistics.pstdev(scores), 3),   # Std dev = stability
            "p95_latency": round(sorted(latencies)[int(len(latencies)*0.95)-1], 2),
            "avg_tokens":  round(statistics.mean(tokens), 1),
        }
    return report

Interpreting the four metrics together:

• High avg_score → capability meets the bar;
• Low score_std → stable performance, safe to put into production;
• Low p95_latency → worst-case experience stays under control;
• avg_tokens × unit price → cost per call.

For subjective tasks such as copywriting and summarization, you can bring in a “model as judge” for automatic scoring—but watch out for the judge model’s own bias. Keep a certain share of manual spot checks for calibration.

Scenario-Based Selection and Hybrid Routing

Different scenarios weight these dimensions very differently:

Scenario	Priority dimensions	Fitting profile
Complex reasoning, architecture design, tricky debugging	Reasoning depth, completeness	Deep thinker, tolerant of high latency
High-frequency interaction (chat, completion, customer service)	TTFT, stability	Balanced
Large-scale offline processing (cleaning, labeling, summarization)	Cost, throughput	Cost-effective
Compliance-sensitive scenarios	Deployment model, data compliance	Private / dedicated instances

You can go a step further and replace “one model for everything” with hybrid routing: first run a lightweight difficulty assessment on each request, send simple requests to a low-cost model, and route only complex requests to a deep-reasoning model. Overall cost drops significantly while quality stays essentially intact.

def route(prompt: str) -> str:
    difficulty = estimate_difficulty(prompt)   # Rules or a lightweight classifier
    if difficulty > 0.7:
        return "deep_reasoning"    # Deep-reasoning model
    if is_interactive(prompt):
        return "low_latency"       # Low-latency model
    return "cost_efficient"        # Cost-efficient model

Five Common Pitfalls

1. Benchmark worship: High leaderboard scores don’t mean the model works well for your business—your private evaluation set is the final arbiter.
2. One evaluation, decided forever: Models iterate on a weekly cadence; establish a monthly regression evaluation routine.
3. Looking only at averages, not variance: A model with a nice average but high variance usually delivers a poor production experience.
4. Counting only token unit price, not the total bill: Retries, rework, and manual verification are all real costs.
5. One prompt to rule every model: Models vary widely in their sensitivity to prompt style. When switching models, be sure to re-adapt your prompts instead of drawing conclusions from a direct head-to-head comparison.

Conclusion

The essence of LLM selection is matching a model’s “profile” against quantifiable business metrics—not chasing a nonexistent all-around champion. The recommended path: shortlist with public benchmarks → finalize with a private evaluation set → run two models in parallel as an A/B test → roll out gradually → re-evaluate on a periodic schedule. Only by turning selection from a one-time decision into a continuous engineering practice can you keep the optimal configuration amid rapid model iteration.