LLM Evaluation and Model Selection Methodology for Budget-Constrained Teams

When the price of a flagship closed-source model exceeds the budget, model selection stops being “which one should we buy?” and becomes “how should we test, calculate, and use it?” Many teams start by checking leaderboard rankings, then look at comparison charts from official launches, and rarely run a full evaluation of their own before production integration. The result is often one of these problems:

• The unit price looks acceptable, but the model output is overly verbose and token consumption is high.
• The leaderboard score looks good, but accuracy drops significantly on the team’s own business corpus.
• Average response time looks acceptable, but the real user experience suffers because time to first token is too long.

At a higher level, the real questions are:

1. Capability: On the specific tasks we care about, is the model accurate enough to meet a usable threshold?
2. Efficiency: Under realistic concurrency, does the latency distribution satisfy product experience and timeout budgets?
3. Economics: When converted into “the cost of one successful business action,” is the model cheaper than the existing option?

These three questions correspond to capability, efficiency, and economic metrics. If any layer is missing, the conclusion will be distorted. Everything below is built around this three-layer metric system, and the most important rule is simple: trust the numbers you measure yourself.

1. Three-Layer Metrics: Turning Subjective Impressions into Comparable Numbers

1.1 Capability Layer: Accuracy, Domain Accuracy, and Task Completion Rate

The capability layer is often reduced to a single overall score, but that is usually the least informative number. What matters more is a domain capability matrix: split business tasks by domain and measure accuracy separately. Model strength varies widely across domains. An average score can hide both a large improvement in one domain and a quiet regression in another.

1.2 Efficiency Layer: Latency Percentiles, Throughput, and Token Consumption

The efficiency layer should at least include:

• TTFT: time to first token;
• TPOT: time per output token;
• End-to-end latency percentiles: P50, P90, and P99;
• Average token consumption.

The first three determine interaction quality. The last one directly affects cost. Throughput should be evaluated under the target concurrency level rather than inferred from isolated single-request tests.

1.3 Economic Layer: Cost per Completed Task

The economic layer is not “how much does one million tokens cost?” It is how much does one successfully completed task cost? The former is unit price. The latter is what can actually support decision-making. The gap between them comes from hidden overhead such as retries, invalid output, and manual review.

2. Capability Evaluation: The Question Bank Determines How Trustworthy the Conclusion Is

2.1 Question Volume and Mix

Too few questions make the conclusion drown in noise. In practice, a domain-specific evaluation set often contains hundreds of questions per domain and several thousand to tens of thousands in total. The domain mix should match real traffic distribution, not be evenly divided by academic subject. If 70% of production traffic is code-related requests, the evaluation set should also be approximately 70% code-related.

2.2 Difficulty Stratification

It is useful to construct the question bank in three difficulty levels: easy, medium, and hard. This makes it easier to see whether a version upgrade only improves performance on easy questions.

A common trap is this: a new version increases the overall score by 2 percentage points, but the entire gain comes from easy questions, while hard-question performance actually declines. The production impact may be much larger than the overall score suggests.

2.3 Answer Scoring Methods

The scoring method must match the business use case:

• Objective questions such as multiple choice, extraction, and classification: exact match or rule-based validation.
• Code questions: compile and pass unit tests.
• Structured output questions: JSON Schema validation pass rate plus field correctness rate.
• Open-ended generation questions: model review plus human sampling calibration. The review model must be fixed, and its version must be recorded.

2.4 Domain Capability Matrix (Illustrative)

The table below is an illustrative evaluation report structure. The values are only used to demonstrate the format and do not represent any specific model.

Capability dimension	Baseline accuracy	Candidate accuracy	Change
Overall score	66.6%	68.7%	+2.1pp
Legal and public administration	72.7%	80.3%	+7.6pp
Education	41.2%	46.9%	+5.7pp
Healthcare and mental health	73.9%	77.6%	+3.7pp
Reasoning and math	62.2%	65.8%	+3.6pp
Finance	76.9%	77.7%	+0.8pp
Language and instruction following	66.2%	64.6%	-1.6pp
Agent and tool use	72.7%	69.9%	-2.8pp

The value of this table is that it clearly shows the possible gap between “capability improvements advertised officially” and “capabilities measured under your own evaluation criteria.” When any dimension regresses, regardless of the size, it should trigger manual review to determine whether the cause is question-bank bias or real capability degradation.

3. Latency Evaluation: A Single Average Is Almost Always Misleading

Average latency is one of the most frequently cited metrics, but it is also one of the easiest to distort. A single long-tail timeout can inflate the average by several times. Conversely, a beautiful average measured under low concurrency can collapse quickly under high concurrency.

At minimum, evaluation should collect:

• TTFT: time from sending the request to receiving the first token. This determines whether the interaction feels stuck.
• TPOT: average generation time for subsequent tokens. This determines whether output feels fast to read.
• End-to-end latency percentiles: P50, P90, P95, and P99.
• Timeout rate and retry rate: the proportion of requests interrupted by the client after exceeding the threshold.

When collecting data, fix the concurrency level, disable client caching, and record rate-limit response codes. The following code is for illustration only.

import asyncio, os, time
import httpx
CONCURRENCY = int(os.environ.get("EVAL_CONCURRENCY", "8"))
BASE_URL = os.environ["EVAL_BASE_URL"]   # self-hosted proxy or platform endpoint
TIMEOUT_BUDGET = 60.0                    # upper limit acceptable to the business
async def one_call(client: httpx.AsyncClient, sem: asyncio.Semaphore, prompt: str) -> dict:
    async with sem:                      # semaphore limits concurrency to simulate real load
        payload = {
            "model": os.environ["EVAL_MODEL"],
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.0,          # fix randomness for cross-version comparability
            "stream": True,              # streaming is needed to separate TTFT
        }
        t0 = time.perf_counter()
        ttft, out_tokens, timed_out = None, 0, False
        try:
            async with client.stream("POST", "/chat/completions",
                                     json=payload, timeout=TIMEOUT_BUDGET) as resp:
                resp.raise_for_status()
                async for line in resp.aiter_lines():
                    if not line:
                        continue
                    if ttft is None:
                        ttft = time.perf_counter() - t0
                    out_tokens += 1       # in production, read the usage field
            total = time.perf_counter() - t0
        except (httpx.ReadTimeout, httpx.ConnectTimeout):
            total, timed_out = time.perf_counter() - t0, True
        return {"ttft": ttft, "total": total, "out_tokens": out_tokens, "timed_out": timed_out}
async def run(cases: list[str]) -> list[dict]:
    sem = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient(base_url=BASE_URL, http2=True) as client:
        return await asyncio.gather(*(one_call(client, sem, c) for c in cases))

For statistics, keep one principle: always report percentiles, not only the average.

import statistics
def summarize(results: list[dict]) -> dict:
    ok = [r for r in results if not r["timed_out"] and r["ttft"] is not None]
    totals = sorted(r["total"] for r in ok)
    def pct(p):  # linear interpolation percentile
        if not totals:
            return None
        k = (len(totals) - 1) * p
        lo, hi = int(k), min(int(k) + 1, len(totals) - 1)
        return totals[lo] + (totals[hi] - totals[lo]) * (k - lo)
    return {
        "samples": len(results),
        "timeout_rate": 1 - len(ok) / max(len(results), 1),
        "ttft_p50": statistics.median(r["ttft"] for r in ok) if ok else None,
        "total_p50": pct(0.50), "total_p90": pct(0.90),
        "total_p95": pct(0.95), "total_p99": pct(0.99),
        "avg_out_tokens": statistics.fmean(r["out_tokens"] for r in ok) if ok else 0,
    }

4. Cost Evaluation: From Token Price to Cost per Completed Task

4.1 Basic Conversion

Cost per call = average input tokens × input price + average output tokens × output price. For 1,000 calls, multiply by 1,000 and add an overhead factor.

def cost_per_1k(in_tokens: float, out_tokens: float,
                price_in: float, price_out: float,
                retry_rate: float = 0.03, invalid_rate: float = 0.0) -> float:
    """price_in and price_out are in currency units per 1,000 tokens.
    Use the official price list for actual billing."""
    base = (in_tokens / 1000) * price_in + (out_tokens / 1000) * price_out
    overhead = 1 / max(1 - invalid_rate, 0.01)   # invalid output requires repeat calls
    return base * 1000 * (1 + retry_rate) * overhead

4.2 Hidden Overhead Checklist

• Retries: rate limiting, timeouts, and content filtering can all cause repeated billing.
• Invalid output: format violations and hallucinations can cause rework that is often more expensive than the unit-price difference.
• Context expansion: in multi-turn conversations, repeatedly carrying history can multiply input tokens by several times.
• Manual review: low accuracy means more human fallback work. This cost is often completely ignored.

4.3 Optimization Levers

Improving cache hit rate by reusing identical prefixes, trimming unnecessary history turns, routing simple requests to smaller models, enabling batch APIs, and setting max_tokens limits to suppress verbose output can usually reduce cost more than simply switching models.

5. Long Context and Agent Capabilities: Build Separate Evaluation Sets

Improvements such as expanding the context window from 128K to 200K should not be judged by the nominal value alone. What matters is factual recall within the effective window. A dedicated “needle in a haystack” style evaluation is recommended: insert several pieces of key information at different depths and positions in a long document, ask questions about them, and measure how recall accuracy decays with position and length.

A common real-world phenomenon is this: a model may officially support very long context, but recall for information in the middle of the document is significantly lower than recall for information at the beginning or end.

Agent capabilities should be evaluated separately from text generation capabilities. At least four metrics are needed:

1. Tool selection accuracy: given multiple candidate tools, does the model select the correct target tool?
2. Parameter filling accuracy: are parameter names, types, enum values, and required fields complete?
3. Multi-step task success rate: for tasks that require multiple tool calls and reasoning over intermediate results, can the model complete the task end to end?
4. Tool calling during reasoning: can the model correctly invoke tools and integrate results while producing reasoning content, rather than waiting until the very end?

A practical evaluation approach is to use a sandboxed tool set: provide several restricted fake tools, such as listing a directory, reading a file, calculating, and writing a file to a temporary directory. All tasks run in an isolated environment, which allows automatic scoring without producing real side effects.

6. Reproducibility: Make Evaluation Conclusions Survive Review

• Fix the baseline: version-control the question bank, prompts, temperature, max output, scoring script, and model version together to form an immutable evaluation snapshot.
• Isolate variables: change only one variable at a time, such as model version, prompt, or concurrency. Otherwise, results cannot be attributed.
• Statistical significance: report confidence intervals for accuracy. If the difference between versions is smaller than sampling error, do not claim an improvement.
• Regression gate: integrate evaluation into continuous integration. If accuracy drops beyond a threshold, such as a 3 percentage point drop in any domain, block release.

7. Deployment Strategy: Tiered Routing Instead of One Model for Everything

The real output of evaluation is not a ranking, but a cost-quality curve. Based on that curve, you can design tiered routing:

• Route simple intent recognition, format conversion, and summarization tasks to low-cost small models.
• Route tasks that require multi-step tool calls or long-document reasoning to stronger models.
• Route latency-sensitive interactive paths to small models, and precision-sensitive offline batch processing to large models.

def route(task: dict) -> str:
    if task["need_tools"] or task["context_tokens"] > 32_000:
        return "strong_model"
    if task["difficulty"] == "hard":
        return "mid_model"
    return "light_model"

To turn evaluation from a one-time activity into a permanent capability, combine it with asynchronous processing, object-storage archiving, centralized log search, and monitoring alerts on timeout rate and error rate. Non-real-time tasks can be placed into message queues to smooth peaks, evaluation results can be archived in object storage, and logs can be centrally searched and monitored.

If you use a cloud platform such as Baidu AI Cloud, this workflow can be built from general product categories: container hosting for evaluation jobs, logging and monitoring/alerting for observability, object storage for evaluation snapshots and samples, and message queues for asynchronous decoupling. No vendor-specific interfaces are required.

8. Selection Checklist

1. Did you complete evaluation with your own business question bank instead of relying only on public leaderboards?
2. Did you output a domain capability matrix and identify regression items?
3. Did you report P90/P99 latency and timeout rate, and collect data under the target concurrency?
4. Did you convert cost into both “cost per 1,000 calls” and “cost per successful task”?
5. Did you separately validate the effective long-context window and tool-use capability?
6. Can the evaluation process be reproduced with one click and included in CI as a regression gate?
7. Did you design tiered routing and caching strategies to reduce cost per unit?

Budget constraints are not the obstacle to model selection. The real obstacle is the lack of a quantitative method. Once evaluation becomes an engineering asset, replacing a model becomes routine regression testing rather than a gamble.