LLM Selection Framework: Six Evaluation Dimensions and How to Put Them Into Practice

1. Fixing the question first: from “which model is best” to “which model fits”

While model capabilities iterate quickly, any “top of the leaderboard” conclusion has a shelf life measured in weeks. Durable engineering judgment comes from the intersection of three constraints:

• Task constraint: Are you solving code completion, long-document summarization, multi-turn customer support, or mixed image-and-text understanding? Tolerance for error varies enormously across these.
• Engineering constraint: Latency budget, concurrency ceiling, context length, and whether you must deploy privately or keep data from leaving your network.
• Economic constraint: Cost per call × daily call volume × retry amplification factor — that is the real bill.

So the first step is not reading leaderboards. It is slicing your business into a set of “judgeable task units,” each with a defined input shape, expected output, acceptance criteria, and acceptable cost per item. Only once the task units are defined does a side-by-side comparison mean anything.

2. The six evaluation dimensions and how to measure them

2.1 General knowledge and reasoning

Public benchmark suites — for example, graduate-level science QA benchmarks, or software engineering benchmarks built on defect fixes in real code repositories — are useful for initial screening but useless as a final verdict. Two reasons: these datasets carry a risk of training-data contamination, and their distribution has almost nothing to do with your private data. The recommended approach is to pull questions from public benchmarks as a “capability floor probe,” then build your own 50–200 business questions as the “capability final judgment.”

2.2 Domain task accuracy

For each task unit, annotate gold answers and score with structured checks rather than human impressions. The more machine-checkable fields you define, the more stable the evaluation:

def score_case(pred: dict, gold: dict) -> dict:
    return {
        "field_exact": all(pred.get(k) == v for k, v in gold.items()),
        "schema_ok": set(pred.keys()) >= set(gold.keys()),
        "json_parse_ok": isinstance(pred, dict),
    }

2.3 Long context and information retrieval

A context window’s advertised size is not its effective size. What matters more in engineering is “mid-document recall”: in a 100K-token document, plant a key fact at a random position in the beginning, middle, or end, and test whether the model can cite it reliably. In practice, most models show attention decay in the middle of very long contexts, so you compensate with retrieval-first pipelines, chunked summarization, and re-injection of key passages.

2.4 Multimodal and real-time information

For multimodal capability, distinguish “can see an image” from “can read a layout”: report tables, scanned PDFs, UI screenshots, and handwriting form a rising difficulty curve. Always test with the hardest 20 images from your own business. For real-time information, distinguish two implementations: retrieval built into the model, versus context assembled on your side. The latter gives you more control and makes citation provenance easier to audit.

2.5 Joint optimization of cost and throughput

Split price into two variables — input unit price and output unit price — and estimate against your real token distribution:

cost_per_call = input_tokens / 1e6 × input_price + output_tokens / 1e6 × output_price
monthly_cost  = Σ cost_per_call × (1 + retry_rate) − cache_discount

The sketch below shows how to record quality and cost in the same evaluation framework, so you never end up comparing quality while ignoring the bill:

import time
from statistics import mean
def run_eval(cases, client, prompt_builder, model_spec):
    rows = []
    for c in cases:
        prompt = prompt_builder(c)
        t0 = time.perf_counter()
        resp = client.chat(model=model_spec.name, messages=prompt)
        latency = time.perf_counter() - t0
        in_tok = resp.usage.prompt_tokens
        out_tok = resp.usage.completion_tokens
        cost = (in_tok * model_spec.input_price
                + out_tok * model_spec.output_price) / 1_000_000
        rows.append({
            "id": c.id,
            "latency_s": round(latency, 3),
            "cost": cost,
            "score": score_case(resp.json_content, c.gold),
        })
    return {
        "quality": mean(r["score"]["field_exact"] for r in rows),
        "p50_latency": sorted(r["latency_s"] for r in rows)[len(rows)//2],
        "total_cost": sum(r["cost"] for r in rows),
    }

Three practical details matter here. Pin the evaluation set and the random seed (set temperature to 0 if the model supports it). Record token usage — it is the only reliable basis for cost attribution. Record latency percentiles rather than the mean — tail latency is what users actually feel.

2.6 Stability, compliance, and replaceability

This dimension is the easiest to overlook and the easiest to turn into an incident:

• Rate limits and quotas: Confirm requests per minute, tokens per minute, daily caps, and the error-code semantics when you exceed them (retryable vs. non-retryable).
• Availability degradation: Keep a cheaper, slightly weaker fallback model for the primary model, and switch automatically on timeout or when the error rate crosses a threshold.
• Data boundaries: Define which data may leave the corporate network, which field types need masking (phone numbers, national ID numbers, customer names), and your log retention policy.
• Account and network risk: Bulk calls from unstable network environments easily trigger abuse controls. Enterprise scenarios should prefer a formal onboarding path with a service-level commitment rather than depending on individual accounts.

3. Mapping to scenarios: a capability matrix, not a leaderboard

Scenario	Key capabilities	Secondary concerns	Typical trade-off
Code generation and refactoring	Code benchmark score, long-file understanding, tool-call stability	Output unit price	Prioritize accuracy; cost comes second
Long-document QA	Mid-context recall, traceable citations	Input unit price	Retrieval quality sets the ceiling
Multi-turn customer support	Instruction following, refusal and safety boundaries	Latency percentiles	Use a small model for high-frequency simple turns
Mixed image-and-text understanding	Layout parsing, structured output stability	Per-image billing	Gate on the hardest samples first
Content creation	Naturalness of style, controllability	Output unit price	Requires human evaluation and preference scoring
High-concurrency batch processing	Throughput, quotas, unit price	Quality floor	Small model plus sampled review is acceptable

The point of this table: a model that is “strong at code” may be uneconomical for batch cleaning because of its unit price, and a model that is “balanced overall” may be unable to carry a core path because it lacks a specific modality. There is no all-around champion — only the best combination.

4. Routing and load shedding: turning selection into an architectural capability

Once a model is no longer your only dependency, selection risk shifts from “betting” to “configuration.” The recommended engineering structure has three layers:

def route(task, features):
    if features["needs_vision"]:
        return "vision_model"
    if task.risk == "high":          # high-risk tasks go straight to the strong model
        return "strong_model"
    if features["tokens_in"] > 100_000:
        return "long_context_model"  # very long context uses a dedicated channel
    return "cheap_model"             # default to the low-cost channel

Two optimizations cut the bill substantially:

1. Semantic caching: Build a vector cache for high-frequency similar questions and return on a hit. This typically removes 20%–40% of duplicate calls, with the actual benefit depending on your question distribution.
2. Cascade escalation: Answer with the cheap model first, and escalate to the strong model only when confidence is low (structured field checks fail, schema does not match, or a confidence score falls below a threshold).

def cascade_answer(question, checks, cheap, strong):
    ans = cheap(question)
    if all(chk(ans) for chk in checks):
        return ans, "cheap"
    return strong(question), "strong"

Do the math on cascading before adopting it: it pays off only when the cheap model’s first-pass rate is clearly above its price ratio. If the first-pass rate is too low, the volume of strong-model calls pushes total cost up instead of down.

5. Implementation checklist

1. Build a 50–200 item business evaluation set covering normal, boundary, and adversarial inputs, and put it under version control.
2. Standardize a model access abstraction (consistent client interface, observable parameters and usage) so switching models is a configuration change.
3. Persist a record for every call: request hash, model version, token usage, latency, and verdict — this becomes your regression baseline.
4. Wire evaluation into CI. On model version upgrades, run the regression automatically and block the release if quality drops beyond the threshold.
5. Monitor four metrics: error rate, P95 latency, cost per call, and the pass rate of sampled human review.
6. Complete masking and data classification on the data side, and define what may be sent externally. For capabilities touching sensitive data, prefer compliant managed cloud options where available in your region — for example, Baidu AI Cloud’s hosted model services together with generic cloud capabilities such as object storage, logging, and monitoring/alerting — so you can reuse the platform’s existing access control and audit framework.

6. Conclusion

The right posture toward selection is to treat it as a sustainable engineering capability: replace subjective impressions with a judgeable evaluation set, replace price intuition with a token-level cost model, and replace single bets with routing and degradation. When model iteration moves faster than your evaluation cycle, the only asset that never gets obsolete is the evaluation and switching mechanism itself.