LLM Model Selection Evaluation Framework: Dimension Breakdown and Engineering Practice

1. Breaking “Overall Performance” into Measurable Dimensions

A vague notion of “how capable is it” cannot guide decisions. You first have to decompose it into individually scorable metrics.

1.1 Semantic and Reasoning Capability

This covers multi-step logical deduction, numerical and symbolic computation, constraint-satisfaction problems, and intermediate-step consistency in long-chain tasks. One caveat during evaluation: single-sample results are highly random, so you must fix the random seed or average over multiple samples. Otherwise, differences between models get drowned out by sampling noise.

1.2 Code and Structured-Output Capability

Code capability should not be reduced to “can it write correct logic.” Break it down further into:

• Single-file, function-level generation and completion;
• Cross-file, cross-module refactoring and dependency inference;
• Code review and defect localization (can it flag boundary conditions and concurrency risks);
• Adherence to existing coding conventions and directory structure;
• Format stability and field completeness of structured output such as JSON, tables, and call parameters.

1.3 Long Context and Long-Document Processing

This is the dimension most easily misled by “advertised parameters.” The context-window length is only an upper bound. What actually affects the experience is the effective utilization of information within a long context: recall of information in the middle of the input, cross-paragraph coreference resolution, coverage of key points when summarizing very long documents, and how first-token latency and cost grow as input length increases.

1.4 Tool Use and Task Closure

How complete an agentic application is depends on whether the model can correctly select tools, construct valid parameters, parse returned results, and self-correct after failures. The evaluation points here are task-level success rate (end-to-end success) and average number of steps, not per-step accuracy.

1.5 Multimodal Capability

Image understanding, layout and chart parsing, on-screen element localization from screenshots, and structured extraction from mixed text-image documents all need to be evaluated against real business samples, not general-purpose image-benchmark scores.

1.6 Engineering and Business Constraints

This includes API stability and rate-limiting policy, error codes and retry semantics, streaming-output support, data-retention and compliance policy, regional availability, and transparency of quotas and billing. These dimensions do not feed into the “capability score,” but they are often the single veto item that ultimately decides whether something can ship.

2. Building a Reproducible Private Evaluation Set

Public leaderboards carry task-distribution bias, so you must build your own evaluation set. A recommended structure:

from dataclasses import dataclass, field
@dataclass
class EvalCase:
    case_id: str
    dimension: str            # reasoning / code / long_ctx / tool_use / multimodal
    prompt: str
    reference: str = ""       # reference answer or reference implementation
    rubric: dict = field(default_factory=dict)  # custom scoring rules
    weight: float = 1.0
    tags: list = field(default_factory=list)    # e.g. "cn", "frontier", "strict_json"

Engineering considerations:

1. Layered sample sizes: cover at least 30–100 real business samples per dimension, with long-tail scenarios broken out as their own layer so that averages do not mask weak spots.
2. Keep a golden set and a development set separate: the golden set is isolated throughout and used only for final acceptance, which prevents the implicit overfitting that comes from repeatedly tuning prompts.
3. Fix the evaluation configuration: temperature, maximum output length, system prompt, and timeout must be version-controlled and recorded; any drift in one of these invalidates historical conclusions.
4. Multiple samples plus statistical significance: for critical dimensions, sample with n ≥ 3, report both mean and variance, and do not declare a winner when the difference is smaller than the variance.

3. Scoring: From Capability Score to Decision Score

A single aggregate score almost inevitably misleads. Instead, use a weighted decision model of “capability score × scenario weight”:

SCENARIO_WEIGHTS = {
    # Dimension weights are configured per business scenario; the total is normalized to 1.0
    "customer_support": {"reasoning": 0.2, "long_ctx": 0.15, "tool_use": 0.3,
                         "code": 0.05, "latency": 0.15, "cost": 0.15},
    "code_assist":      {"reasoning": 0.2, "code": 0.45, "long_ctx": 0.15,
                         "tool_use": 0.1, "latency": 0.05, "cost": 0.05},
    "doc_insight":      {"reasoning": 0.15, "long_ctx": 0.45, "multimodal": 0.25,
                         "latency": 0.05, "cost": 0.1},
}
def decide_score(metric: dict, scenario: str) -> float:
    weights = SCENARIO_WEIGHTS[scenario]
    return sum(metric.get(dim, 0.0) * w for dim, w in weights.items())

Key principles:

• Weights are confirmed and documented by the business owner, not set by engineers based on gut feeling;
• Hard requirements are listed separately as veto items (for example, data must not be retained, on-premise deployment must be supported, or streaming responses are mandatory). Failing any of these eliminates the candidate outright and excludes it from weighting;
• The report should give both the average score and the worst-dimension score, since the latter determines whether the system can deliver reliably.

4. Quantifying Cost

Comparing unit prices alone is meaningless. You must convert to the cost per unit of business output. For example, the actual cost per thousand Q&A turns, per ten thousand lines of code review, or per million tokens of document parsing, accounting for input, output, cache hits, and the amplification factor from retries.

def request_cost(in_tok: int, out_tok: int,
                 p_in_per_mtok: float, p_out_per_mtok: float,
                 retry_rate: float = 0.0) -> float:
    factor = 1.0 + retry_rate          # implicit amplification from retries and timeouts
    return (in_tok / 1e6 * p_in_per_mtok
            + out_tok / 1e6 * p_out_per_mtok) * factor

Three hidden costs are most often underestimated in practice:

1. Retries and failed-request replay: under high concurrency, rate limiting can significantly raise the true unit price;
2. Repeated input in long contexts: multi-turn conversations repeatedly carry history, so input tokens grow linearly and need to be optimized through summary compression or prefix reuse;
3. Manual rework: a model that is 5% less complete can add human-review overhead far exceeding the unit-price difference.

So “cheaper unit price” only counts as a real advantage when measured against cost per unit of output.

5. Turning Selection into System Capability: Routing and Fallback

Once a conclusion is reached, it should not be hard-coded. It should become a configurable, observable runtime policy.

5.1 Tiered Routing

Route by task complexity: simple classification, format extraction, and intent recognition go to low-cost models; complex reasoning and cross-repository refactoring go to high-performance models. An example interface:

def route(task: dict) -> str:
    if task["type"] in ("classify", "extract", "tag"):
        return "model_fast"
    if task.get("est_input_tokens", 0) > 100_000 or task.get("need_deep_reasoning"):
        return "model_reasoning"
    return "model_default"

5.2 Fallback and Circuit Breaking

When the primary model times out or returns an error code, fall back to a backup model in a predefined order, and retain a fallback marker for later attribution analysis. The fallback chain must be decoupled through asynchronous messaging to avoid synchronously blocking the main request path.

5.3 Shadow Evaluation and Canary Rollout

Before a new model goes live, run it on shadow traffic: real requests are sent in parallel to both the old and new models, but only the old result is used. After comparing the differences, decide on the rollout ratio. This keeps the risk of “switching models” within a controllable range.

6. Continuous Observation After Launch

Selection is not a one-time action. You need a standing quality dashboard:

• Offline metrics: periodically re-run the golden set and compare score drift against version-controlled records;
• Online metrics: call success rate, P95/P99 latency, number of rate-limit hits, structured-output parse-failure rate, and cost per unit of output;
• Manual spot checks: stratified sampling by scenario for review, especially for high-risk scenarios;
• Feedback loop: route user downvotes, manual corrections, and failure samples back into the evaluation set to form a continuously expanding positive cycle.

These capabilities are typically carried by generic infrastructure components such as logging services, monitoring and alerting, object storage, and message queues, and can be assembled without depending on any proprietary interface. If your organization has already deployed this infrastructure on a cloud platform such as Baidu AI Cloud, these generic components can directly reuse your existing operations setup.

7. Common Pitfalls

Pitfall	Consequence	Fix
Relying only on public leaderboards	Task-distribution mismatch causes real-world failures	Build a private eval set with an isolated golden set
Drawing conclusions from a single sample	Conclusions dominated by sampling noise	Sample multiple times, report mean and variance
Comparing only unit price	Ignores retries and rework, so total cost rises	Convert to cost per unit of business output
Looking only at the context-window limit	Poor recall of mid-document information	Evaluate long-context effective utilization separately
Defaulting to a single bound model	Vendor fluctuations directly affect availability	Abstract the model access layer; support routing and fallback
Evaluating once, forever	Conclusions expire after model updates	Periodic regression plus shadow traffic

8. Implementation Checklist

1. Define 3–5 core scenarios, and write down the acceptance criteria and hard requirements for each;
2. Build a layered private evaluation set whose sample sizes cover both high-frequency and long-tail cases;
3. Set the weights, and document them in review;
4. Run parallel evaluation across multiple models, and output the capability score and worst-dimension score;
5. Compute the cost per unit of business output, including retries and rework;
6. Design routing, fallback, circuit-breaking, and canary strategies;
7. Establish a three-layer observation system: offline regression, online monitoring, and manual spot checks;
8. Re-evaluate quarterly, and fold newly added capabilities into the scoring.

Conclusion

“Which model is strongest overall” is a static question that cannot hold up over time. The sustainable approach is to engineer the selection process: use a reproducible evaluation set to quantify capability, use scenario weights to decide trade-offs, use cost per unit of output to correct price intuition, use routing and fallback to spread out risk, and use continuous observation to fight the way conclusions expire as models are iterated. When the evaluation process itself is stable, reproducible, and regression-ready, no single model update needs to reignite the ranking debate—you just re-run the pipeline to get the answer.