LLM Model Selection and Evaluation: From Capability Dimensions to Total Cost Optimization

1. Why “all-subscription experience” is not a selection conclusion

Many teams choose LLM services by asking, “I have subscriptions to all of them, so which one feels best?” This can be useful during exploration, but it quickly breaks down in production for three reasons.

Sample bias: Personal evaluations often focus on a few familiar task types, while business traffic is usually long-tailed. A model that performs well on code refactoring may drop noticeably on structured extraction, numerical calculation, or dialect-colloquial expression.

Scenario mismatch: Interactive Q&A requires low latency and high fault tolerance. Batch offline processing requires extremely low unit cost and retryability. Features exposed to external users also need strict content safety policies. A conservative alignment policy can be an advantage in enterprise compliance scenarios, but it may appear as irrelevant answers or an excessively high refusal rate in open creative scenarios.

Unclear cost basis: Subscription price is only a fixed cost. What truly determines the bill is token consumption, retry rate, rework rate, and manual correction time. A model that succeeds on the first attempt may have a lower total cost than a model with a lower unit price but frequent rework.

Therefore, model selection should be upgraded from “experience comparison” to an engineering problem that is measurable, regression-friendly, and cost-accountable.

2. Break “overall performance” into measurable dimensions

A general-purpose LLM evaluation should cover the following dimensions. Each dimension needs its own sample set and judging criteria.

1. Code and engineering capability: This is not just whether the model can write a function. It also includes cross-file refactoring, legacy code compatibility, dependency version constraints, error recovery, and self-testing. The evaluation set should include executable unit tests or compilation checks.

2. Reasoning and symbolic operations: This covers multi-step logic, mathematical calculation, and constraint-satisfaction problems. Accuracy on these tasks is highly sensitive to temperature and sampling count, so parameters should be fixed and multiple samples should be used.

3. Long context and retrieval augmentation: This includes long-document summarization, cross-paragraph information aggregation, and citation accuracy. The focus should be on whether the model only uses the provided context rather than its own memory.

4. Creative writing and emotional understanding: This covers style transfer, tone control, and conceptual discussion. Such tasks are difficult to judge automatically and require human scoring rubrics.

5. Multimodal understanding and generation: This includes image and video understanding, image-text consistency, and the usability of generated results. A separate visual asset library and manual acceptance criteria are needed.

6. Tool calling and agent orchestration: This includes function signature adherence, parameter hallucination rate, failure recovery in multi-turn tasks, and context management strategies.

7. Factuality and timeliness: When facing information that changes over time, the model should acknowledge uncertainty, correctly trigger retrieval, and provide verifiable references.

8. Compliance and safety policies: Refusal boundaries, sensitive content handling, data residency, and log retention requirements often matter more than capability scores when deciding whether a model can go live.

After breaking down the dimensions, the selection discussion shifts from “which model is stronger” to “under my scenario weights, which model has a higher overall score and lower cost per task.”

3. Evaluation set design and regression-friendly scoring

The quality of the evaluation set determines the credibility of conclusions. It is recommended to build a golden set of 200–1,000 samples based on real business distribution. Each sample should include task category, prompt, reference answer or executable assertion, and scoring rubric.

from dataclasses import dataclass, field
@dataclass
class EvalCase:
    task_id: str
    category: str            # code / math / long_context / creative / multimodal / tool_use
    prompt: str
    weight: float = 1.0      # business weight
    check: str | None = None # executable assertion name: unit test, regex, numeric comparison, JSON schema
    rubric: list[str] = field(default_factory=list)  # human scoring rubric

Scoring should combine three methods to avoid systematic bias from any single method:

• Executable assertions: Run unit tests, compare numerical results, and validate structured output with schema checks. This is the most objective method and should be the main approach.
• LLM-as-a-judge: Suitable for open-ended outputs, but it must include a scoring rubric, require evidence citations for each item, and be periodically calibrated against human scores.
• Manual sampling review: Used for high-weight samples and cases where automatic reviewers disagree.

During execution, fix sampling parameters, record complete requests and responses, and calculate pass rates over multiple samples:

def run_eval(cases, client, model, k=3, temperature=0.2):
    results = []
    for c in cases:
        latencies, tokens, passed = [], [], 0
        for _ in range(k):
            t0 = time.perf_counter()
            resp = client.chat(model=model, messages=[{"role": "user", "content": c.prompt}],
                               temperature=temperature)
            latencies.append((time.perf_counter() - t0) * 1000)
            tokens.append(resp.usage.total_tokens)
            passed += 1 if evaluate(c, resp.content) else 0
        results.append({
            "task_id": c.task_id, "category": c.category, "weight": c.weight,
            "pass_rate": passed / k,
            "p95_latency_ms": percentile(latencies, 95),
            "avg_tokens": sum(tokens) / k,
        })
    return aggregate(results)   # aggregate by category and weight

Two engineering points are easily overlooked. The first is silent version changes. Commercial model services continue to iterate, and the behavior of a model with the same name may change. Evaluation must run regressions regularly, and alerts should be raised when the model version identifier returned by the service changes. The second is statistical significance. A 2–3 percentage point difference on a small sample is usually meaningless. Pass rates should be examined by category rather than only focusing on the total score.

4. Include rework and manual correction in total cost

A real cost model can be roughly expressed as:

Total task cost ≈ Σ(input tokens × input price + output tokens × output price) × (1 + retry rate)
                 + number of reworks × cost per rework
                 + manual correction hours × hourly labor cost
                 + platform and operations overhead (storage, queue, logging, monitoring)

Several practical conclusions are robust under this formula:

• First-pass success rate is closer to cost than average score. With the same prompt, one model may pass on the first attempt, while another requires three rounds of rework. The latter may consume 2–3 times the output tokens and incur higher manual intervention costs.

• Long context is not free. Putting an entire document into context may seem convenient, but input token cost grows linearly with length, and information loss in the middle can reduce accuracy. For tasks that can be split, retrieval augmentation is usually more cost-effective.

• Multi-sample strategies need upper limits. Increasing the number of samples can improve pass rates, but costs grow proportionally. This should be enabled only for high-value tasks and combined with automatic checks for early termination.

5. Tiered routing: send different difficulty tasks to different tiers

A more economical approach for enterprises is to build a unified adapter layer and tiered routing. Requests should be assigned to different capability tiers and providers based on difficulty, risk, and latency requirements.

class ModelRouter:
    def __init__(self, registry, providers):
        self.registry = registry          # model capability profile: strong categories, unit price, rate limit quota
        self.providers = providers        # provider clients with a unified abstraction interface
    def route(self, task):
        tier = classify(task)             # lite / standard / flagship
        chain = self.registry.pick(task.category, tier)
        last_err = None
        for model in chain:               # try in ascending cost order and fall back on failure
            try:
                return self.providers[model.provider].invoke(model, task)
            except RateLimited as e:
                last_err = e              # switch to the next candidate if rate-limited
            except TransientError as e:
                last_err = e
        raise AllProvidersFailed(last_err)

Supporting measures include semantic caching or prefix caching for high-frequency repeated requests, asynchronous queues for batch offline tasks instead of synchronous interfaces, smaller model tiers for simple tasks such as pure extraction, classification, and format conversion, and the highest tier only for complex reasoning, cross-file refactoring, and high-value long-form generation.

An abstract adapter layer also has a strategic value: avoiding vendor lock-in. Only by unifying prompt templates, message formats, streaming protocols, tool-calling protocols, and billing definitions in one layer can you migrate at low cost when prices change, rate-limit policies change, or models are retired.

6. Stability, observability, and compliance

In production systems, availability often matters more than peak capability.

• Observability: Record the model version, token usage, latency percentiles, retry counts, and failure reasons for each call. Use logging services and monitoring alerts with thresholds such as sudden P95 latency spikes, rising error rates, or abnormal increases in cost per task.

• Fallback path: If the primary model times out or is rate-limited, fall back to a backup tier and return a usable result. If the system is completely unavailable, return a templated result or route the request to a human queue instead of directly throwing errors to users.

• Context and data management: Store conversation history, evaluation sets, and generated outputs in separate repositories. Desensitize sensitive fields before persisting them. Use object storage for evaluation sets and visual assets, and message queues for asynchronous batch processing to form a replayable pipeline.

• Compliance requirements: The strictness of content safety policies, whether data will be used for training, log retention periods, and data residency should be written into procurement and architecture decisions rather than added after launch. For strongly regulated scenarios, evaluate the feasibility of private deployment or dedicated resource pools in advance.

7. Collaboration pipeline: drafting and refinement by different models

A single model is rarely optimal on all dimensions. A more reliable practice is a division-of-labor pipeline: one model produces the structure and content skeleton, while another model handles language polishing, terminology consistency checks, and style convergence.

Raw requirement → retrieval → Model A: generate draft (focus on facts and structure)
              → terminology/forbidden-item check → Model B: refine (focus on expression and compliance)
              → automatic checks (format, terminology, length) → human spot check → storage

Prompt-level constraints should be explicit rather than relying on the model to behave properly:

• Specify the output structure and field order. Require structured output, such as JSON with a schema, so that programs can validate it.
• Provide a termbase and require consistency throughout the text. Prohibit synonym substitution.
• Explicitly declare forbidden items and style constraints, such as avoiding excessive use of specific punctuation or banning opening statements without informational value.
• Require uncertainty labels and information sources to facilitate downstream verification.
• Set upper limits on length and redundancy to avoid “correct but verbose” outputs that increase manual editing costs.

8. Selection decision checklist and common mistakes

Confirm the following items in order. This can significantly reduce rework:

1. Define scenario weights: how much traffic and business value each of code, reasoning, long-form, creative, and multimodal tasks accounts for.
2. Build a domain evaluation set with executable assertions, run a baseline once, and put it under version management.
3. Record pass rates, P95 latency, token consumption, and rework rates by category.
4. Calculate total cost per task rather than comparing unit prices.
5. Verify the stability and error-code semantics of tool calling, structured output, and streaming responses.
6. Confirm rate-limit quotas, scaling methods, version-change notification mechanisms, and retirement policies.
7. Complete the security and compliance review: data usage, retention, desensitization, and audit.
8. Implement a unified adapter layer and tiered routing, keeping at least two switchable providers.
9. Continuously run regression evaluations after launch, and treat model iterations as routine change events.

There are three common mistakes: drawing conclusions from demo-level samples only; using total scores to hide category-level weaknesses; and equating the lowest price with the lowest total cost. Avoiding these three makes selection conclusions engineering-valuable.

9. Conclusion

Model capabilities are converging rapidly. Real differentiation comes from the engineering maturity of the evaluation system, routing strategy, and cost accounting. When building this system, you can reuse object storage for evaluation sets and outputs, message queues for asynchronous batch processing, and logging and monitoring for observability, while keeping the unified adapter layer as a long-term asset. When the evaluation set becomes a regression baseline, the cost model covers rework and human effort, and routing and fallback paths have been load-tested, model replacement is no longer an architecture risk but a controlled routine upgrade.