LLM Selection in Practice: Evaluation, Inference, and Deployment Decision-Making

1. When Leaderboards Make Selection Harder

Over the past two years, large language models have been iterating at an accelerating pace, with domestic and international models constantly leapfrogging each other. Community leaderboards covering general Q&A, code generation, and web development now refresh their rankings almost every month. Each time a new model climbs to the top, technical communities debate whether it is time to migrate.

For developers, this constant movement creates real decision-making difficulty:

• Leaderboard rankings do not necessarily align with real-world business performance.
• Models differ significantly in API specifications, context length, and cost structure, so migration carries real costs.
• New models are released so quickly that today’s best choice may be outdated in three months.

The real question is not “which model is the strongest,” but “which model performs best overall in my scenario.” Answering that requires a systematic evaluation methodology rather than trend-chasing trial and error.

2. Using Leaderboards Correctly: A Screening Tool, Not a Decision-Making Basis

Leaderboards have value, but only if you understand their limitations:

1. Distribution mismatch. Leaderboard questions resemble an exam syllabus, while real business problems are far more varied. The question distribution on a leaderboard often differs significantly from your actual request distribution.
2. Data contamination risk. Public evaluation sets can leak into training data, inflating scores.
3. Single capability dimension. A leaderboard usually measures only one slice of capability. A high coding score does not guarantee reliable long-document extraction.
4. Prompt sensitivity. The same model can fluctuate by several percentage points under different prompt templates. The templates used by leaderboards may not match how you actually use the model.

A practical approach is to place leaderboards inside a three-stage funnel: leaderboard screening to narrow down to three to five candidates, in-house evaluation set testing for quantitative comparison, and small-traffic canary deployment for real-world validation. Leaderboards narrow the field; decisions should always come from your own data and traffic.

3. Building an In-House Evaluation Suite: The Core Engineering Work

3.1 Define Your Task Profile First

Different tasks place completely different weight on model capabilities:

• Conversation and writing: fluency, style control, factual accuracy.
• Retrieval-augmented generation (RAG): adherence to provided materials, resistance to irrelevant content, long-document information extraction.
• Code generation: syntactic correctness, unit test pass rate, multi-turn modification ability.
• Agents and tool calling: function-calling format stability, task decomposition quality, error recovery ability.
• Multimodal: image-text understanding precision, OCR-class task performance.

Define your task profile clearly before deciding what belongs in the evaluation set.

3.2 Where Evaluation Data Comes From

The most reliable data source is your own business logs. Sample real requests covering both high-frequency and long-tail scenarios, apply data desensitization, and have people familiar with the business annotate gold-standard answers or scoring criteria. In terms of scale, a few hundred carefully annotated samples per scenario often provide more discrimination than tens of thousands of uncleaned examples. You can supplement with a small number of public benchmark questions to observe the model’s baseline capabilities, but business samples should dominate.

3.3 Choosing Metrics

• Extraction or classification tasks: accuracy and F1, with rule-based automatic scoring.
• Generation tasks: score against a rubric covering correctness, completeness, and format compliance, or use a stronger model as an automatic judge. Note that judge models have their own biases and require periodic human spot checks for calibration.
• Coding tasks: run unit tests in a sandbox, with pass rate as the primary metric.
• Agent tasks: end-to-end task completion rate, supplemented by average step count and invocation cost.

3.4 Engineering and Cost Dimensions

Beyond capability scores, these indicators also affect deployment decisions:

• Time to first token and tokens per second, which directly determine user experience.
• Throughput and error rate under peak concurrency.
• Effective context length. In long-document scenarios, the practically usable length is often smaller than the advertised value.
• Cost structure: token pricing, discounts for cache hits and batch processing. For self-hosted deployment, factor in GPU depreciation, utilization, and operational labor.

4. API or Self-Hosted: Weighing Inference Paths

After identifying candidate models, the next step is deciding how to run them.

Path one: managed inference services. Major cloud providers generally offer managed LLM inference, charged by usage, with no need to maintain GPU clusters. The advantages are fast onboarding and good elasticity, making this path suitable for prototyping and moderate-traffic workloads. Complete a compliance assessment for data egress before using such services.

Path two: private or dedicated inference. When data cannot leave your environment, or when traffic is large enough that usage-based pricing becomes uneconomical, self-hosting makes sense. In recent years, the industry has widely adopted an “inference microservice” approach: model loading, weight sharding, KV cache management, and batch scheduling are encapsulated as a standardized service delivered in containers, with a single image running across environments and exposing an API on startup.

In self-hosted scenarios, several common optimization techniques are worth knowing:

• Quantization: compressing weights from FP16 to INT8 or INT4, trading a small amount of precision for memory and throughput gains.
• Continuous batching and paged KV cache management: significantly improving GPU utilization.
• Prefix caching: eliminating large amounts of redundant computation for multi-turn conversations and fixed system prompts.
• Speculative decoding: using a small model for drafting and a large model for verification to reduce generation latency.

A simplified decision path: if data is sensitive, prioritize private deployment. If traffic is small or the product is still in validation, use a managed API. If traffic is large and stable, evaluate the ROI of self-hosting. For hybrid scenarios, keep sensitive workloads private and route generic workloads through an API behind a unified gateway.

5. A Minimal Evaluation Workflow: Example Code

Below is a vendor-neutral evaluation skeleton. A unified client abstraction hides API differences, allowing the same test set to run across all candidate models and output a score and latency comparison.

import time
from dataclasses import dataclass
from typing import Callable
@dataclass
class TestCase:
    task_id: str
    prompt: str
    reference: str
    check: Callable[[str, str], float]  # scoring function: returns 0 to 1
class ModelClient:
    """Unified call abstraction: hides differences across inference services"""
    def __init__(self, name: str, invoke: Callable[[str], str]):
        self.name = name
        self._invoke = invoke
    def chat(self, prompt: str) -> str:
        return self._invoke(prompt)
def evaluate(client: ModelClient, cases: list[TestCase]) -> dict:
    total, score_sum, latency_sum = 0, 0.0, 0.0
    for case in cases:
        start = time.time()
        output = client.chat(case.prompt)
        latency_sum += time.time() - start
        score_sum += case.check(output, case.reference)
        total += 1
    return {
        "model": client.name,
        "score": round(score_sum / total, 4),
        "avg_latency_s": round(latency_sum / total, 3),
    }
# invoke_x should be implemented according to each service's official docs; omitted here
clients = [ModelClient("model_a", invoke_a),
           ModelClient("model_b", invoke_b),
           ModelClient("model_c", invoke_c)]
for c in clients:
    print(evaluate(c, test_cases))

Replace the scoring function according to task type: extraction tasks can use normalized exact matching, coding tasks can run sandboxed unit tests, and generation tasks can feed a rubric into a judge model. After running, you will get a score and latency comparison table across candidate models on your own business test set. That table is the real basis for a selection decision.

6. Launch Is Not the End: Turning Selection into a Continuous Loop

Model selection is not a one-time exam; it is an ongoing operational loop:

1. Canary and A/B testing: route 5-10% of traffic to a new model and compare business metrics such as manual feedback rate and regression rate.
2. Online monitoring: continuously track latency percentiles, error rates, and content safety interception rates.
3. Regression testing: every time a model version is changed or upgraded, run the in-house evaluation suite first to avoid a situation where the overall score improves but your core scenario degrades.
4. Model routing: split by task type. Use a smaller model for simple tasks to control cost and a larger model for complex tasks to maintain quality, with a gateway handling dispatch.

7. Conclusion

A steady stream of new models is a good thing, but for engineering teams, the correct approach to selection remains the same: use leaderboards to narrow the field, use your own data for quantitative comparison, use canary traffic for validation, and use monitoring and regression testing to protect quality. Turn this process into team infrastructure, and no matter when the next generation of models arrives, you will be able to produce a data-driven answer to “should we switch, and to what” within a week. That is the key to staying in control in an era of rapid LLM iteration.