LLM Selection Guide: An Evaluation Method from Requirements to Production
Choosing a large language model is now an engineering problem, not a leaderboard contest. This article explains how to define measurable tasks, evaluate models on quality, context use, cost, latency, compliance, and operational stability, build an in-house eval set, isolate models behind a gateway, and roll out with shadow traffic, canaries, and rollback drills.
Why model selection is an engineering problem
Public leaderboard scores used to be the main way to pick a model. Releases now move quickly, and different sizes from the same vendor can behave very differently across tasks. Benchmarks measure general capability. Production cares about your data, your prompts, and your users. A top-ranked model can lose to a cheaper, faster mid-size model on something as specific as support-ticket classification.
The cost of a bad choice is high. Model spend can dominate the operating cost of an AI feature. Latency past user tolerance hurts retention. Data handling that violates policy can create legal risk. Selection should look like choosing a database or a message queue: decide from explicit requirements, not from brand heat.
Define the task before you talk about models
Start by breaking business needs into measurable tasks. Common types include:
- Open-ended Q&A and content generation: fluency and factual accuracy.
- Structured extraction: stable fixed formats such as JSON fields and strong instruction following.
- Code generation and review: correctness, context understanding, and adherence to existing style.
- Multi-turn dialogue and support: memory, tone control, and refusal boundaries.
- Long-document analysis: effective use of context, not just maximum window size.
- Tool use and agent workflows: accurate function arguments and stable multi-step planning.
For each task, capture input/output characteristics: average input length, average output length, daily request volume, peak concurrency, and acceptable failure rate. Those numbers drive cost estimates and architecture. Without them, claims about “cheap” or “fast” have no basis.
Six evaluation dimensions
1. Quality
Score quality per task. General tests are only a reference; what matters is performance on your data. Split quality into sub-metrics such as factual accuracy, field completeness, reasonable refusal rate, and naturalness of the target language.
2. Context window
Window size is easy to overvalue. Accepting a long prompt is not the same as using every part of it. In evaluation, plant key facts at the beginning, middle, and end of long inputs and check whether recall drops by position.
3. Cost
Billing is usually by input and output tokens, sometimes plus cache hits, batch discounts, and extra fees for multimodal inputs. Include retries and the length of system prompts, which are charged on every request.
A simple formula:
Request cost = input tokens × input price + output tokens × output price
Multiply by daily volume and a retry factor for monthly cost. When comparing models, use measured token counts on the same sample set, not estimates.
4. Latency and throughput
Separate time to first token from time to full response. Streaming cares about first-token latency; batch jobs care about throughput. In production, watch tail latency (for example p99), because a few slow calls can stall the whole user flow.
5. Compliance and deployment
Clarify data residency, whether data may be used for training, log retention, and whether private deployment is supported. For personal data or trade secrets, this dimension often has veto power.
6. Reliability and ecosystem
Reliability covers availability, rate limits, version-change notices, and how long old versions stay online. Ecosystem covers whether existing code can integrate with small changes and whether the team already knows the debugging and monitoring tools.
Build your own evaluation set
The eval set is the core asset. Its quality determines whether conclusions are trustworthy.
Sample source: Sample from real production logs, and over-cover long-tail and edge cases. Idealized examples overestimate quality.
Annotation spec: Write scoring rules per task. For extraction, missing fields, wrong fields, and format errors should be scored separately. Prefer two independent annotators, measure agreement, and have a third person resolve large disagreements.
Scoring: Closed tasks can be scored by rules or scripts. Open generation can combine automatic scoring with human spot checks (often around 10%, depending on budget).
Eval loop (pseudocode):
for case in eval_set:for model in candidates:start = now()output = call_model(model, case.prompt)latency = now() - startscore = score_fn(output, case.reference)record(model, case.id, score, latency, input_tokens, output_tokens)aggregate mean scores, latency percentiles, and unit cost
Report quality, latency, and cost together, and plot them so trade-offs are visible.
Access layer: treat the model as a replaceable component
Do not hard-code the chosen model in business logic. Add a model adapter gateway that:
- Normalizes vendor request/response formats to an internal protocol.
- Selects a model from config by task type, user tier, or budget.
- Logs latency, token usage, and error codes for ongoing monitoring.
- Fails over to a backup model on timeout or error and raises an alert.
Example routing config:
routes:support_intent_classification:primary: small_model_afallback: mid_model_btimeout_ms: 1500long_doc_summary:primary: long_context_model_cfallback: long_context_model_dtimeout_ms: 8000
Upgrades then become config changes, and circuit breaking keeps one upstream from taking down the system.
Launch: canary and rollback
Before serving users, run shadow traffic: call old and new models in parallel, return only the old result, and compare differences.
If the delta is acceptable, canary a small share (for example 5%) and watch quality, latency, and user feedback. Predefine rollback thresholds, such as task success dropping more than two percentage points or p99 latency exceeding a limit.
Practice rollback. Teams often discover stale configs only during an incident. Put rollback steps in the runbook and check them regularly.
Common mistakes
Leaderboards only. They measure general skill; use your eval set.
One model for everything. Large models waste money on simple classification; small models fail on hard reasoning. Layer by task.
Ignoring output length. Output tokens often cost more than input. Unbounded explanations inflate bills. Constrain output format in the prompt.
Ignoring compliance. Finding a data-handling issue after launch is expensive. Review it during selection.
Production checklist
- Business needs are split into tasks with input/output characteristics recorded.
- An in-house eval set covers long-tail cases and has annotation-agreement checks.
- Candidates were compared on quality, latency, cost, and compliance with real measurements.
- Calls go through an adapter; models switch by config.
- Canary ratios, rollback thresholds, and alerts exist, and rollback has been rehearsed.
- Cost bills and eval results are reviewed regularly, and models are re-evaluated after version changes.
Selection is a continuous engineering process. With stable evaluation and a clean access layer, the team can decide quickly whether a new model is worth swapping in and turn each change into a controlled gain.
