Selecting a Large Language Model: Define Metrics, Estimate Costs, and Roll Out Gradually
This guide provides a practical framework for choosing a large language model (LLM) for production use. It covers how to measure key indicators like time-to-first-token and throughput under concurrency, how to build a custom evaluation set for your specific use case, and how to estimate monthly costs using a simple template. It then outlines acceptance checklists for typical scenarios such as content generation, structured output, and code assistance. Finally, it explains cloud-native deployment and monitoring best practices, a weighted decision matrix, and a gradual rollout strategy with clear kill criteria. All templates and thresholds can be directly adapted to your own evaluation process.
1. Standardize Evaluation Criteria Before Selection
Before comparing models, it is critical to use consistent measurement conditions. The same model can show vastly different numbers under different testing setups, and relying on vendor-reported figures can lead to incorrect conclusions. Below are four areas that form a repeatable evaluation framework: metric measurement, typical scenarios, deployment and monitoring, and decision execution.
1.1 Time to First Token (TTFT)
TTFT is the interval from when the client sends a request to when the first output token is received. It directly drives the perceived responsiveness of a conversational or real-time assistant.
Key measurement guidelines:
- Record timestamps on the client side, not just in server logs. Client-side values include network round-trip and queueing time, which better represent the user experience.
- Report P50, P95, and P99 percentiles, not just averages. Averages hide long-tail requests, which are often the source of user complaints.
- Group by input length. At minimum, test with 1K, 8K, and 32K token inputs, because longer inputs increase prefill latency.
- Test under different concurrency levels. Use 1, 8, 32, and 64 concurrent requests to observe how latency degrades as load rises.
1.2 Throughput and Concurrency Degradation
Output throughput is calculated as:
Output throughput (tokens/sec) = Total output tokens / Generation time
A single-request throughput figure is easy to make look good, but real traffic involves many simultaneous requests. Therefore, also compute the concurrency degradation rate:
Concurrency degradation rate = 1 − (Per-request throughput at high concurrency / Per-request throughput at single concurrency)
A lower degradation rate means the model remains stable as load increases. The acceptable threshold should be set by your business. For example, if peak-hour experience may not degrade by more than 30%, then any candidate model with a degradation rate above 0.3 should be disqualified.
1.3 Building a Test Set for Comprehension and Reasoning
Generic benchmark scores often fail to represent your specific business needs. A more reliable approach is to build a custom test set with fixed scoring rules. A practical structure includes:
- 30 multi-turn dialogue items — each with 3 to 5 rounds of context, testing the model’s ability to retain earlier constraints.
- 40 implicit reasoning items — the conclusion is not directly stated; the model must derive it from conditions, such as determining whether a reimbursement request complies with a given policy.
- 50 domain knowledge questions — sourced from business documents; answers must be traceable to a specific section of those documents.
Scoring guidelines:
- Score each item from 1 to 5, and additionally record whether the answer is supported by evidence as a separate flag.
- Two independent raters score each response. Calculate the agreement rate between raters. If agreement falls below a predefined level, align on scoring criteria before continuing.
To measure the hallucination rate, randomly sample responses, check each sentence against the source material, and calculate the proportion of statements without supporting evidence. Compare this proportion against your business tolerance.
1.4 Cost Estimation Template
For usage-based pricing, the monthly cost can be estimated with:
Monthly cost = 30 × (daily requests × avg input tokens / 1000 × input price per 1K tokens + daily requests × avg output tokens / 1000 × output price per 1K tokens)
Example calculation:
Assume 20,000 requests per day, average input 2,000 tokens, average output 500 tokens. Input price is 0.002 CNY per 1K tokens, output price is 0.006 CNY per 1K tokens (prices are illustrative only; refer to your provider’s current rates).
- Daily input cost = 20,000 × 2 × 0.002 = 80 CNY
- Daily output cost = 20,000 × 0.5 × 0.006 = 60 CNY
- Monthly cost = 30 × (80 + 60) = 4,200 CNY
When deciding between a subscription and usage-based billing, consider usage stability. If monthly usage is consistently below half of the subscription quota with low volatility, subscription is often cheaper. If usage fluctuates widely or input/output ratios vary significantly, usage-based pricing with volume discounts may be more suitable. Always calculate both options using the formula above; do not rely on intuition.
2. Capability Requirements and Acceptance Checklists for Common Scenarios
2.1 Content Generation and Document Summarization
This scenario focuses on generation quality and long-text handling, with relatively low demands on complex reasoning.
- Input: Original documents. If a document exceeds 70% of the context window, split it into segments, summarize each segment, then merge the segment summaries into a final summary.
- Prompt structure: Include four parts: role definition, task description, output format, word limit, and prohibited content. For example, explicitly ask the model to avoid introducing numbers or names that are not present in the original text.
- Output validation: Check whether the length is within bounds; verify that key entities are retained; scan the output for numbers that did not appear in the source.
- Failure fallback: If validation fails, automatically regenerate once. If it fails again, route to manual handling.
- Acceptance metrics: Use 20 sample documents and compute the key-information coverage rate and the number of factual errors. The acceptable thresholds must be defined by the business owners before go-live.
2.2 Automated Tasks and Structured Output
In RPA or agent workflows, the model often serves as a component for intent recognition and field extraction. Stable structured output is critical.
- Input: Unstructured text such as tickets, emails, or chat logs.
- Prompt structure: List each field with its name, type, and meaning, and provide a JSON Schema. For enumeration fields, specify all allowed values.
- Output validation: Confirm that the JSON parses; check that required fields are present; verify that enum values are valid.
- Failure fallback: Retry parsing once on failure. If it still fails, send to a human queue and log the original input for later analysis.
- Acceptance metrics: Field accuracy rate and parse success rate. For example, your business may require field accuracy ≥95% and parse success ≥99%. The exact numbers should be based on the risk level of the automated operation.
2.3 Code Generation and Developer Assistance
This scenario has the highest demands on reasoning and context understanding. The model must read relevant files to produce usable modification suggestions.
- Input: Relevant code snippets, interface definitions, error logs, and the project’s language version and dependencies.
- Prompt structure: Clearly state the scope of changes, files that must not be modified, and code style requirements. The clearer the scope, the easier the output is to review.
- Output validation: Code must compile, unit tests must pass, and static analysis should not introduce new high-severity warnings.
- Failure fallback: Limit the number of files the model can modify. Require the model to also produce a change description, making manual review easier.
- Acceptance metrics: Unit test pass rate, developer acceptance rate, and correctness of cross-file references.
3. Cloud-Native Deployment and Observability
3.1 Containerized Deployment and Elastic Scaling
Whether you self-host an open-source model or call a cloud API, you can manage the service via a container platform. Key considerations:
- When self-hosting, declare GPU resources for inference containers in Kubernetes, including
requestsandlimits, to avoid memory contention between services. - Horizontal Pod Autoscaler (HPA) should not rely only on CPU utilization. Inference bottlenecks are usually GPU and memory; CPU often does not reflect the real load. Use custom metrics based on request queue length or concurrent requests instead.
- Model loading can take several minutes. During scaling, new instances need to wait until the model is loaded before receiving traffic. Therefore, keep a minimum number of replicas (
minReplicas) and configure readiness probes for new instances. - Client-side timeout and retry policies are mandatory. Implement exponential backoff and cap the maximum retry count; otherwise, retries during an outage can amplify traffic and worsen the load. For non-idempotent requests (e.g., those that trigger external side effects), confirm whether the operation was already executed before retrying.
For cloud API calls, also configure rate limiting, timeouts, and circuit breakers.
3.2 Monitoring Metrics and Alert Thresholds
The thresholds below are starting points; adjust them based on your business baseline and historical data.
- P95 time-to-first-token: Alert if it exceeds 1.5× the target value for 5 consecutive minutes.
- Error rate (5xx + timeout share): Alert if it exceeds 1%.
- Token consumption: Alert if the daily usage surpasses 80% of the budget; enforce rate limiting when it hits 100%.
- Queue length: Alert if it persistently exceeds 2× the concurrency limit; this usually indicates a need to scale out or throttle.
- Output quality sampling failure rate: Alert if it exceeds 5%, triggering human review.
All metrics should be segmented by model version and business line to quickly isolate the source of problems.
4. Decision Matrix and Gradual Rollout
4.1 Weighted Matrix Template
Below is an example matrix. Weights must be set by the business team according to actual priorities.
| Criterion | Weight | Scoring Method |
|---|---|---|
| Accuracy & task completion rate | 40% | Blind test set score |
| P95 time-to-first-token | 20% | Stress test results |
| Unit cost | 25% | Cost estimation template |
| Stability (error rate & degradation rate) | 15% | Stress test results + historical data |
To compute the total score, normalize each criterion to 0–100 based on the best performer among the candidate models, then multiply by the weight and sum the results.
4.2 Blind Testing Process
- Prepare at least 100 real business samples per scenario, and anonymize them.
- Remove any model identifiers (names, signatures) from outputs so raters cannot infer the source.
- Two raters score each output independently. If there is significant disagreement, a third rater arbitrates.
- Set a pass mark for each scenario. Models scoring below that mark are eliminated.
4.3 Gradual Rollout and Kill Criteria
- Start with 5% of traffic. Observe each traffic level for at least 24 hours, ensuring sufficient sample size.
- Define kill criteria. For example: error rate exceeds 2× baseline, user negative feedback increases noticeably, or unit cost exceeds budget by 20%. If any condition is met, roll back to the original model.
- Review biweekly. Record metric changes, rollback events, and reasons, and adjust weights and thresholds accordingly.
Conclusion
Model selection is not a one-time task. New model versions, price adjustments, and changing business volumes can all invalidate previous conclusions. We recommend putting the templates above into practice and following three key steps:
- Use the test set template from Section 1 to run a small blind test on two or three candidate models.
- Apply the cost formula from Section 1 to forecast the next three months of usage and confirm the billing method.
- After selecting a model, roll out gradually and review metrics every two weeks, adjusting the model or parameters based on the data.
