A Guide to Large Language Model Selection: Metrics, Costs, and Risks

This article provides a reusable selection process: first, define business constraints; next, conduct empirical tests by capability dimension; then, calculate the unit task cost; and finally, establish multi-model routing rules. All thresholds listed are examples and must be determined by each team based on their own business risks and test data.

1. Define Constraints Before Comparing Models

Before evaluating models, write down three constraints: the acceptable response latency for your business, the cost ceiling per task, and whether your data can cross borders or enter third-party training sets. Without clear constraints, benchmark scores cannot be translated into procurement decisions.

Once the constraints are defined, test the models by capability dimension. The table below outlines the testing methods and core metrics for each dimension.

Capability Dimension	Testing Method	Core Metrics	Pass Threshold (Example, team-defined)
Long-context retrieval	Under various lengths (4K, 32K, 128K), place key facts at the beginning, middle, and end, then ask questions. Use at least 50 samples per length.	Retrieval accuracy, Time to First Token (TTFT)	Accuracy at each position >= 90%, middle position accuracy drop compared to the beginning <= 10 percentage points
Instruction following	Use system prompts with nested constraints, call 100 consecutive times.	JSON parsability rate, Field completeness rate	Parsability rate >= 99%
Code generation and repair	Build an internal test set by language (Python, Java, Go, etc.), with unit tests attached to each problem.	Unit test pass rate, Repair success rate, Defects found in manual review	Pass rate meets team baseline, 10% manual spot check, no security-related defects
Multi-step reasoning	Design business problems requiring 3 to 5 steps of derivation, with standard answers provided.	Task success rate, Contradiction rate	Success rate based on business risk; >= 95% recommended for high-risk scenarios
Tool calling	Provide 5 to 10 functionally similar tools, requiring the model to select the correct tool and fill in parameters.	Tool selection accuracy, Parameter error rate, Call failure rate	Parameter error rate < 2%, can correctly retry or report an error upon failure

Test sets must be isolated from public leaderboards and training data; otherwise, the results will be inflated. The exact same test set should be re-run after any model version change.

2. Compare Models by Category

Model names and prices change rapidly, so comparing specific models directly can quickly become outdated. A safer approach is to compare them by category, and then fill in the data using your own test set.

Category	Typical Use Cases	Key Metrics to Measure	Cost Sensitivity
Lightweight & Fast	Text classification, information extraction, simple Q&A	Accuracy, response latency	High; costs dominate with high call volumes
General Flagship	Complex document understanding, comprehensive writing	Long-context retrieval, instruction following	Medium; suitable for low-frequency, high-value tasks
Reasoning Enhanced	Multi-step math problems, complex planning	Multi-step reasoning success rate	High; large output token count and long latency

To determine if a category is suitable, you need to look at its score on your test set relative to its unit cost.

3. Cost Calculation Method

The formula for calculating the cost of a single API call is as follows:

Single Call Cost = (Input Tokens ÷ 1,000,000 × Input Price) + (Output Tokens ÷ 1,000,000 × Output Price)

The formula for monthly cost is:

Monthly Cost = Single Call Cost × Monthly Call Volume × (1 + Retry Rate)

Vendors typically price per million tokens; refer to the official documentation for specific units. Do not estimate token counts based on experience. Different tokenizers handle Chinese and English very differently. It is recommended to use the vendor’s counting API or calculate the average from 100 real requests.

Here is a calculation example. Assume an input price of $2 per million tokens and an output price of $8 per million tokens (hypothetical prices for calculation purposes). If each call uses 2,000 input tokens and 500 output tokens, the single call cost is $0.004 plus $0.004, totaling $0.008. If there are 100,000 calls per month with a 5% retry rate, the monthly cost is 0.008 × 100,000 × 1.05, which equals $840.

Common cost reduction methods include the following, each with specific applicable conditions:

• Prompt caching: Repeated system prompts or long document prefixes can be cached. Whether a cache hit occurs depends on vendor support and whether the prefix is exactly identical. The hit rate must be confirmed during testing.
• Batch API: Suitable for tasks that do not require real-time responses, such as report generation or offline labeling. Vendors usually offer discounts; refer to the official documentation for specific rates.
• Small model routing: Fixed-format tasks like classification and extraction can be delegated to lightweight models. Before switching, you must use a test set to confirm that the accuracy drop is within acceptable limits.
• Limit output length: Constrain output length via the max_tokens parameter and prompts. This is particularly effective for reasoning-enhanced models because they generate a large number of output tokens.
• Distillation: Train a small model using data generated by a large model. This is suitable for stable tasks with sufficient data. The development cycle is long, so you need to evaluate whether the investment is worthwhile.

4. Multi-Model Dynamic Routing

No single model is the best across all dimensions, so many teams use a routing layer to dispatch tasks to different models. The code below is pseudocode and only demonstrates the routing structure. The client is a hypothetical client object and should be replaced with your actual SDK.

# Pseudocode: Only demonstrates routing structure, replace client with actual SDK
import logging
import time
logger = logging.getLogger("model_router")
ROUTES = {
    "simple_qa": {"model": "economy-model", "max_cost_per_call": 0.002},
    "code_gen": {"model": "coding-model", "max_cost_per_call": 0.05},
    "complex_reasoning": {"model": "reasoning-model", "max_cost_per_call": 0.20},
}
FALLBACK_MODEL = "economy-model"
TIMEOUT_SEC = 30
MAX_RETRIES = 2
def route(task_type: str, complexity: int) -> str:
    # Thresholds 6 and 8 should be determined by evaluation results, not set by guesswork
    if task_type == "code_gen" and complexity >= 6:
        return ROUTES["code_gen"]["model"]
    if task_type == "complex_reasoning" or complexity >= 8:
        return ROUTES["complex_reasoning"]["model"]
    return ROUTES.get(task_type, ROUTES["simple_qa"])["model"]
def call_with_fallback(task_type: str, complexity: int, prompt: str, client):
    model = route(task_type, complexity)
    policy = ROUTES.get(task_type, ROUTES["simple_qa"])
    # Downgrade to economy model if estimated cost exceeds budget
    estimated_cost = client.estimate_cost(model, prompt)
    if estimated_cost > policy["max_cost_per_call"]:
        model = FALLBACK_MODEL
    for attempt in range(MAX_RETRIES + 1):
        start = time.time()
        try:
            resp = client.call(model, prompt, timeout=TIMEOUT_SEC)
            # Log only metadata like model name and latency, do not log raw prompts
            logger.info(
                "model=%s task=%s attempt=%d latency=%.2fs",
                model, task_type, attempt, time.time() - start,
            )