LLM Selection Guide: Six Core Dimensions for Model Evaluation

Introduction: How to Choose an LLM Without Common Pitfalls

Large language models are updated frequently, with new releases often changing parameter scale, context length, and inference pricing. Many teams still choose models by checking leaderboards, following hype, or relying on intuition.

To choose an LLM more rationally, teams should first understand the common mistakes in LLM selection and how to run model evaluation scientifically.

Public benchmark scores often differ from real business performance. Published performance numbers may not reflect how a model behaves under your own workload. Common selection mistakes include:

• Focusing only on the price per million tokens while ignoring the real token distribution of your application.
• Directly trusting official peak latency claims without testing under your own conditions.
• Ignoring the impact of false positives from safety filters on business workflows.
• Drawing conclusions from a single test run.

A more reliable approach is to treat model selection as an engineering problem. Build a fixed set of evaluation dimensions, a reproducible test suite, and quantitative decision criteria.

This article breaks the process into six core dimensions: context window, cost structure, inference latency, safety guardrails, output quality, and the evaluation system that connects them together.

Dimension 1: Context Window and Output Length

The context window determines the total amount of information a model can see in a single interaction. It usually includes the input prompt, conversation history, retrieved reference materials, and model output. A larger context window is better suited for long-document understanding, multi-turn memory, and repository-level code analysis.

However, several points are often overlooked:

• Effective context is not the same as nominal context. A very large nominal limit may still suffer from “lost in the middle,” where important information in long inputs is forgotten. This reduces practical usability. Evaluation should include dedicated long-context recall tests.
• Output length is a separate metric. The input context window and maximum output tokens are different concepts. Generating long reports or complete code files requires sufficient output length.
• Longer context usually means higher cost. Most models are billed by token usage, so context consumption directly affects the cost per request.

Long-context test example: Create a long document and insert several target facts, such as key numbers or specified conclusions, at different positions: the beginning, middle, and end. Then ask the model about those facts. Measure the recall rate at each position to quantify the model’s effective context range instead of relying only on the advertised context window.

Dimension 2: Token Pricing and Cost Structure

Token-based billing is common in the industry. Pricing usually distinguishes between input tokens and output tokens, and output tokens are often more expensive than input tokens.

Do not evaluate cost by unit price alone. Estimate cost based on the real token distribution of your business. A directly usable formula is:

Cost per request = (input tokens × input price + output tokens × output price) ÷ 1,000,000

Multiply the cost per request by the expected daily request volume and then by 30 to estimate a realistic monthly cost.

Beyond basic token pricing, consider the following cost factors:

1. Caching mechanisms: Context caching stores repeated long prefixes and can significantly reduce costs for scenarios with fixed system prompts or multi-turn conversations. Pay attention to cache hit rate.
2. Batch discounts and tiered pricing: High-throughput workloads may qualify for batch request discounts or tiered pricing.
3. Free tiers and self-hosting costs: Public cloud services often provide free quotas for experimentation. For private deployment or self-hosting, include GPU, operations, and personnel costs.
4. Hidden costs: Retries, context overflow, and post-processing of model outputs can consume additional tokens.
5. Quotas and concurrency: Account-level rate limits such as RPM/TPM and concurrency limits directly affect whether a service can support the target throughput. Quota constraints outside the advertised price should also be evaluated.

Dimension 3: Inference Speed and Latency

Speed directly affects user experience and system throughput. Two metrics should be distinguished:

• Time to First Token (TTFT): The time from sending a request to receiving the first token. It determines the responsiveness of streaming output.
• Generation speed (tokens per second): The speed at which subsequent tokens are generated. It determines how long long-form outputs take to complete.

Official claims about speed improvements are often based on specific test conditions. Real performance depends heavily on your network, concurrency level, and request length.

A reliable evaluation approach is to run load tests under target workloads and collect P50/P95/P99 latency distributions instead of relying on a single average.

Stress test parameter recommendations: Clearly define and record:

• Concurrency level, such as 1, 10, or 50 concurrent requests.
• Input length, such as 500, 2,000, or 8,000 tokens.
• Output length, such as 200 or 1,000 tokens.
• Repeat count, preferably at least 30 runs to stabilize percentile statistics.
• P95 latency target, for example, no more than 2 seconds for interactive scenarios, with a more relaxed target for offline batch processing.

Fixing these parameters makes latency results comparable across model versions and candidate models.

Dimension 4: Safety Guardrails and False Positive Rate

Safety guardrails are responsible for blocking prohibited content. However, overly strict guardrails can introduce false positives, where normal requests are incorrectly classified as risky content.

For example, mistaking an ordinary biomedical assignment or a cybersecurity learning question for a threat can interrupt legitimate workflows.

Evaluating this dimension should not focus only on false positives. Three metrics should be considered together:

• False positive rate: The proportion of normal requests that are incorrectly blocked in real business data. The calculation is:

  False positive rate = incorrectly blocked normal samples ÷ total normal samples × 100%

• False negative rate: The proportion of violating samples that are not blocked. The calculation is:

  False negative rate = unblocked violating samples ÷ total violating samples × 100%

• Decision criteria and sample size: Define in advance what counts as normal and what counts as violating. Ensure the sample size is sufficient, preferably at least 200 samples per category and covering typical edge cases. Otherwise, the rates may not be statistically meaningful.

Security and business usability need to be balanced. Being overly strict increases false positives, while being too loose increases false negatives. Ideally, guardrail policies should be configurable by business domain, with exception handling where needed.

Dimension 5: Output Quality and Fine-Grained Performance

This dimension best reflects a model’s real-world capability, but it is also the hardest to quantify. It combines language fluency, structure, detail completeness, and instruction following.

To reduce individual bias, use a fixed scoring card and blind review by multiple evaluators.

A reference scoring card is shown below:

Evaluation item	Score	Deduction rule
Instruction following	30	Deduct 10 points for each violation of format, length, role, or constraint
Long-text coherence	25	Deduct 5 points for each contradiction, repetition, or break between paragraphs
Detail restoration	25	Deduct 5 points for each missing key constraint or fact
Natural expression	20	Deduct 5 points for each obvious mechanical tone, grammar issue, or unnatural phrasing

The total score is 100. After multiple reviewers score the samples, take the average. Samples with major disagreements should be reviewed again to further reduce subjective variance.

Dimension 6: A Reproducible Evaluation System

To turn the dimensions above into practice, teams need a reproducible evaluation framework.

The following is illustrative pseudocode for describing the workflow design idea. It is not a real API.

# Evaluation dimensions (the quantifiable parts of the six core dimensions)
dimensions = ["context", "cost", "latency", "safety", "quality"]
# test_cases: fixed evaluation set. Each case includes input_text (input content),
# expected (expected output or annotation), category (scenario category: long-context
# recall, instruction following, boundary safety, etc.), and target_facts (target
# facts in long-context tests).
test_cases = load_benchmark_set("internal_eval_v1")
def evaluate(model_client, case):
    result = {}
    # model_client.price: {"input": input price per million tokens, "output": output price per million tokens}
    result["latency"]  = measure_ttft(model_client, case["input_text"])          # TTFT in milliseconds
    result["cost"]     = estimate_tokens(case) @ model_client.price             # cost per request
    result["context"]  = check_long_context_recall(model_client, case)          # long-context recall rate 0~1
    result["safety"]   = check_false_positive_and_negative(model_client, case)  # false positive / false negative
    result["quality"]  = human_blind_review(model_client, case)                 # scoring card total 0~100
    return result
# percentile_stats(list): returns percentile statistics such as P50/P95/P99
# from multiple samples of a metric
def run_suite(model_client, repeats=30):
    agg = defaultdict(list)
    for _ in range(repeats):
        for case in test_cases:
            for k, v in evaluate(model_client, case).items():
                agg[k].append(v)
    return {k: percentile_stats(v) for k, v in agg.items()}

Field descriptions:

• model_client.price is a dictionary containing two unit prices: input and output.
• test_cases contains the fields described in the comments above.
• percentile_stats accepts a list of multiple samples for one metric and returns percentile statistics instead of a single value.

Weighted total score formula:

Overall score = Σ (dimension score × corresponding weight), with Σ weights = 1.

For example, context could have a weight of 0.15, cost 0.20, latency 0.15, safety 0.15, and quality 0.35. Weights should be adjusted according to business priorities, and the sum of all weights must equal 1.

Key design principles: Use a fixed test set, repeated runs, distribution-based results, and multi-dimensional weighting. This makes it possible to compare results vertically across model versions and horizontally across multiple candidate models.

Compliance notes: Evaluation data should be anonymized or desensitized to avoid privacy leakage. API credentials should follow the principle of least privilege. Log retention should minimize sensitive content. Access controls should be applied to both evaluation and production environments to prevent unauthorized access to test data.

Conclusion

Model selection should not be driven by version numbers or leaderboards alone. A more robust approach is to evaluate models around six core dimensions: context, cost, latency, safety, quality, and the evaluation system.

Use real business data to make decisions. Once the evaluation process becomes a standard team asset, teams can quickly and rationally choose models even as the model ecosystem continues to change.