Large Language Model Selection: How to Evaluate Capability and Cost-Performance

1. Selection pitfalls hidden by specification sheets

In practical engineering selection, many teams directly compare “parameter scale” or “benchmark rankings,” but both approaches can distort the decision. Parameter scale is not directly equivalent to model capability, while leaderboard scores are often detached from real business scenarios. A high score on a general benchmark does not necessarily translate into usability in a specific product. Conversely, a model that does not rank at the top on an aggregate leaderboard may perform more reliably on a particular vertical task.

A rational selection process should therefore start from the relative relationship between capability and cost, rather than focusing on any single metric in isolation.

2. Two cost-performance views: plotting the capability-cost quadrant

When evaluating models, teams often use a scatter plot with “capability” on the horizontal axis and “unit cost” on the vertical axis to locate candidate models. Two common perspectives are built around this chart:

• Traditional view (upper-left to lower-right): focus on dominated models.
  Models that fall on the inferior side of a reference line are both weaker and more expensive than a comparable reference point. From a pure cost-performance perspective, they have little reason to be retained and should usually be eliminated first.

• Reverse view (upper-right to lower-left): focus on top-end dominance.
  When a flagship model is both the most expensive and the strongest option, it can suppress almost all competitors on the capability dimension. However, it largely ignores cost and is mainly suited to scenarios where output quality is extremely sensitive.

These two views are not opposed; they represent different tradeoffs within the same coordinate system. The real value lies in placing your business into this chart. Budget-sensitive, high-volume tasks should prioritize cost-effective models in the lower-left region. Scenarios with extreme requirements for accuracy or complex reasoning, and with controllable usage volume, may justify paying for the strongest model in the upper-right region.

3. Four core dimensions for evaluating a model

To apply the capability-cost quadrant in practice, “capability” and “cost” need to be broken down into measurable dimensions:

1. Context window length
   A long context window, such as one million tokens, determines how much material a model can take in at once. It directly affects capabilities such as long-document understanding, whole-repository code analysis, and multi-turn conversation memory. When evaluating this dimension, teams should focus on the effective context length, not just the advertised maximum. Many models experience information-retrieval degradation under very long contexts.

2. Maximum output length
   The maximum number of output tokens limits the size of a single generation. This is especially important for tasks that require long-form generation or large structured outputs, such as complete reports or batch code generation. Maximum output length and context length are independent metrics and are often confused with each other.

3. Generation speed and throughput
   Speed is usually measured by “output tokens per second” and “first-token latency.” Interactive applications are sensitive to first-token latency, while batch offline tasks care more about overall throughput. Speed directly affects user experience and the cost per unit of time.

4. Billing model
   Mainstream APIs commonly charge by token, and input pricing is often asymmetric with output pricing. In most cases, output tokens are significantly more expensive than input tokens. When estimating cost, teams should not look at unit price alone. They should consider the input-output ratio of the task, retry rates, and context cache hit rates. Price reductions, batch discounts, and caching mechanisms can also materially affect total cost of ownership.

4. Reading benchmarks: general-purpose models vs. specialized models

Leaderboards are useful references, but they must be interpreted carefully, especially by distinguishing between two types of models:

• General-purpose large models: Designed for a wide range of tasks and balanced capabilities. Their leaderboard scores reflect overall capability.
• Specialized or vertical models: Optimized for a particular task or a specific evaluation set. They can achieve very high scores on that individual benchmark.

A common mistake is comparing the results of a general-purpose model with the highest score achieved by a specialized model on a specific leaderboard. This comparison is not fair, because the advantage of a specialized model comes precisely from excelling in one narrow area.

When reading leaderboards, teams should focus on three things:

1. Whether the evaluation task is similar to their real scenario.
2. Whether the evaluation method is disclosed, and whether there are signs of data contamination or overfitting.
3. Reproducibility and stability beyond the score itself.

In short, leaderboards should be used to narrow the candidate pool, not to make a final purchase decision. The decisive validation should be performed on your own business data.

5. The validation dilemma for real tasks

Individual developers often face a common difficulty during evaluation: the available tasks are not difficult enough to challenge model capabilities. Most samples are simulated, virtual, or pure-text scenarios, making it difficult to cover tasks that involve interaction with the physical world, such as hardware design or real robot control. The scale of experimentation is also much smaller than that of vendor or third-party evaluations.

This leads to two recommendations:

• Build a private evaluation set that closely reflects the business.
  Accumulate real API call samples into a repeatable evaluation set. Even if it is small, it can better reflect actual performance than a general leaderboard.

• Pay attention to verifiability.
  For capabilities that cannot be validated through a closed-loop process in a real environment, such as physical interaction, be cautious about accepting marketing claims. Instead, use controlled proxy metrics for relative evaluation.

6. Selection checklist for developers and enterprises

Based on the above, the selection process can be advanced using the following checklist:

1. Define scenario constraints: latency requirements, context length needs, input-output ratio, and call volume.
2. Plot the capability-cost quadrant: position candidate models by capability and real cost, then eliminate dominated models first.
3. Cross-check multiple dimensions: evaluate context length, maximum output length, speed, and billing model together to avoid decisions based on a single metric.
4. Interpret leaderboards carefully: distinguish general-purpose models from specialized models, and verify evaluation methods and reproducibility.
5. Validate with a private evaluation set: use business-relevant samples for A/B comparison, and pay attention to stability and failure modes.
6. Leave room for cost flexibility: monitor batch processing, caching, and other cost-reduction mechanisms, and reassess as the business grows.

On the platform side, this evaluation process can usually be supported by object storage for accumulating evaluation data, message queues for batch inference scheduling, container platforms for elastic scaling, and monitoring, alerting, and logging services for observability of quality and cost. In this way, model selection can become a sustainable operational process rather than a one-time decision. Generic cloud capabilities from providers such as Baidu AI Cloud can serve as infrastructure support for implementing this evaluation workflow.