LLM Selection in Practice: A Methodology for Evaluating Capability and Balancing Cost

I. Starting With: “Great Benchmarks, Prohibitive Pricing”

Whenever a highly anticipated new model is released, the first wave of discussion in the tech community often revolves around the price list rather than capability boundaries. Complaints like “impressive parameter scale and excellent benchmark scores, but the pricing is a dealbreaker” are common. Behind this debate lies a repeatedly proven engineering principle: Model capability determines the business ceiling, while inference cost determines whether the technology can actually enter the production environment.

In recent years, models released by open-source communities and global R&D teams have continuously narrowed the gap with the top-tier international models on multiple public benchmarks, sparking discussions with every release. While these expectations are valid, the discussion should return to technical facts and engineering data. It is worth noting that, from a generational perspective, the unit price of LLM APIs has generally been trending downward: the input unit price of a new generation’s flagship model is often significantly lower than that of the previous generation’s similarly positioned model. The flourishing open-source ecosystem has also forced the broader market to reduce prices. However, “price reduction” and “inexpensive” are two different things. When a model enters the top tier of capability, its pricing typically aligns with flagship tiers. Thus, asking “is being expensive an original sin” becomes a pseudo-proposition. The real question is: Can this cost be covered by the business value?

II. Capability Evaluation: Treat Benchmarks as Health Check Reports, Not User Manuals

1. The Value and Limitations of Public Benchmarks

Common evaluation sets cover dimensions like knowledge QA, mathematical reasoning, code generation, and multilingual understanding, providing an initial horizontal positioning for new models. However, from an engineering perspective, one must clearly recognize three limitations:

• Data Contamination Risk: Evaluation questions might have been included in the training corpus, leading to potentially inflated scores.
• Saturation Effect: The gap among top-tier models on most benchmarks has narrowed to single digits, reducing differentiation.
• Distribution Shift: The task distribution of public questions often does not align with your actual business scenarios.

Therefore, the correct approach is: use public benchmarks for initial screening, and use a custom evaluation set for the final verdict.

2. Building Your Own Evaluation Set

A minimum viable custom evaluation process consists of five steps:

1. Sample real requests from business logs, covering both high-frequency tasks and long-tail edge cases.
2. Prepare reference answers and scoring rubrics for each sample (accuracy, format compliance, key point coverage, etc.).
3. Combine manual sampling reviews with model-assisted scoring for batch evaluation.
4. Sample the same model multiple times to observe output stability and variance.
5. Record scores alongside token consumption to directly generate raw data across the two dimensions of “capability” and “cost.”

Below is a pseudo-code example (refer to your specific platform’s API documentation for actual interfaces):

import statistics
cases = load_cases("business_eval.jsonl")   # Custom business evaluation set
records = []
for c in cases:
    resp = chat(model=c["model"], messages=c["messages"])
    score = evaluate(resp.text, c["reference"], rubric=c["rubric"])
    records.append({
        "score": score,
        "in": resp.usage.input_tokens,
        "out": resp.usage.output_tokens,
    })
cap  = statistics.mean(r["score"] for r in records)            # Capability score
cost = sum(r["in"] * PIN + r["out"] * POUT for r in records)   # Cost
print(f"Average score {cap:.2f}, total cost for this batch {cost:.2f}")

3. Key Capability Dimensions to Focus On

• Complex Reasoning and Task Planning: Can it stably decomose multi-step problems rather than superficially imitating reasoning?
• Code Capability: Generation, completion, defect repair, and cross-file understanding.
• Long Context: Positioning accuracy in ultra-long document QA and the “lost in the middle” phenomenon.
• Instruction Following and Format Stability: Can it strictly output the agreed structure? This directly determines downstream parsing costs.
• Hallucination Rate: It is better to explicitly say “I don’t know” when uncertain than to fabricate answers with high confidence.

III. Understanding API Pricing: The Cost Structure Behind Unit Prices

1. Differentiated Pricing for Input and Output

The pricing structure of mainstream platforms is almost universally a dual-track system: “input unit price + output unit price.” The output unit price is typically several times that of the input. This means that making the model “say less nonsense” (controlling output length) is often more cost-effective than “providing less background” (compressing input), although both are worth doing.

2. The Hidden Costs of Multi-Turn Conversations

In multi-turn sessions, historical messages are repeatedly counted as input tokens with each request. In a 50-turn customer service conversation, if no context trimming is applied, the input cost accumulates approximately quadratically with the number of turns. Common countermeasures include sliding window summarization, structured storage of key information, and on-demand re-injection.

3. Common Cost Optimization Mechanisms (General Categories)

• Context Caching: When repeated prefixes hit the cache, input costs can be significantly reduced, which is suitable for scenarios with fixed, ultra-long system prompts.
• Batch Inference APIs: Trading offline, non-real-time tasks for discounted unit prices.
• Tiered Pricing: Differentiating prices based on context length or latency tiers, and selecting as needed.

4. Converting Unit Price to “Per-Call Cost”

The unit price itself is not intuitive; converting it into a business perspective is more meaningful. An illustrative calculation is as follows (numbers are for demonstration only; refer to actual platform pricing pages):

Assume the input costs $0.50 per million tokens and output costs $2.00 per million tokens. If a single call averages 3,000 input tokens and 800 output tokens, then:

Cost per call = (3000 × 0.50 + 800 × 2.00) / 1,000,000 = $0.0031

That is, 10,000 calls cost about $31, and 100,000 calls cost about $310. Comparing this figure with the manual labor cost it replaces (one manual agent interaction, one manual review), you immediately have an anchor point for whether it is “expensive.”

IV. A Scorecard: Making Cost-Effectiveness a Calculable Decision

It is recommended to replace intuitive judgment with a weighted scorecard. The weights for each dimension can be adjusted according to your business needs:

Dimension	Suggested Weight	Description
Task Fit	40%	Score on the custom evaluation set
Unit Cost	25%	Cost converted to a single business call
Output Stability	15%	Format adherence rate, variance, refusal rate
Engineering Ecosystem	10%	SDK completeness, observability, deployment, and compliance options
Iteration Pace	10%	Version update frequency, context and multimodal roadmap

Total score = 0.4 × Capability + 0.25 × Cost + 0.15 × Stability + 0.1 × Ecosystem + 0.1 × Iteration

In practice, there is another often-overlooked technique: Tiered Routing. Assigning simple tasks (classification, extraction, rewriting) to lightweight models and invoking flagship models only for complex tasks (reasoning, planning, coding) can often reduce overall costs by an order of magnitude with virtually imperceptible quality loss.

V. From Evaluation to Production: Four Engineering Recommendations

1. Gradual Rollout and A/B Testing: Run new and old models in parallel based on traffic ratios, and use real business metrics (rather than benchmark scores) to determine the switching pace.
2. Unified Gateway Abstraction: Add a unified access layer between the application and the model to shield interface differences across platforms, making switching and price comparison costs approach zero. LLM service platforms like Baidu AI Cloud provide such managed access and scheduling capabilities, which can be prioritized for evaluation.
3. Cost Observability: Establish dashboards for token usage, latency, failure rates, and costs for each business line to keep the “burn rate” visible at all times.
4. Prompt and Cache Governance: Regularly compress system prompts, enable prefix caching, and streamline examples. This typically yields double-digit cost reductions.

VI. Conclusion

“Expensive” has never been an LLM’s original sin; costs that cannot be covered by business value are. The correct sequence for evaluating a new model is: first use public benchmarks for initial screening, then use a custom evaluation set to measure true capabilities, convert token unit prices into per-call business costs, and finally compare it with existing solutions using a weighted scorecard. Benchmarks can generate buzz, but model selection decisions must be built upon the triangle of data, cost, and business value. When you possess a reusable evaluation process, you can provide a rational answer within a day when facing any new model release, rather than being led by public sentiment.