A Practical Guide to Selecting Chinese LLMs: Capability Evaluation and Cost Modeling
This article explains why installing more AI apps or relying only on public leaderboards is not enough when selecting a large language model. It presents a practical selection framework covering four layers: chat applications, model APIs, coding assistants, and subscription plans. It then shows how to build a small, business-aligned evaluation set, score candidates with weighted and reproducible metrics, test long-context behavior, estimate total API cost including cache hits, retries, and tiered pricing, and evaluate coding assistants using repeatable delivery metrics. It also covers hidden subscription costs, routing, caching, observability, compliance, and a staged rollout roadmap.
1. Why Installing More AI Apps Does Not Solve the Selection Problem
A common pattern in industry observations is that individual users install multiple AI chat applications, yet their usage usually remains at a shallow Q&A level. In enterprise settings, the problem becomes more serious: purchasing decisions are often driven by a single demo, a public leaderboard, or an ad hoc comparison test, without a unified measurement framework.
The root cause is not a lack of tools. It is a lack of an evaluation coordinate system. Different products are complementary rather than homogeneous, and they need to be assessed on different dimensions.
| Layer | Main concern | Common mistake |
|---|---|---|
| Chat applications | Interaction smoothness, voice experience, and multi-turn usability | Treating a good user experience as proof that the model is strong |
| Model APIs | Cost per token, context length, and structured output stability | Comparing only input price while ignoring output and retry costs |
| Coding assistants | Completion acceptance rate, cross-file context, and first-byte latency | Judging by demo quality rather than performance at repository scale |
| Subscription plans | Seats, quota periods, overage pricing, and rate limits | Ignoring idle quota and non-carryover rules |
Model selection should be the result of evaluating all four layers together, not simply choosing the product with the highest overall score.
2. How to Use Public Leaderboards, and Why They Are Not Enough
Leaderboards for code generation, general knowledge, and blind tests are useful for coarse filtering. However, three limitations should be kept in mind:
- Benchmark contamination risk. Public test sets may remain online for a long time, and their content may already be covered by training corpora. Scores can therefore be inflated.
- Task distribution shift. Leaderboard questions are often single-turn, short-input, and have one clear answer. Real business tasks are more likely to involve long context, multiple constraints, domain terminology, and private format requirements.
- Single-dimension metrics. Leaderboards usually do not reflect throughput, time to first token, rate-limiting behavior, or compliance with structured output requirements such as JSON or Schema. These factors directly determine production usability.
In practice, leaderboards can help narrow the candidate pool from more than twenty options to five or six. The final decision should rely on a self-built evaluation set.
3. Building a Self-Hosted Evaluation Set: The Most Cost-Effective Reliable Method
3.1 Construction principles
Samples should be drawn from real traffic, not from leaderboard questions. A practical size is 150 to 300 samples, weighted according to business share:
- High-frequency simple tasks, such as intent classification, field extraction, and summarization: 50% of the set. Metrics should focus on accuracy and latency.
- Medium-reasoning tasks, such as constrained rewriting, rule application, and table-to-code conversion: 30% of the set. Metrics should focus on constraint satisfaction rate.
- Difficult tasks, such as cross-document reasoning and long-horizon planning: 20% of the set. Metrics should focus on one-shot success rate.
Each sample should have a fixed input, a fixed set of expected output points, and human scoring anchors. These anchors are used to calibrate automated evaluation.
3.2 Scoring and weighting
A weighted and reproducible total score avoids relying on subjective impressions:
S = Σ w_i · s_i, with Σ w_i = 1 and s_i between 0 and 5.
The cost dimension should be reverse-normalized so that lower cost produces a higher score:
s_cost = 5 × C_min / C_candidate
The following example uses only the Python standard library and does not depend on any vendor SDK:
from dataclasses import dataclass@dataclassclass Dimension:name: strweight: float # Weight; the total should be 1.0score: float # 0-5 score obtained from the evaluation setdef weighted_total(dims, cost_ratio: float) -> float:"""cost_ratio = candidate cost / lowest candidate cost (must be >= 1)."""total, wsum = 0.0, 0.0for d in dims:s = 5.0 / cost_ratio if d.name == "cost" else d.scoretotal += d.weight * swsum += d.weightreturn round(total / wsum, 3)dims = [Dimension("task accuracy", 0.30, 4.1),Dimension("latency and throughput", 0.20, 3.8),Dimension("long-context effectiveness", 0.15, 4.0),Dimension("structured output stability", 0.15, 4.3),Dimension("cost", 0.10, 0.0), # Calculated from cost_ratioDimension("operations and integration cost", 0.10, 3.5),]print(weighted_total(dims, cost_ratio=1.4)) # Prints the composite score of the candidate
Weights should be agreed upon by the business and written into documentation. The same evaluation data should not produce the same weighting for every scenario. A customer service scenario and a software engineering scenario should generally use different weights. Otherwise, the scoring formula merely quantifies bias.
4. Long Context: Window Size Does Not Equal Effective Understanding
A advertised million-token context window is a capacity limit. In engineering practice, what matters more is effective context: the proportion of information that can still be reliably recalled at a given input length.
4.1 Three practical tests
- Middle-section retrieval decay. Place key constraints at the beginning, middle, and end of a very long input, then measure recall for each position. A noticeable drop in the middle is common. Do not assume that the model reads all content equally.
- Multiple constraints at once. Provide 8 to 12 constraints simultaneously and count how many are missed. Accuracy often decreases nonlinearly as the number of constraints increases.
- Instruction following under long inputs. Check whether the output format drifts or whether the model starts summarizing instead of executing the requested task.
4.2 Context budget allocation
Do not fill the entire window. A practical approach is to divide the available window into a fixed budget:
def build_context(system, retrieved, history, user_input, window, reserve=0.25):"""Assemble context by budget and keep output headroom to avoid overflow and middle dilution."""budget = int(window * (1 - reserve)) # Reserve 25% for output and safetyused = len(system) + len(history) + len(user_input)allowed = max(0, budget - used)if allowed <= 0: # Input alone exceeds the budgethistory = compress_history(history, keep_ratio=0.4)allowed = max(0, budget - len(system) - len(history) - len(user_input))# Truncate retrieved snippets by rank, not by original orderpicked = truncate_by_rank(retrieved, allowed)return f"{system}\n\n# Reference snippets\n{picked}\n\n# History\n{history}\n\n# Current question\n{user_input}"
A practical engineering rule is to use retrieval and compression to improve effective information density. Reserve very long context for the small number of requests that truly require cross-document reasoning, rather than for all requests.
5. Cost Estimation: Calculate to the Decimal Place
An API bill is determined by at least four variables. Looking only at unit price will usually lead to a wrong judgment.
- Input and output prices are different. Output tokens are usually more expensive, and reasoning tasks often produce long outputs.
- Context cache hits can reduce input cost. Cached portions can be significantly cheaper, but this requires stable prefixes.
- Tiered pricing may apply. Very long requests may fall into a higher price band.
- Failures and retries create hidden costs. Timeouts, rate limits, and format validation failures can lead to duplicate billing.
A structured cost model is better than mental estimation:
def estimate_cost(in_tok, out_tok, p_in, p_out,cache_hit=0.0, retry_rate=0.0, price_cache=None):"""Calculate per-million-token pricing. Returns per-call and monthly estimates. Example values are placeholders only."""price_cache = p_in * 0.25 if price_cache is None else price_cachecached = in_tok * cache_hitbillable_in = (in_tok - cached) * p_in + cached * price_cacheper_call = (billable_in + out_tok * p_out) / 1_000_000return {"per_call": round(per_call, 6),"with_retry": round(per_call * (1 + retry_rate), 6),"monthly_10w_calls": round(per_call * (1 + retry_rate) * 100_000, 2),}
It is also important to account for costs beyond tokens: vector retrieval and storage, logging and monitoring, gateway and rate limiting, and engineering labor. For small and medium traffic scenarios, labor and operations costs can exceed token costs. This is often the real dividing line between self-hosted and managed deployments.
6. Coding Assistants: Replace Subjective Feelings with Reproducible Metrics
Coding assistants should be evaluated by their deliverables, not by demos.
- Completion acceptance rate. This is the number of accepted suggestions divided by the number of suggestions shown. A common range is 20% to 40%. A rate below the expected threshold may indicate that the context or language support does not match the repository.
- First-byte latency. Report P50 and P95. This directly affects typing experience and is often more important than average latency.
- One-pass success rate. This is the proportion of generated code that passes unit tests on the first attempt. It is a stronger signal than whether the code merely looks correct.
- Cross-file and repository-level understanding. The assistant should correctly reference existing symbols, naming conventions, and dependency versions within the project.
- Multi-turn repair cost. One successful fix is much cheaper than three rounds of back-and-forth. Excessive repair cycles can consume the time saved by faster typing.
A simple evaluation method is to extract 30 real historical tasks, have each candidate complete them under the same repository context, run the existing CI unit tests, and record the pass rate and total number of interaction rounds.
7. Hidden Cost Checklist for Subscription Plans
Subscription plans are harder to compare than usage-based pricing because key parameters are often hidden in the details. Before signing, confirm the following:
- Seats and idle capacity. Is access assigned per person or shared through a pool? Are seat fees still charged when concurrency is low?
- Quota reset period. Does the quota reset monthly, or can unused capacity be carried over? A reset rule creates an implicit use-it-or-lose-it cost.
- Overage pricing. How much higher is the overage price compared with the standard rate? Is there a circuit breaker or spending cap?
- Rate and concurrency limits. If P95 latency does not meet expectations, is the issue caused by the model or by queueing?
- Data and compliance. Is input used for model training? What is the retention period? Who can access logs? What deployment options are available, such as public cloud, dedicated resources, or private deployment?
- Exit cost. Is there a contract lock-in period? Can quota or usage data be migrated or exported?
These factors should be converted into cost per effective task, calculated as monthly total spend divided by monthly effective tasks. This is the only truly comparable metric. A price that appears extremely cheap is only valid when utilization is close to the limit.
8. Engineering Practices: Routing, Caching, and Observability
Selecting a model is not the end of the process. In production, three practices can reduce both cost and quality risk.
1. Model cascade
Send simple tasks to a smaller model and escalate complex tasks to a larger model. A simplified routing example is shown below:
def route(prompt, small, large, confidence_gate=0.85, max_retry=1):r = small(prompt)if r.confidence >= confidence_gate and r.validate():return r.text, "small"for _ in range(max_retry):r = large(prompt)if r.validate():return r.text, "large"return fallback_template(prompt), "fallback"
2. Semantic cache
The hit rate for semantically similar queries is often underestimated. For high-frequency FAQ traffic, caching can significantly reduce spending. However, it is important to set expiration times and provide a manual invalidation mechanism.
3. Observability and regression testing
The evaluation set should be integrated into CI. Whenever the model or prompt changes, run the scoring pipeline again and track accuracy, P95 latency, and cost over time. It is also important to validate model output through structured checks, including Schema validation and business rule validation. Failed outputs should enter a fallback path or a human review queue.
9. Data Compliance and Security Boundaries
When user data, code assets, or business documents are involved, at least confirm the following: encryption in transit and at rest, whether logs retain request content by default, whether data is used for model training, cross-border data transfer paths, and the granularity of audit and access control.
For sensitive fields, it is recommended to perform redaction or local preprocessing at the gateway layer. A deterministic fallback path should also be maintained for cases where the model is unavailable. Business continuity should not depend on a single external inference service.
10. Implementation Roadmap
| Stage | Timeline | Deliverables |
|---|---|---|
| 1. Requirement segmentation | 1 week | Scenario list, priorities, and quantitative metric definitions |
| 2. Evaluation set construction | 1 to 2 weeks | 150 to 300 golden samples, scoring scripts, and baseline scores |
| 3. Candidate testing | 1 to 2 weeks | Weighted score table, cost model, and P95 latency report |
| 4. Canary rollout and stabilization | Ongoing | Routing strategy, CI regression, and quarterly review process |
11. Common Pitfalls
- Using leaderboards instead of evaluation. High benchmark scores combined with low business constraint satisfaction are common.
- Looking only at unit price. Ignoring output and retry costs often causes the real bill to be 1.5 to 3 times the original estimate.
- Treating window size as effective understanding. Without testing middle-section decay, long-document tasks may fail silently.
- Ignoring labor cost. The long-term cost of building and maintaining gateways, caches, and monitoring must be included in total cost of ownership.
- Choosing once and never revisiting. Model iteration cycles are often measured in months. Quarterly reviews should be written into the process.
- Working backward from subscription discounts. Traffic characteristics should be defined first, and only then should usage-based or subscription pricing be selected.
The core conclusion is that differences between AI tools are structural, but information gaps can be eliminated. Instead of asking which model is best, teams should ask which dimensions matter for their scenarios, what weights apply, and what scores are required for the model to be usable. With that shift, model selection becomes a measurable, reproducible, and continuously testable engineering decision rather than a one-time brand comparison.
