How to Choose and Deploy Chinese Open-Source LLMs: A Practical Guide
When evaluating Chinese large language models, teams typically face three questions at once: Is quality good enough for our tasks? Can we get deployment working? Can we afford the bill? This guide is organized around those three questions and provides checklists and formulas you can follow directly. After reading, you will be able to build an evaluation sheet for candidate models and know exactly what to verify before going live.
How to Choose: Break Tasks into Measurable Metrics First
Public leaderboards are useful for initial screening, but leaderboard scores reflect average performance on generic tasks and do not directly represent your business. Follow these steps:
- Record the leaderboard name, version, and evaluation date. Leaderboards update frequently; without version records, conclusions cannot be reproduced.
- Split tasks by business scenario, such as code completion, knowledge-based Q&A, long-document summarization, and structured information extraction. Set a separate pass threshold for each task type.
- Prepare at least 200 real samples per task type. Have two annotators score independently, with disputed samples resolved by a third reviewer.
- Use identical prompts and inference parameters across all candidate models, and save full outputs for later comparison.
Organize your selection checklist in four columns: task type, key metric, test method, and elimination criterion. For example, for knowledge-based Q&A, the key metric is the hallucination rate; the test method is manually verifying 200 randomly sampled responses; a hallucination rate above a preset threshold (e.g., 5%) eliminates the model. For code completion, the key metric is pass@1 rate, tested by running generated code against your project’s own test suite.
Use A/B testing for the final decision. Route a small percentage of real traffic—say 5%—to the candidate model and run it for at least a week. Track P95 latency, QPS (requests per second), token cost per call, and user satisfaction. Average latency alone hides long-tail problems; P95 and P99 reflect real user experience much better.
How to Deploy: Estimate VRAM First, Then Pick a Framework
Before deployment, confirm three things: the inference framework and its version, hardware type and count, and the model’s quantization method.
The basic VRAM estimation formula is: weight VRAM ≈ parameter count × bytes per parameter. FP16 uses 2 bytes per parameter, INT8 uses 1 byte, and INT4 uses roughly 0.5 bytes. For a 7B-parameter model, FP16 weights require about 14GB and INT8 about 7GB. Actual usage also includes KV cache and runtime overhead, so reserve a 20%–30% margin. KV cache grows with context length and concurrency; long-context scenarios need separate estimation.
Quantization involves a trade-off between quality and cost. INT8 usually has a minor impact on quality, but INT4’s effect on complex reasoning and coding tasks must be measured with real samples—never assume it’s lossless. Before launch, compare scores between FP16 and the quantized version on the same test set; if the gap exceeds your threshold, drop that quantization option.
Dynamic batching has three key parameters: max batch size, max wait time, and max concurrent sequences. A wait time set too long raises per-request latency; too short, and GPU utilization stays low. Start with a small wait time—around 10 milliseconds—then use load-testing curves to find the inflection point between throughput and latency.
At the API layer, if the platform offers an OpenAI-compatible interface, code changes are minimal: switching models usually only requires changing the model name and service endpoint. It’s still wise to keep an adapter layer of your own to handle differences in parameter ranges and response fields across models.
A minimal auto-scaling configuration includes: minimum instances, maximum instances, a scaling trigger metric (such as queue length or GPU utilization), and cold start time. If cold starts are slow, don’t set minimum instances to zero, or traffic spikes will cause noticeable first-token delays.
A minimal observability checklist includes: request volume, success rate, P50/P95/P99 latency, time to first token, token usage, and error code distribution. Set alert thresholds based on your business’s historical baselines rather than defaults.
Troubleshooting path: when latency rises, first check whether the request queue is backing up, then check GPU utilization and VRAM against their limits, and finally check whether per-request input length has grown abnormally. When error rates rise, categorize by error code first—separating timeouts, out-of-memory failures, and content-filter blocks—then handle each accordingly.
How to Calculate Cost: Break Down Token Pricing
The basic inference cost formula is: cost per call = input tokens × input price + output tokens × output price. Monthly cost = cost per call × monthly call volume + instance costs + storage costs.
Example: a customer support scenario averages 1,200 input tokens and 300 output tokens per call, with 500,000 calls per month. Assume input pricing of $2 per million tokens and output pricing of $8 per million tokens (figures for illustration only—use your actual pricing). Cost per call = 1,200 ÷ 1,000,000 × 2 + 300 ÷ 1,000,000 × 8 = $0.0048, giving monthly token costs of roughly $2,400. Instance and storage costs must be added separately.
Three main ways to reduce costs:
- Compress prompts: remove redundant descriptions while keeping fixed system instructions unchanged, improving cache hit rates.
- Cache results for repetitive questions, and pay attention to cache TTL and user data isolation.
- Build model routing: send simple intents to lightweight models and escalate complex tasks to flagship models. Validate routing rules with log data to avoid misrouting that degrades quality.
Model upgrades also need a process. Before a new version goes live, run it with 5% traffic in a gray release, compare the same set of metrics against the old version, and only then ramp up gradually. Have kill switches and rollback plans ready before the gray release begins.
Security and Compliance: Verify Item by Item Before Launch
Data isolation, access control, audit logging, and content safety filtering are four baseline requirements. Check the following before going live:
- Does the service agreement explicitly state that user inputs will not be used for further model training? Get written confirmation.
- Are API credentials separated across development, testing, and production environments? Are secrets stored in a secret management service rather than in code or logs?
- Do logs record the caller, call time, and model used? Does the retention period meet compliance requirements?
- Are model outputs passed through content safety filtering? Are interception records queryable?
- Do role permissions follow least privilege—for example, developers can only access the test environment, and production keys are readable only by operations staff?
Model selection should be based on how a model actually performs under your tasks, architecture, cost, and compliance constraints—not on parameter counts or leaderboard rankings. Spend a week on sample evaluation and small-scale traffic testing before committing to large-scale investment. This step typically saves significant rework costs later.
