How to Choose a Chinese Open-Source LLM: A Practical Selection Guide

    As more Chinese open-source LLMs appear on inference platforms, teams increasingly need a reliable way to decide which one to use. This guide explains the "model marketplace" model and why it matters, shows where public leaderboards fall short, and introduces a four-dimension selection framework covering task fit, engineering compatibility, cost and performance, and stability, compliance, and data boundaries. It also walks through building a small, reproducible evaluation set, architecture patterns that make models swappable, and a pre-launch checklist. The goal is to move model selection from "looking at rankings" to "looking at evidence."

    Over the past year, the number of Chinese open-source LLMs available on model inference platforms has grown noticeably, and more and more teams are facing the same question: with so many models, which one should we actually pick?

    The hard part of choosing a model isn’t finding models—it’s having a reliable basis for the decision. Leaderboard rankings are only an entry point. What really determines whether a deployment succeeds is whether the model can consistently produce usable results on your task distribution, and whether integration, cost, and operations are sustainable over the long term. Set against the backdrop of a wave of flagship Chinese open-source models landing on platforms at once, this article moves model selection from “looking at rankings” to “looking at evidence.”

    The “Model Marketplace” Model: Why It Matters

    What a model inference platform is

    A model inference platform (sometimes called inference hosting or a “model marketplace”) is a service in which cloud or chip vendors pre-deploy mainstream open-source models on their own hardware and expose them through a unified API. Developers don’t need to provision their own GPU servers or deploy and tune an inference framework; with just API credentials, they can start calling models and pay for what they use.

    Put simply, a model used to be like a large appliance you had to buy, install, and maintain yourself. Now it’s a product you pick off the shelf and use right away, and models on the same tier often share a single integration method.

    Why Chinese open-source models are landing on platforms in numbers

    There are at least three forces behind the arrival of Chinese open-source models on mainstream inference platforms:

    • A lower capability barrier: A number of Chinese flagship models are now close to the top tier, handling everyday text generation, code generation, and tool calling well, which makes them clearly more attractive.
    • A lower deployment barrier: Most models are released as open weights, so inference platforms can reuse their existing inference engines to deploy them at very low cost.
    • Rising demand: Growing needs around Chinese-language scenarios, compliance requirements, and data localization keep increasing the willingness of developers and enterprises to use Chinese models.

    Three practical benefits of mass listings

    • Very low trial cost: In a few days and with a small budget, you can run a side-by-side comparison across several models.
    • Unified integration: Most models on the same tier are compatible with the industry-standard chat completions style API, so switching models usually means replacing a single config value. Note that “compatible” here means the request and response field structures match; the degree of support for extensions like tool calling and structured output still needs to be verified case by case.
    • No longer locked to a single vendor: Models become replaceable components, so you can pick the best and combine as needed instead of betting your entire business on one model at selection time.

    It’s worth noting that a model appearing on an inference platform usually means the platform has already adapted and validated it—but that doesn’t mean there’s an official announcement, nor does it amount to an official endorsement of the model’s capabilities. A selection team should treat this as “one more convenient testing entry point,” not “someone already did the selection for you.”

    What Leaderboards Can and Cannot Tell You

    The value of leaderboards: narrowing candidates quickly

    Public evaluation leaderboards (such as specialized leaderboards for web-development tasks) still have real value. They can quickly filter out clearly unqualified models and, for a given period, provide a list of top-tier models with roughly comparable capabilities, letting teams focus limited testing effort on a few candidates.

    Four blind spots of leaderboard rankings

    • Benchmark contamination: With rapid model iteration, public test questions may already be in the training data, so a high score doesn’t necessarily reflect generalization ability.
    • Mismatched scope: Leaderboards lean toward a single task type, while business scenarios are usually mixed—multi-turn dialogue, tool calling, long-document understanding, Chinese table extraction, and so on. Leaderboard scores can’t be extrapolated directly to a real distribution.
    • Averages hide the long tail: Two models with similar composite scores may differ—one is stable overall with occasional slips, while the other shines on most cases but fails outright on a few critical ones. In production, the long tail often determines the experience.
    • Missing cost dimension: Leaderboards usually don’t reflect how many tokens different models consume to generate content of the same length. Differences in token efficiency translate directly into differences in your bill.

    The right way to use them: leaderboards for shortlisting, real tests for deciding

    Leaderboards are for making the shortlist; real tests are for setting the standard. A pragmatic process: first pick one or two leading models from the leaderboard as a baseline, then add one or two models with different positioning as controls (for example, a lightweight model for high-frequency simple tasks), and finally let a self-built evaluation set be the judge.

    A Four-Dimension Selection Framework: From “Which Is Better?” to “Which Fits Me?”

    Task fit

    First, identify which category your task falls into, then validate accordingly:

    • Generation: long-form writing, summarization and rewriting, translation.
    • Code: completion, single-file generation, cross-file changes, web page generation.
    • Structured output: information extraction, classification, generating JSON that conforms to fixed field requirements.
    • Agents: tool calling, multi-step planning, long-horizon tasks.

    Each category needs to be evaluated separately on real samples. The same model ranking high in code generation doesn’t mean it’s equally stable at structured extraction.

    Engineering compatibility

    • API compatibility: whether request and response fields, and the incremental structure of streaming responses, are consistent.
    • Extended capabilities: whether features like tool calling, structured output (e.g., JSON mode), and logprobs are supported, and whether the returned format is strict.
    • Context length: distinguish between “the maximum supported context” and “still performing well at that length”—these are two different things.
    • SDKs and ecosystem: whether mature SDKs, documentation, and error-code descriptions are available.
    • Version strategy: whether model versions are silently upgraded and whether fixed version snapshots are offered.

    Cost and performance

    Cost isn’t just about unit price; build an estimation model for the combined cost per task:

    cost per task ≈ (input tokens × input price + output tokens × output price) + extra consumption from retries

    The more verbose the prompt and the more talkative the model, the higher the cost. For performance, measure time to first token, per-token generation speed, and end-to-end latency separately. For interactive products, time to first token often matters more to the user experience than total throughput.

    Stability, compliance, and data boundaries

    • Service level: availability commitments, rate-limit thresholds, and whether error semantics on failure are clear.
    • Data security: whether request data is retained or used for training, and whether sensitive data needs to be masked before sending.
    • Deployment options: whether private or dedicated-instance deployment is available, and whether it can meet data-residency requirements.
    • Replaceability: whether a self-hosted fallback is preserved, to avoid being locked in by a single vendor.

    The model market changes fast, so conclusions on these dimensions will shift over time; it’s worth re-checking them once a quarter.

    Building a Reproducible Evaluation at Minimal Cost

    The key to selection is turning gut feeling into numbers. I recommend building a small evaluation set of 50 to 200 cases that samples your real business distribution, while keeping a batch of edge cases.

    You can organize the evaluation set using a structure like the one below:

    1. # Example evaluation set: each case contains an input, judging rules, and key points of a reference answer
    2. eval_cases = [
    3. {
    4. "id": "case_001",
    5. "task": "structured_extract", # task type, used for grouped statistics
    6. "messages": [
    7. {"role": "system", "content": "Extract fields from the text; output JSON only."},
    8. {"role": "user", "content": "Order number A-1024, order time 2025-03-12, amount 199 yuan."}
    9. ],
    10. "checks": ["json_valid", "field_match"] # list of validators, can be judged automatically
    11. },
    12. # ... more real business samples
    13. ]

    The test-running script’s job is to run each candidate model and output comparable metrics:

    1. import time, json, statistics
    2. def run_eval(client_factory, cases):
    3. """
    4. client_factory: takes a model identifier and returns a client with a chat method.
    5. The access addresses and keys for each platform are injected via environment variables,
    6. making it easy to switch between environments.
    7. """
    8. results = {}
    9. for model_tag in ["baseline-model", "candidate-model-a", "candidate-model-b"]:
    10. client = client_factory(model_tag)
    11. rows = []
    12. for case in cases:
    13. start = time.perf_counter()
    14. resp = client.chat(
    15. messages=case["messages"],
    16. temperature=0.2,
    17. max_tokens=1024
    18. )
    19. latency_ms = (time.perf_counter() - start) * 1000
    20. usage = resp.get("usage", {})
    21. rows.append({
    22. "id": case["id"],
    23. "latency_ms": latency_ms,
    24. "in_tokens": usage.get("prompt_tokens", 0),
    25. "out_tokens": usage.get("completion_tokens", 0),
    26. "content": resp["choices"][0]["message"]["content"],
    27. "passed": all(check(case, resp) for check in case["checks"])
    28. })
    29. passed = sum(r["passed"] for r in rows)
    30. results[model_tag] = {
    31. "pass_rate": passed / len(rows),
    32. "p50_latency_ms": statistics.median(r["latency_ms"] for r in rows),
    33. "p95_latency_ms": sorted(r["latency_ms"] for r in rows)[int(len(rows) * 0.95)],
    34. "avg_out_tokens": statistics.mean(r["out_tokens"] for r in rows),
    35. "failures": [r["id"] for r in rows if not r["passed"]],
    36. "detail": rows
    37. }
    38. return results
    1. # Minimal routing example: simple tasks go to the lightweight model first, then fall back to the flagship model on failure
    2. ROUTE_TABLE = {
    3. "simple": {"primary": "lite-model", "fallback": "flagship-model"},
    4. "complex": {"primary": "flagship-model", "fallback": "lite-model"},
    5. }
    6. def route_call(client_factory, task_type, messages):
    7. route = ROUTE_TABLE[task_type]
    8. for tag in (route["primary"], route["fallback"]):
    9. try:
    10. return client_factory(tag).chat(messages=messages)
    11. except (TimeoutError, ConnectionError):
    12. continue # can hook in monitoring and alerts, and log downgrade events
    13. raise RuntimeError("all candidates failed")

    I recommend summarizing the results into a table like the one below, so differences across dimensions are visible at a glance:

    Model Task pass rate P50 latency P95 latency Avg output tokens Main failure type
    Baseline TBD TBD TBD TBD TBD
    Candidate A TBD TBD TBD TBD TBD
    Candidate B TBD TBD TBD TBD TBD

    Combining pass rate with latency exposes a typical kind of problem: a model with a high composite score may often emit extra explanatory text on structured output, causing parsing failures and higher costs. Only real testing surfaces problems like this.

    Architecture: Making Models Swappable Components

    A unified access layer

    Don’t let business code depend directly on a specific model’s client. Add an adapter layer in the middle to hide the differences between platforms in authentication, fields, and error codes, exposing only a unified interface upward.

    Model routing and fallback

    Not every request needs the flagship model. Routing by task complexity can significantly reduce average cost:

    • Simple classification, extraction, and rewriting go to the lightweight model;
    • Complex reasoning, long-form generation, and code refactoring go to the flagship model;
    • When the primary model times out or is rate-limited, automatically fall back to a backup model to ensure availability.

    Prompt and context engineering

    With the same model, differences in prompt quality can lead to more than 20% in effectiveness. The system prompt should clearly state the role, output format, and boundaries; few-shot examples should cover typical hard cases; and in long-context scenarios, pay attention to the positional effect of key information, putting the most important instructions at the beginning and end.

    Caching and batching

    High-frequency repeated requests (like summaries from a fixed template) can use semantic caching or prefix caching to reduce redundant computation. Offline batch jobs should prefer APIs that support batch calls, further lowering the unit cost.

    Pre-Launch Checklist

    • Evaluation coverage: Does the evaluation set cover the top 80% of scenarios in your real distribution?
    • Long-tail performance: Do failure samples have a clear fallback strategy (human review, a second attempt, a degraded response)?
    • Cost estimate: Based on estimated QPS and average token consumption, does the monthly bill stay within budget?
    • Monitoring and alerts: Have you instrumented latency, error rate, token consumption, and downgrade counts?
    • Compliance review: Does the data involve personal or sensitive information, and does it need masking or localized deployment?
    • Exit plan: Have you kept at least two switchable models or platforms?

    Conclusion

    Chinese large models entering mainstream inference platforms is a double signal of maturing capability and an open ecosystem, but it only broadens your options—it doesn’t make the decision for you. Use leaderboards for shortlisting, real tests for setting the standard, and architecture as a safety net. Turning model selection into a reproducible, iterative process is more resilient to a fast-changing technical landscape than betting everything on a single strongest model in one shot.