How to Choose a Domestic LLM for Production: A Scenario-Driven Selection Methodology

    As the number of domestic LLMs grows, choosing one for real-world deployment requires more than leaderboard rankings. This article presents a practical selection methodology: define scenario weights, evaluate five core dimensions—code generation and agent engineering, long-context handling, multimodal support, speed/concurrency/cost, and open-source/private deployment—build a small business-specific evaluation set, and use a decision checklist to narrow candidates while keeping the model access layer abstract enough to switch later.

    Recently, people often ask: there are nearly a hundred domestic LLMs—which one should we use? The standard answer is really just one: there is no absolutely best model, only the model that best fits your scenario. Leading domestic LLMs are converging quickly on general benchmarks, but they differ in emphasis across specific capability dimensions. This article does not discuss specific products; instead, it breaks selection down into an executable methodology.

    1. Why Rankings Are Not the Answer

    LLM capability is not a single metric but a multidimensional vector. A model that leads in mathematical reasoning may not be good at long-document summarization; a model that is strong in code generation may not be equally strong in multimodal understanding. Public leaderboards, such as various general capability benchmarks, usually reflect a model’s average performance on standardized questions, and they have two problems:

    1. Generalization bias: Leaderboard questions differ greatly from real business data distributions. A high ranking does not mean the model performs well on your customer-service scripts, internal technical documents, or industry-specific corpora.
    2. Timeliness bias: Model iteration cycles have shortened to monthly levels, so any static ranking has a very short shelf life.

    The right approach is therefore: first define the weights for your scenario, then match models by dimension, and finally validate with your own data.

    2. Five Core Dimensions for Selection

    1. Code Generation and Agent Engineering Capability

    If your scenario is development assistance—writing business code, doing architecture design, generating unit tests, or running code reviews—focus on two levels:

    • Code quality: Can the model understand complex business context and generate code that follows engineering standards, rather than only toy examples?
    • Agent stability: When the model needs to chain multiple tools and execute multi-step tasks (such as calling internal APIs, querying databases, and generating reports), can it reliably complete the entire chain? Weak agent capability shows up as “drifting off course” mid-task: forgetting tool call results, incorrectly parsing function return values, or falling into retry loops.

    Evaluation suggestion: prepare several tasks with multi-step dependencies (for example, “read API documentation → generate calling code → add exception handling → write tests”) and observe the model’s task completion rate and mid-task failure rate. This reflects engineering value better than single-turn Q&A.

    2. Long-Text Processing Capability

    Long context is a hard requirement in many enterprise scenarios: contract review, aggregation of annual meeting minutes, and cross-analysis across multiple research reports. When evaluating, distinguish between two concepts:

    • Nominal context window: The maximum number of tokens the model claims to support.
    • Effective context length: The range within the window where the model can actually retrieve and understand information reliably. Some models forget middle content or misalign citations when approaching the window limit.

    Test method: construct “needle-in-a-haystack” style cases—embed key information at different positions in a long document, ask the model to answer, and check the recall rate for information in the middle region.

    3. Multimodal Support

    If business inputs include design mockups, screenshots, scanned documents, or audio and video, native multimodal capability can significantly reduce engineering cost. Without native multimodality, you need to build additional preprocessing pipelines for OCR, format conversion, and image slicing; with native capability, these can be eliminated. During evaluation, focus on chart understanding accuracy, layout restoration quality (such as the usability of design-to-frontend code), and robustness against complex layouts.

    4. Inference Speed, Concurrency, and Cost

    For consumer-facing businesses and large-scale Agent clusters, these three factors are often more important than peak capability:

    • Output speed: Flagship models typically operate at the level of tens of tokens per second, but speed drops noticeably during complex reasoning or long outputs. In streaming output scenarios, time to first token (TTFT) and generation speed jointly determine user experience.
    • Concurrency capability: Throughput stability under high concurrency, rate-limiting policies, and queueing latency directly determine the upper limit of business capacity.
    • Unit cost: Billing is based on input/output tokens. In long-context scenarios, input costs may account for a high proportion and need to be measured separately. A rough estimate: if the output speed is 50 tokens per second, in Chinese scenarios this corresponds to roughly tens of Chinese characters per second, which is generally sufficient for interactive applications; batch offline tasks can tolerate slower but cheaper models.

    5. Open Source and Private Deployment

    For scenarios involving data compliance, intranet isolation, or industry regulation, private deployment is almost mandatory. In this case, focus on:

    • Whether the model is open source and whether the commercial license terms are permissive;
    • Whether the open-source version’s weight specifications (parameter-size tiers) match your GPU resources, and whether quantized versions exist to lower the deployment barrier;
    • Whether there is mature deployment ecosystem support (adaptation to mainstream inference frameworks and completeness of fine-tuning toolchains).

    3. Build Your Own Evaluation Set

    General benchmarks cannot replace business validation. A lightweight approach is to build a dedicated evaluation set and run scoring through a unified API. Example code:

    1. import time
    2. def evaluate(model_endpoint, test_cases):
    3. results = []
    4. for case in test_cases:
    5. start = time.time()
    6. resp = call_llm(model_endpoint, case["prompt"])
    7. latency = time.time() - start
    8. score = judge(case["reference"], resp) # Rule-based scoring or model-as-judge
    9. results.append({
    10. "case": case["id"],
    11. "score": score,
    12. "latency": round(latency, 2),
    13. })
    14. return results

    The evaluation set should cover: the 20 most frequent task types in your business, the boundary samples most likely to cause errors, and typical long-text inputs. When using a “model-as-judge” for scoring, keep samples for manual spot checks to avoid the judge model systematically favoring a certain output style.

    4. A Decision Checklist

    During selection, narrow the scope in the following order:

    1. Compliance constraints: Is private deployment mandatory? Is a domestic technology stack required?
    2. Scenario weights: Code / long text / multimodal / high-concurrency Agent—rank them and take the top two.
    3. Cost budget: Estimate daily average token volume and calculate the monthly cost range for candidate models.
    4. Empirical validation: Run 2–3 candidate models on your own evaluation set and compare accuracy, latency, and cost.
    5. Architecture fallback: Use a unified model gateway layer for abstraction, retain the ability to switch models, and avoid lock-in to a single vendor.

    5. Final Thoughts

    The LLM market is still evolving rapidly, and today’s leading model may be overtaken in six months. Rather than betting on one “most promising” model, it is better to focus on two things: first, build evaluation and validation capabilities for your own business; second, design an abstraction layer for model access. Tools will change, but methodology will not become obsolete—this is the optimal strategy for dealing with rapid model iteration.