A Practical Guide to Selecting Domestic LLMs: Evaluation Methods and Inference Deployment

1. Background: Once Models Reach the First Tier, Selection Gets Harder

For years, the shelf of large model inference options was dominated by overseas open-source models. Domestic teams that wanted to use domestic models often had to assemble their own inference engines, package their own containers, and handle acceleration and scheduling themselves. That situation is changing in a substantive way: flagship models released by several domestic vendors are now being included in mainstream inference platforms as standardized inference microservices. Developers no longer need to worry about differences between underlying engines — they can call an interface and get results. One engineering detail worth noting is that a new model sometimes appears in a platform catalog before the official announcement. In practice, verification should be based on actual calls, not on press releases.

For developers, this is both good news and a new problem. The good news is that the cost of accessing models has dropped significantly. The new problem is this: when several domestic flagship models all cluster within the narrow band of the “first tier,” and their leaderboard rankings are nearly identical, “which one to choose” is no longer a question that rankings alone can answer. It becomes a decision that requires engineering validation. This article is organized around that decision process: first, how to read leaderboards; second, how to test with your own data; and finally, how to evaluate deployment and cost.

2. First, Clarify What Inference Microservices Actually Solve

Before discussing model selection, it is worth understanding the nature of the “inference microservice” delivery format, because it is an important trend in model deployment and directly affects how comparisons are made.

A standardized inference microservice typically does three things:

1. Packaging the model and inference engine: It bundles model weights, inference acceleration components, and dependencies into a container that can run independently, abstracting away differences in underlying hardware and engines.
2. Unified interface: It exposes a chat completion interface format that is common across the industry. Switching models often requires only changing a model name parameter, without rewriting business code.
3. Elastic scaling: It relies on a container platform to scale instances up and down and handle traffic on demand.

These services initially focused on overseas open-source models. Now that domestic flagship models are entering the catalog one after another, there is a direct benefit for anyone doing selection work: multiple candidate models can be compared horizontally using the same evaluation code and the same deployment format, decoupling “differences in model capability” from “differences in deployment method.” This is a prerequisite for rigorous comparison.

3. Step One: Interpret Evaluation Leaderboards Correctly

Leaderboards are the first filter in model selection, but they must be used correctly.

3.1 Look at leaderboards by domain. General capability leaderboards measure the “foundation” of knowledge and reasoning, while real business scenarios often focus on specific capabilities: code generation, long-document understanding, instruction following, and tool calling. In programming scenarios, for example, there are leaderboards specifically designed for web development tasks, with questions that are closer to real front-end engineering. If your business is a coding assistant, these vertical leaderboards are usually more informative than a general one.

3.2 Cross-validate; do not rely on a single leaderboard. A single leaderboard may suffer from data contamination, question leakage, or targeted optimization. A safer approach is to compare at least two or three independent leaderboards for the same capability dimension. If the conclusions agree, confidence increases substantially. If the conclusions conflict, the capability is in a range where the leaderboards cannot distinguish between models, and their discriminating power is limited. At that point, move to the next step: let your own data speak.

3.3 Pay attention to the test conditions behind the leaderboard. The same score may come from different context lengths, different tool-calling permissions, or different sampling configurations. This is especially true for long-context and agent-style evaluations, where configuration differences can have a large impact on results. When reading a leaderboard, look at the evaluation setup first, then the ranking.

4. Step Two: Build a Business-Specific Evaluation Set

Leaderboards can only filter candidates down to the “first tier.” Ranking within that tier depends on your own data. This step is commonly called a POC (proof of concept), and the key lies in two areas: samples and scoring.

4.1 Sample from real business scenarios. Draw 100–300 samples from online logs, historical tickets, or manually curated typical questions, and classify them by capability dimension:

• Factual Q&A: knowledge accuracy and the ability to refuse when the model does not know;
• Logical reasoning: multi-step reasoning, mathematical calculation, constraint solving;
• Code generation: from natural language requirements to runnable, maintainable code;
• Long-document understanding: summarization, cross-paragraph information retrieval, multi-document comparison;
• Instruction following: format constraints, length constraints, and prioritization when multiple instructions conflict;
• Tool calling: function selection and parameter filling accuracy in multi-turn agent scenarios.

4.2 Design the scoring method. There are two mainstream approaches. The first is human scoring: define a rubric and score by dimension, which suits small-sample fine-grained evaluation. The second is LLM-as-Judge: use a stronger model to score automatically against the same rubric, which suits large-batch initial screening, followed by human review of disputed samples.

4.3 Example evaluation script. The following code uses the industry-standard chat completion interface format and can be used directly for horizontal comparison:

import json
import urllib.request
ENDPOINT = "https://<your-inference-service-endpoint>/v1/chat/completions"
API_KEY = "<your-access-credential>"
def call_model(model_name, prompt, temperature=0.2):
    payload = {
        "model": model_name,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
        "max_tokens": 1024,
    }
    req = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + API_KEY,
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        body = json.loads(resp.read().decode("utf-8"))
    return body["choices"][0]["message"]["content"]
def judge(reference, answer):
    """Scoring function: human scoring against a rubric, or automatic scoring by a stronger model (LLM-as-Judge)."""
    raise NotImplementedError  # Implement according to your business rubric; return a score from 0 to 1
def run_benchmark(cases, model_name):
    """cases: [{"id": 1, "prompt": "...", "reference": "...", "category": "code"}]"""
    scores = []
    for case in cases:
        answer = call_model(model_name, case["prompt"])
        score = judge(case["reference"], answer)
        scores.append({"id": case["id"], "score": score})
    avg = sum(s["score"] for s in scores) / len(scores)
    return {"model": model_name, "avg": avg, "details": scores}
if __name__ == "__main__":
    cases = json.load(open("bench_cases.json", encoding="utf-8"))
    for m in ["model-a", "model-b", "model-c"]:
        print(run_benchmark(cases, m))

Three engineering details deserve emphasis. Fix sampling parameters: keep parameters such as temperature consistent across all candidate models, otherwise comparisons become distorted. Watch stability: run each sample three times and look at the variance. A model with high variance is a clear negative in serious business scenarios. Report by dimension: when total scores are close, look at which model is stronger on your core dimensions rather than looking at the average.

5. Step Three: Evaluate the Inference Deployment Path

Once evaluation determines “which model to use,” you still need to decide “how to run it.” There are currently three main paths:

5.1 Call a hosted API. This is the lightest option, suitable for rapid validation and small-scale business. Cloud providers such as Baidu AI Cloud offer hosted large model inference capabilities, along with common features such as authentication, rate limiting, and monitoring and alerting, so teams do not need to manage underlying compute or inference engines. The advantages are fast startup and no operations burden; the disadvantage is that deep customization is relatively limited.

5.2 Deploy a standardized inference container. This is the inference microservice format mentioned above: the model is packaged as a standard container, runs on a container platform, and the platform handles scheduling and elastic scaling. It suits teams with some operations capability that want both “out-of-the-box” convenience and controllable resources, and it is also the most convenient format for horizontal comparison across multiple models.

5.3 Build your own inference cluster. Teams with extremely high requirements for data compliance and cost control may choose to build their own: select an open-source inference engine and deploy it on their own GPU cluster. This offers the greatest flexibility, but requires investment in inference optimizations such as quantization, continuous batching, and KV cache reuse, as well as cluster operations. Choose carefully if your team is not large enough.

The three paths can be summarized as follows: hosted APIs start fastest and require the least operations, making them suitable for the validation phase; standardized containers sit in between and suit stable business at medium scale; self-built clusters offer the most customization and suit large-scale and strongly regulated scenarios.

Whichever path you choose, you should test three inference metrics carefully: time to first token (TTFT), which determines the perceived responsiveness of interactive scenarios; throughput and concurrency, which determine how many requests a given unit cost can support; and long-context cost, since longer contexts increase the memory overhead of attention computation and KV cache, and the token unit price may not be linearly related to the real cost.

6. Step Four: Cost and Engineering Implementation

6.1 Look beyond unit price to “actual cost.” With the same token unit price, actual bills can differ greatly. Confirm whether prefix caching is supported (this has a huge impact on workloads with repeated system prompts), whether batch discounts exist, and how tiered pricing rules work. Estimate monthly cost by replaying real traffic rather than simply multiplying unit price by token volume.

6.2 Prepare fallback and routing. A model service is still a service, and it will also experience jitter, rate limiting, and timeouts. In a production architecture, the model name should be configuration rather than hard-coded, and a fallback path should be prepared:

def log_event(model, message):
    # Connect to logging and monitoring/alerting; shown here for illustration only
    print(f"[router] {model} failed: {message}")
class ModelRouter:
    def __init__(self, primary, fallbacks):
        self.primary = primary
        self.fallbacks = fallbacks
    def complete(self, prompt, **kwargs):
        candidates = [self.primary] + self.fallbacks
        last_err = None
        for m in candidates:
            try:
                return call_model(m, prompt, **kwargs)
            except Exception as e:
                last_err = e
                log_event(m, str(e))
        raise RuntimeError(f"All candidate models failed: {last_err}")
router = ModelRouter("model-a", ["model-b", "model-c"])
answer = router.complete("Help me summarize the three core conclusions of this document")

This improves availability and also lays a foundation for continuous iteration with “canary rollout for new models and fallback to old models.”

6.3 Establish regression evaluation. After a large model version upgrade, capabilities may drift. Make the evaluation set a fixed part of the pipeline, automatically running core cases every time the model version changes, to avoid incidents where “an upgrade becomes a regression.”

7. Common Pitfalls Checklist

• Parameter-count worship: Parameter count and capability are not linearly related; architecture design and training data quality matter just as much.
• Leaderboard worship: Leaderboards are for filtering tiers; ranking within a tier must rely on business data.
• Ignoring context cost: Long-context capability does not mean long context is free; latency and cost rise with length.
• Ignoring version iteration: Large models iterate quickly, and today’s conclusion may be invalid in six months. Model selection is a continuous process, not a one-time decision.
• Ignoring evaluation set leakage: Make sure evaluation samples have not appeared in public corpora; otherwise scores are inflated and rankings distorted.

8. Conclusion

Returning to the original question: which domestic large model should you choose? When flagship models from multiple vendors have all reached the first tier and can all be accessed conveniently through standardized inference services, the answer depends increasingly on your business profile, cost structure, and engineering capability — not on a ranking on some leaderboard. A viable decision path is: use vertical leaderboards to filter down to two or three candidates; use a business-specific evaluation set to rank within the tier; use real traffic to measure latency, throughput, and cost; and use routing and fallback architecture to preserve the ability to switch at any time. Model selection is not the endpoint. Turning evaluation sets, regression pipelines, and multi-model routing into team assets has more long-term value than “betting on one model.”