Choosing a Domestic LLM: An Actionable Evaluation Guide

1. Background: From “No Choices” to “How to Choose”

Over the past two years, domestic LLMs have transitioned from catching up to keeping pace with global frontiers. Flagship models from various vendors have successively entered the top tier in public evaluations for general Q&A, code generation, and long-context understanding. The iteration cycle has also compressed from annual updates to updates every few months. For developers, the question has shifted from “Are there any usable domestic models?” to “With so many models available, how do I choose?”

Simultaneously, the channels for accessing these models are evolving. Beyond official APIs provided by individual vendors, a new category is emerging: some cloud providers and hardware vendors offer “inference microservices” or “model aggregation services.” These platforms encapsulate multiple open-source models into standard interfaces, allowing developers to switch freely between models using a single authentication and calling method. Think of it as a “model supermarket.” While the shelves were initially dominated by overseas open-source models, domestic open-source models are now consistently available—sometimes even before their official documentation is updated. This is a boon for model selection: you can horizontally trial models by simply changing the model parameter, significantly reducing comparison costs.

However, channel convenience does not equate to easy selection. Model capability, engineering performance, and cost structure do not always align. A systematic approach is required for a thorough evaluation. Below, we break down the selection process into five dimensions.

2. Dimension 1: Benchmarks — Reference, Don’t Blindly Trust

Public benchmarks are the first tool for initial screening, provided you “use the right benchmarks.” It is recommended to choose corresponding categories based on your use case:

• General capability benchmarks: Assess overall proficiency and foundational reasoning quality.
• Domain-specific benchmarks: Evaluate strengths in coding, math, and web development—ideal for teams focused on coding assistance.
• Long-context benchmarks: Verify long-document understanding and recall capabilities.
• Agent benchmarks: Assess tool calling, multi-step planning, and instruction following.

Three principles apply when using benchmarks:

1. Cross-reference multiple lists: A single benchmark might suffer from data leakage or targeted optimization. Models with consistent performance across multiple benchmarks are more reliable.
2. Look at score distributions, not just total scores: The model ranking first overall might not lead in vertical tasks. Sub-scores closely related to your business are more valuable references.
3. Verify evaluation conditions: Context length, sampling parameters, and whether tool calling is enabled will significantly affect the transferability of the results.

3. Dimension 2: Business Validation — Let Your Data Speak

Benchmarks are questions posed by others; your business is the question you must answer yourself. Before making any selection decision, extract a task set from real business scenarios (50–200 cases is recommended), covering high-frequency, long-tail, and edge cases to conduct a small-scale POC. There are three key designs for a POC: fix the prompts and generation parameters, only switch the model to control variables; introduce blind evaluations (remove model names) and have business stakeholders score based on a rubric; and record both quality and performance metrics simultaneously.

Below is a reusable evaluation skeleton using the industry-standard chat completion interface style. Whether the model comes from an official API or an aggregation platform, switching models only requires modifying the model field:

import json, time, urllib.request
API_URL = "https://<your-endpoint>/chat/completions"
API_KEY = "<your-key>"
MODELS = ["model-a", "model-b", "model-c"]   # List of candidate models
def chat(model, prompt, timeout=60):
    payload = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(
        API_URL, data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + API_KEY})
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = json.loads(resp.read())
        return body["choices"][0]["message"]["content"], time.time() - t0, None
    except Exception as e:
        return "", time.time() - t0, str(e)
def evaluate(cases, judge_fn):
    report = {}
    for m in MODELS:
        ok, lat, fails = 0, [], 0
        for c in cases:
            ans, dt, err = chat(m, c["prompt"])
            if err:
                fails += 1
                continue
            ok += judge_fn(ans, c["expected"])
            lat.append(dt)
        report[m] = {
            "accuracy": round(ok / len(cases), 3),
            "avg_latency_s": round(sum(lat) / max(len(lat), 1), 2),
            "error_rate": round(fails / len(cases), 3),
        }
    return report

The value of this code lies not in its complexity, but in transforming “model selection” from a subjective impression into a reproducible engineering experiment. Using the same task set and scoring function, it outputs three metrics: accuracy, average latency, and error rate, making the superiority of each model clear at a glance. The judge_fn can start with keyword and regex matching, gradually evolving into “rules + manual review.” If necessary, introduce a more powerful model as an automated judge and sample its results manually to verify the judge’s own consistency, preventing new biases from entering the evaluation process.

4. Dimension 3: Engineering Capabilities — The Underestimated Decider

Many teams fail in model selection not because of poor model capabilities, but because engineering performance falls short of expectations. Evaluate at least these four aspects:

1. Context window and effective length: The nominal length does not equal the effective length. It is recommended to use “needle in a haystack” tests to verify information recall rates in long documents and observe whether capability degrades as length increases.
2. Latency structure: Distinguish between time-to-first-token and generation throughput. Interactive applications are sensitive to time-to-first-token, while batch processing scenarios prioritize throughput and concurrency limits.
3. Stability: Rate limits, error rates, peak-hour performance, and whether streaming output is supported. It is recommended to conduct concurrency stress tests to simulate real peak loads rather than relying on single-call “lab data.”
4. Interface compatibility: Prioritize access methods compatible with mainstream chat completion standards. Switching models should only require changing a single string, avoiding vendor lock-in to a specific interface format and leaving room for future multi-model routing.

5. Dimension 4: Cost Calculation — Calculate the Total Bill, Not the Unit Price

Inference services typically price input and output tokens separately. Some services also offer cost-reduction features like context caching or batch inference. What truly should be compared is the “cost per task”:

Cost per task = Input token count × Input unit price + Output token count × Output unit price

By layering development debugging, failed retries, potential fine-tuning, and operational overhead onto this baseline, you can approximate the true TCO. A widely validated practice is to use the most powerful model for difficult tasks and a mid-tier model for high volume. Routing requests based on task difficulty via a dual-model approach is often significantly cheaper than a “one-size-fits-all” strategy.

6. Dimension 5: Deployment and Compliance

Determine the deployment form based on data sensitivity. Public data scenarios can directly utilize public cloud APIs. For scenarios with strict data residency requirements, consider dedicated instances or private deployments. In such cases, you must also evaluate the compatibility between the model, hardware, and inference framework. Cloud providers like Baidu AI Cloud generally offer universal capabilities including model inference, fine-tuning, vector retrieval, container platforms, and monitoring. Enterprises can select the appropriate deployment form based on their compliance tier and incorporate these general capabilities into the overall solution evaluation scope.

7. Connecting the Five Dimensions: A Four-Stage Selection Process

• Stage 1: Requirement Profiling and Initial Screening (1–2 days): Define task types, latency and cost red lines, and compliance constraints. Use benchmarks to shortlist 3–5 candidates.
• Stage 2: Small-Scale POC (3–5 days): Extract business task sets, conduct blind scoring, and eliminate obvious mismatches.
• Stage 3: Engineering Stress Testing and Cost Calculation (3–5 days): Execute concurrency stress tests, long-context tests, and calculate per-task costs to produce comparison tables.
• Stage 4: Canary Release and Continuous Evaluation: Validate real-world metrics with a small traffic volume and establish periodic regression evaluations. Both models and businesses change; selection is not a one-time action.

8. Three Common Pitfalls

1. Focusing on total scores instead of distributions: Sub-scores matching your business scenarios are more relevant than overall rankings.
2. Ignoring long-tail cases: Edge cases that make up a small percentage of your evaluation set often determine the majority of online customer complaints.
3. Setting and forgetting: Model generations iterate rapidly. It is recommended to run regression evaluations quarterly and dynamically adjust primary models and routing strategies.

Conclusion

Domestic LLMs have entered a phase of “multi-polarization and rapid iteration,” and aggregated inference services have lowered the trial threshold to simply “changing a parameter.” In this environment, a team’s competitiveness lies not in “knowing which model is the best,” but in “owning an evaluation and routing mechanism that can continuously validate and quickly switch.” Benchmarks provide a starting point, POCs provide answers, engineering capabilities dictate user experience, and cost and compliance determine the finish line. Once this process is running smoothly, “which model to choose” is no longer a subjective question, but a reproducible engineering task.