Evaluating Domestic Large Language Models: A Practical, Repeatable Selection Framework

Why LLM Selection Feels Harder Than It Should

If you have followed the LLM space over the past year, you have likely noticed a pattern: new models are released at a relentless pace, with leaderboards for general capability, coding, agentic behavior, and long-context tasks appearing constantly. Each vendor’s marketing claims look impressive on paper, but when you run the same model against your real workloads, results are often inconsistent. Three root causes explain this difficulty:

1. Uneven capability distribution. No single model is best at everything. One model may excel at code generation, another at long-document comprehension, another at stable tool calling for agents, and yet another at nuanced writing. Comparing models without defining the target task is meaningless.
2. Benchmark-to-experience gap. Leaderboard scores are heavily influenced by the evaluation set and scoring methodology. A model may be fine-tuned or optimized for common public benchmarks and score high, yet perform mediocrely on your actual request distribution.
3. Hidden switching costs. Switching models is not just changing an API endpoint. You have to re-engineer prompts, re-validate output formats, reassess latency and cost structures, and account for differences in context windows that may break existing business logic.

Therefore, scientific selection is not about finding “the strongest model,” but rather about making decisions based on reproducible evaluation data across a candidate set that satisfies your business constraints. The rest of this article shows you how.

Understanding the “Model Supermarket”: The Standard Shape of Inference Platforms

Over the past few years, mainstream cloud platforms have widely offered a capability commonly referred to as “model hosting and inference microservices.” You can think of it as a “model supermarket”: the platform packages models as standardized services, deploys them on a shared compute cluster, and lets developers call them on demand. Early catalogs were dominated by open-source models from overseas. Today, domestically-developed models are increasingly well-represented, and many newly released models are available on these platforms quickly.

For developers, the value of such platforms falls into four categories:

• Standardized interfaces. Regardless of which model sits underneath, calls are handled through a unified HTTP + JSON chat-completion interface. Authentication and parameter structure are essentially the same, keeping the cost of switching models low.
• Zero-ops. You do not need to build your own inference cluster, manage GPU selection, or handle autoscaling. Billing is based on token usage.
• Integrated engineering tooling. Monitoring, alerting, logging, rate limiting, and circuit-breaking are typically provided by the platform. You can combine them with container platforms and object storage to compose a full production pipeline.
• Flexible deployment options. In addition to public cloud APIs, many platforms support deploying models as inference microservices in a privatized or VPC environment to meet data-residency and compliance requirements.

Take Baidu AI Cloud or similar domestic cloud platforms as an example: their model-serving products generally provide unified inference endpoints, pay-as-you-go pricing, and call analytics. Developers can run head-to-head multi-model comparisons within the same technical stack—which is the ideal infrastructure for selection experiments. A critical caveat: hosting platforms solve the “how to call” problem, not the “which model to call” problem. Model selection still requires evaluation.

First Principles of Selection: Define the Task Before You Pick a Model

Many teams fail at selection because they start by looking at leaderboards rather than defining the problem. Start by writing down a requirements table that captures the following dimensions:

Dimension	Key Question	Example
Task type	What will the model primarily do?	Customer support Q&A / Code generation / Document summarization / Structured extraction / Agent tool calling
Input/output format	How long is context? Is output format constrained?	Average input 8K tokens; output must be valid JSON
Latency constraints	What are the first-token and total-time limits?	First token < 1.5s; total < 8s
Cost budget	What is the acceptable cost per call and monthly total?	< ¥0.05 per call; monthly budget ¥20,000
Compliance constraints	Can data leave your domain? Is private deployment required?	User data cannot leave the enterprise; private deployment required
Concurrency scale	What is the peak QPS?	Peak 200 QPS

This table will immediately eliminate a large number of unqualified candidates. For instance, if private deployment is required, cloud-only API options are ruled out. If stable JSON output is required, you focus on format-compliance rate rather than knowledge quiz scores. Constraints first, capability second—this is the first principle of selection.

How to Read Leaderboards: Helpful, but Not Decisive

Public leaderboards still have value, but only when used correctly. The proper use of a leaderboard is as a pre-filter, not a decision-maker. Look at sub-rankings relevant to your task (for example, if you are building front-end code generation, focus on coding-specific leaderboards). Shortlist 3–5 candidates, then move to your own evaluation phase.

Be aware of these common leaderboard limitations:

1. Data contamination. Some benchmark questions may have been included in the model’s training data, inflating scores.
2. Distribution mismatch. The capabilities tested by a benchmark may differ completely from your real-world request distribution.
3. Scoring methodology differences. The same task can produce different rankings depending on whether scoring is rule-based or model-based.
4. Update lag. Leaderboards have a cycle between benchmark revision and re-scoring. A newly released model’s capability changes may not be reflected for weeks.

Use leaderboards to narrow your scope, but base your final decision on evaluation data from your own workloads.

Build a Lightweight Evaluation Harness: Let the Data Decide

A self-built evaluation does not need to be heavyweight. A minimal viable approach has four steps:

1. Construct an evaluation set. Sample 100–300 real requests from your business logs (with sensitive data redacted), covering high-frequency scenarios and edge cases: overly long inputs, adversarial prompts, strict formatting demands, and so on.
2. Define metrics. Track pass rate (whether the answer is correct or contains key required elements), format-compliance rate (e.g., the proportion of responses that parse as valid JSON), average latency, and unit cost per request.
3. Control variables. Use the same prompts, temperature, and maximum output length for all candidate models. Run multiple rounds and take averages to reduce variance.
4. Analyze failures. Beyond total scores, examine the distribution of failed cases. Are they knowledge errors, format violations, or refusal behaviors? This analysis determines whether you need to adjust prompts or change models.

Below is a minimal evaluation script example, written against the standard chat-completions API format. Replace the endpoint and key with your own platform’s values.

import json, time, urllib.request
API_URL = "https://your-inference-endpoint/v1/chat/completions"
API_KEY = "your-api-key"
def run_case(case: dict, model: str):
    payload = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": case["question"]}],
        "temperature": 0.2,
        "max_tokens": 1024,
    }).encode("utf-8")
    req = urllib.request.Request(
        API_URL,
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_KEY}",
        },
        method="POST",
    )
    start = time.time()
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode("utf-8"))
    latency = time.time() - start
    answer = data["choices"][0]["message"]["content"]
    return answer, latency
def check(answer: str, case: dict) -> bool:
    for kw in case.get("must_include", []):
        if kw not in answer:
            return False
    for kw in case.get("must_exclude", []):
        if kw in answer:
            return False
    if case.get("format") == "json":
        try:
            json.loads(answer)
        except Exception:
            return False
    return True
def evaluate(model: str, cases_path="cases.json"):
    cases = json.load(open(cases_path, encoding="utf-8"))
    passed, total, latencies = 0, 0, []
    for case in cases:
        try:
            answer, latency = run_case(case, model)
        except Exception:
            continue
        total += 1
        latencies.append(latency)
        if check(answer, case):
            passed += 1
    avg = sum(latencies) / max(len(latencies), 1)
    print(f"model={model} pass_rate={passed}/{total} avg_latency={avg:.2f}s")
if __name__ == "__main__":
    evaluate("model-a")
    evaluate("model-b")

For subjective tasks that rule-based checks cannot cover (e.g., copywriting quality), sample a subset for manual review, or use a fixed “model-as-judge” approach. If you use an LLM judge, keep it constant across runs and retain a portion of human checks to calibrate for judge bias.

Cost and Engineering: The Half of the Battle Everyone Ignores

Passing the evaluation is only the first half. The details that determine real production experience are engineering concerns:

• Cost structure. Input and output tokens are usually billed separately. In long-context scenarios, input costs often dominate. Trimming context, compressing conversation histories, and using platform-provided caching features can meaningfully reduce spend.
• Concurrency and rate limits. Load-test at peak QPS before launch. Confirm your platform’s rate-limiting policy and queueing behavior to avoid large-scale timeouts during traffic spikes.
• Model routing and fallback. Routing different requests to different models—small models for simple tasks, large models for complex ones—is a common cost optimization. Also configure fallback models so that if the primary path fails, traffic auto-degrades gracefully.
• Prompt regression. Different models respond very differently to the same prompt text. After switching models, you must re-run regression tests across all your prompts. Do not assume the effects will transfer.
• Compliance and privatization. For scenarios involving sensitive data, prioritize inference microservices that support private deployment. Deploy within your enterprise’s internal container platform, and pair it with logging and monitoring to form a closed loop.

From POC to Full Rollout: A Reusable Landing Process

Chaining all the preceding steps together yields a reusable selection process:

1. Requirements definition. Complete the task-type and constraints table.
2. Leaderboard screening. Shortlist 3–5 candidates.
3. Self-built evaluation. Run unified test cases with fixed parameters to collect pass rate, latency, and cost as three hard numbers.
4. Small-traffic gray rollout. Run A/B comparison with 5%–10% of traffic and observe real business metrics.
5. Full switch. Roll out to full traffic only if there is no regression. Always retain a one-click rollback capability.
6. Continuous monitoring. Track pass rate, manual correction rate, latency, and cost through logging and alerting. When a candidate model releases a new version, re-run the evaluation set. This forms a closed loop: evaluate → deploy → monitor → re-evaluate.

Conclusion

LLM selection has no one-and-done answer. Model capabilities are evolving rapidly, and business requirements change over time. What is optimal today may not be optimal in six months. The truly sustainable approach is to turn selection from a gut feeling into an engineering mechanism: constraints first, leaderboard screening, self-built evaluation, gray validation, and continuous monitoring. Once this mechanism is in place, no matter how the model market shifts, your team can complete a safe model upgrade within weeks—not months.