How to Choose a Domestic LLM: A Practical Multi-Model Evaluation Guide

The Core of Model Selection: Choosing the Right One, Not the Strongest

The iteration pace of Chinese LLMs has accelerated notably over the past year, with new versions released every few weeks. Faced with a constantly updating list of models, developers often ask: “Which one should I choose?” At first glance, this seems like a question of comparing model capabilities, but it actually involves four independent decision dimensions:

1. Capability: Performance on tasks such as code generation, long-context understanding, mathematical reasoning, and tool calling.
2. Integration: Whether stable inference services exist, compatibility with mainstream API protocols, and the cost of integration.
3. Cost: API pricing, free quotas, and rate limit policies.
4. Experience: Response latency, context window length, and output stability.

Breaking down these dimensions reveals that much “selection anxiety” actually stems from information asymmetry. Developers often focus on leaderboard scores while overlooking how integration methods and cost structures affect real projects. A pragmatic approach is to run all candidate models at low cost, compare them on real tasks, and then decide which to adopt long-term.

This article describes a “unified entry point + parallel multi-model comparison” evaluation method from an engineering perspective, and provides a complete operational path.

Why a Unified Inference Entry Point Matters: The Value of a “Model Supermarket”

In the early days, integrating each model required separate account registration, separate API keys, and custom adaptation to different interface protocols. When you have three to five candidate models, the time cost of this process multiplies quickly. As a result, “model aggregation platforms” emerged. They wrap multiple model services into standardized APIs, allowing developers to access multiple models with a single set of credentials and one address.

These platforms typically share the following characteristics:

• Standardized protocol: Most use request formats compatible with mainstream chat APIs, so clients do not need a custom adaptation layer for each model.
• Unified key management: One API key can invoke all models available on the platform.
• Free or low-barrier trial: Many platforms offer a certain amount of free usage, which is useful for early evaluation.
• Version lag: Aggregator platforms may not immediately reflect the latest versions released by model vendors. Keep this in mind when evaluating.

For the selection scenarios discussed here, the main value of aggregation platforms is not “saving money” but transforming model evaluation from an “integration engineering” task into a “configuration task”. Developers can focus on comparing results rather than handling accounts and keys.

It is worth noting that different platforms vary in how quickly they onboard new models. If a newly released model is not yet available, you can use the previous version of the same family for capability estimation, then supplement with the new version once it is listed. This “use the old version first, supplement with the new version later” strategy is acceptable during evaluation because models in the same family usually maintain stylistic continuity across versions.

Setting Up an Evaluation Environment: Three Steps

Below, we use a “unified inference platform + desktop client” combination as an example. The entire setup can be divided into three parts: account registration, key generation, and client configuration.

1. Registration and Identity Verification

Visit the developer portal of the target inference platform and complete the following steps:

1. Register with your email address and set a password.
2. Enter the verification code from the email.
3. Set your account name to finish basic registration.

An often-overlooked detail: Some platforms require an additional phone number verification after registration. The entry point is usually located in a corner of the page and is easy to miss. If you skip this step, you may be blocked from generating API keys or calling APIs later. Most phone numbers can receive verification codes normally. Before verification, confirm that the country code selected on the page is correct.

2. Generate an API Key

After logging in, go to the key management page in account settings and create a new key:

• Naming: Use purpose-based names like eval-2025 or team-test to make identification easier later.
• Validity period: For evaluation purposes, a long-lived key is acceptable. For production, set a shorter lifespan and rotate keys regularly.
• Save: Keys are usually shown in full only once at creation time. Copy and store them securely in a credential manager immediately.

The key format is typically a fixed prefix plus a random string. This key grants access to all models available on the platform, so treat it as an account-level credential. The risk of leakage is high. Recommended practices:

• Do not hardcode keys into code repositories.
• Use environment variables locally:

export NIM_API_KEY="your-key-here"

• In team settings, issue separate keys per person for easier auditing and revocation.

3. Client Configuration

The criteria for choosing a desktop client are simple: it should support custom API base URLs and parallel multi-model conversations. Configuration steps:

1. In the client settings, find “Model Service” and select the corresponding inference platform type.
2. Fill in three key parameters:

Parameter	Description
API Key	The key string generated in the previous step
API Base URL	The platform’s unified inference endpoint, usually in the form https://<host>/v1
Model ID	The specific model identifier available on the platform

1. Save the configuration and select the model in the chat interface.

About model IDs: Different platforms use different naming conventions, typically a “vendor prefix/model name” format. The full list of available models can be found on the platform’s model catalog page. If a new version is not yet listed, you can configure the previous version of the same family and replace the identifier once the new version becomes available.

4. Verify Connection

After configuration, run a minimal verification to confirm the entire chain works:

import os
from openai import OpenAI
client = OpenAI(
    base_url="https://<host>/v1",
    api_key=os.environ["NIM_API_KEY"],
)
resp = client.chat.completions.create(
    model="<model-id>",
    messages=[{"role": "user", "content": "Explain what attention mechanism is in one sentence."}],
    max_tokens=128,
)
print(resp.choices[0].message.content)

If the response returns normal text, the key, base URL, and model ID are correctly matched. A 401 error suggests checking whether the key was copied completely. A 404 indicates the model ID does not match the platform catalog. A 429 means you have hit a rate limit; reduce concurrency or stagger calls during evaluation.

Horizontal Comparison: Turning Evaluation into a Repeatable Process

Single-model trials only tell you whether a model “works” or not. Real value comes from horizontal comparison. One practical feature of desktop clients is the ability to select multiple models in the same session, have them answer the same question independently, and display the results side by side. This transforms evaluation from “switching models one by one” to “seeing all results at once.”

Designing Comparison Experiments

Asking multiple models the same open-ended question often produces differences that are hard to attribute. A more rigorous approach is to design a structured set of tasks covering typical scenarios:

Task Type	What to Observe	Example
Code Generation	Syntax correctness, edge-case handling	Implement an HTTP request function with retry logic
Code Explanation	Depth of understanding, clarity	Explain the execution order of a piece of concurrent code
Long-Text Summarization	Information retention, structural organization	Create a hierarchical summary of a technical document
Structured Output	Format adherence	Generate output following a specified JSON Schema
Multi-Turn Instruction	Context maintenance	Modify the same configuration across several turns

Prepare 3–5 cases per task and record outputs from each model. Score them using a predefined rubric. Simplify scoring to three dimensions: correctness, completeness, and readability, each on a 1–5 scale. This yields far more reliable conclusions than “this one feels better.”

Using Parallel Conversations

In the client, the typical workflow is:

1. Create a new conversation.
2. Select multiple candidate models in the model picker at the bottom.
3. Set the display layout to horizontal.
4. Enter the question and retrieve answers from all models at once.

The main advantage is that the input is identical, eliminating bias caused by prompt variations. For multi-turn tasks, keep the same group of models across turns and observe differences in context retention.

Common Pitfalls in Evaluation

• Inconsistent temperature settings: Different models use different default sampling parameters. Directly comparing output diversity can be misleading. Explicitly set the same temperature value for all models during evaluation.
• Context length differences: In long-text tasks, some models may truncate input due to smaller context windows. Note this in the comparison.
• Version drift: Models on aggregator platforms may update at any time. Record the evaluation date and model IDs to ensure reproducibility.
• Rate limit interference: Free quotas usually come with concurrency limits. Batch evaluation can easily trigger 429 errors. Add backoff retry logic:

import time
def call_with_retry(fn, retries=5, base_delay=2):
    for i in range(retries):
        try:
            return fn()
        except Exception as e:
            if i == retries - 1:
                raise
            time.sleep(base_delay * (2 ** i))

From Evaluation to Decision: Four Checkpoints

Once comparison experiments are complete, how do you convert results into a decision? Walk through the following items in order:

First, task fit should take priority over overall rankings. Composite leaderboard scores are only reference points. What actually determines the experience is how well the model performs on your task distribution. If your workload is code-heavy, focus on code task scores. If it is long-document processing, focus on summarization and extraction performance.

Second, integration stability matters more than peak capability. A model that scores slightly lower but provides stable service often suits production better than one with higher capability but frequent rate limits. During evaluation, record average latency and failure rates for each model.

Third, cost structure should match your call volume. Free quotas work for the validation phase, but production requires estimating call volume and calculating per-call pricing. If the aggregator platform lags on model updates, also consider whether waiting for a new version fits your release schedule.

Fourth, preserve architectural flexibility for multi-model use. Model iteration is fast. Today’s best choice may be surpassed in three months. It is advisable to add a thin abstraction layer in the application that externalizes model IDs and request parameters as configuration. This keeps switching costs to a one-line config change:

# models.yaml
default: model-a
fallback: model-b
params:
  temperature: 0.3
  max_tokens: 1024

Summary

Selecting Chinese LLMs essentially comes down to an engineering evaluation problem, not a “find the strongest model” exercise. A sound approach is: lower integration costs through a unified inference entry point, run structured horizontal comparisons, then make final decisions based on stability, cost, and update cadence.

The “unified platform + parallel multi-model comparison” method described here offers core value by making the evaluation process repeatable, recordable, and reproducible. As for which model to choose, the answer changes with task types and timing. What matters is not memorizing a conclusion, but establishing a process that can continuously produce sound conclusions.