Choosing Domestic LLMs: Selection Dimensions and Unified Access Practices

1. Why LLM Selection Has Become Harder: Supply Explosion and Fragmented Access

Over the past year, the iteration pace of domestic large language models (LLMs) has accelerated significantly. Leading vendors release new flagship models almost every quarter, and the open-source community continuously introduces cost-effective small-to-medium models. While having more options is generally good for developers, three pain points currently overlap:

1. Opaque capability differences: Vendors present impressive benchmark scores at their launch events, but business scenarios vary widely. High benchmark scores do not necessarily equate to good performance in your specific use case.
2. Fragmented access methods: Each platform has its own console, authentication method, and SDK style. If you want to horizontally compare three to five models, simply registering accounts, applying for API keys, and modifying code requires repetitive manual effort.
3. High trial-and-error costs: Official API calls are billed per token, making large-scale blind testing expensive. Conversely, drawing conclusions based on one or two chat interactions easily leads to overgeneralization.

A more pragmatic approach is to first establish a reproducible selection methodology, and then leverage a “model supermarket” style unified inference platform to reduce integration costs, transforming model selection from a one-off decision into a continuous engineering practice.

2. Four Core Dimensions for Selection

Here is an overview table before diving into the details:

Dimension	Focus Area	Common Pitfalls
Capability Match	Code, Agent, Long context, Instruction following	Focusing only on general leaderboards, ignoring scenario weights
Cost & Context	Token unit price, Window length, Max output	Assuming long context is “free”
Engineering Traits	Rate limits, Concurrency, Latency, SLA	Inferring production readiness from personal testing
Compliance & Security	Data retention, Logging, Private deployment	Retrofitting compliance after going live

1. Capability Match: Scenario First, Leaderboard Second
General conversation, code generation, Agent tool calling, and long document understanding are the four most common requirements, and their corresponding capability weights differ entirely. For R&D efficiency tools, code completion and multi-turn refactoring capabilities are hard metrics; for customer service bots, instruction following and hallucination control are more important; for document analysis assistants, long context and structured output stability take priority. The first step in selection is to translate business requirements into a capability weight table.

2. Cost & Context: Calculate the Real Bill
The input/output token unit price, context window length, and maximum output per request jointly determine the actual cost. Note specifically: long context does not mean cheap. Stuffing hundreds of thousands of words of documents into the context might incur input token fees far exceeding expectations. You should evaluate the cost curves of the “Retrieval-Augmented Generation (RAG) + short context” route versus the “direct long context reading” route before finalizing the technical architecture.

3. Engineering Traits: Rate Limits, Concurrency, and Latency
The stability requirements for personal toy projects and professional production environments are worlds apart. Free or low-cost tiers usually have strict rate limits, and queuing or throttling during peak hours is the norm. During selection, scrutinize the Requests Per Minute (RPM), Tokens Per Minute (TPM) quotas, and whether enterprise-grade guaranteed channels are available.

4. Compliance & Data Security
When dealing with user privacy or sensitive enterprise data, verify the vendor’s data retention and logging compliance policies. If necessary, consider private deployment or choose LLM services provided by cloud vendors (such as the general form of LLM service platforms provided by Baidu AI Cloud, which typically come bundled with logging services, monitoring alerts, and data management capabilities) to incorporate compliance responsibilities into unified platform-side governance.

3. Utilize Public Leaderboards, but Do Not Be Held Hostage by Them

There are many public evaluation leaderboards oriented towards web development, code generation, and other directions in the industry. Their value lies in providing a rough “first-tier” benchmark—telling you which models are worth entering your candidate pool. However, keep three points in mind:

1. The leaderboard’s prompt distribution may not match your business distribution; high scores might represent “misplaced excellence.”
2. Leaderboard updates lag behind model iterations. Two models with similar version numbers might have already reversed their actual performance gap.
3. There is a risk of evaluation datasets leaking into training data, inflating the scores.

A more reliable approach is to build a small, custom evaluation set: extract dozens to hundreds of typical requests from real business scenarios, define scoring rules (keyword hits, structured field validation, human blind evaluation), and run a round of automated comparisons:

cases = [
    {"q": "Explain common scenarios where relational database indexes fail", "must_include": ["leftmost prefix", "implicit type conversion"]},
    {"q": "Parse the following logs into JSON and count the error code distribution", "must_include": ["error_code"]},
]
def evaluate(model_id: str) -> float:
    passed = 0
    for c in cases:
        answer = call_llm(model_id, c["q"])
        if all(k in answer for k in c["must_include"]):
            passed += 1
    return passed / len(cases)

Even with just a few dozen test cases, this method reflects the real performance gap better than simply looking at leaderboards.

4. Unified Inference Platforms: Turning the “Model Shelf” into Infrastructure

To solve API fragmentation, the industry has broadly adopted “model supermarket” style unified inference platforms. These platforms host multiple models behind a unified inference service, exposing an API compatible with general industry chat protocols. Their value includes:

1. Single integration, multiple models available: Register one account and generate one API key to access all models on the shelf, drastically reducing horizontal comparison costs.
2. Near-zero switching costs: With a unified protocol, switching models typically requires changing only the model field, leaving the code structure untouched.
3. Easy gray release and A/B testing: When a new model goes live, traffic can be split between the old and new models by ratio to validate effectiveness using real traffic.

Of course, these platforms have practical constraints: when popular models are first listed, resources are tight, which may lead to queuing, rate limiting, or intermittent timeouts. The synchronous listing of the latest model versions may also lag. This dictates that engineering must be paired with retry and fallback mechanisms (see Section 7).

5. General Access Workflow: From Registration to First Chat

While console details vary across platforms, the core workflow is highly consistent and can be abstracted into four steps:

1. Registration and Verification: Email registration, setting a password, and entering a verification code are standard. Some platforms also require phone number verification. A frequent pitfall exists here: after registration, there might be a separate “verification/authentication” portal in the console. If you do not click it, subsequent key generation or API calls will be blocked. Ensure your account status is fully active.
2. Generate API Key: Navigate to the console’s key management page, where you are typically asked to fill in a name and expiration time. Naming should follow a “purpose-environment-owner” format, such as chatbot-prod-zhang. For personal debugging, setting a longer validity period is fine, but for team collaboration, setting a rotation cycle is strongly recommended. Save the key immediately upon generation, as most platforms only display the full key at the moment of creation.
3. Record Two Key Pieces of Information: First, the API Base URL (note whether it includes a version path). Second, the Model ID, usually in a “vendor/model-name-version” format, which must exactly match the platform’s model list; a typo will directly result in a “model not found” error.
4. Configure the Client: For graphical interface enthusiasts, choose a general desktop client that supports custom model services, and input the three key elements: API key, endpoint URL, and model ID. Command-line users can directly send an HTTP request for a smoke test:

POST /v1/chat/completions HTTP/1.1
Host: api.example-platform.com
Authorization: Bearer $API_KEY
Content-Type: application/json
{"model": "vendor/model-id", "messages": [{"role": "user", "content": "Hello"}]}

If a standard JSON structure is returned, it means authentication, model ID, and network connectivity are all working.

6. API Key Security: The Most Overlooked Aspect

1. An API key is equivalent to account balance; if leaked, it will be drained. Never hardcode it into source code. Injecting it via environment variables or a secret management service is recommended.
2. Do not commit configuration files containing keys to code repositories; add them to the version control ignore list. If a leak occurs in historical commits, revoke and regenerate the key immediately.
3. Be aware of the risk of accidental exposure via automatic cloud syncing for keys stored in desktop client configuration files.
4. In team environments, issue different keys for different purposes so that issues can be traced and individually revoked based on the specific key.

The standard implementation in Python:

import os
client = LLMClient(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.example-platform.com/v1"),
)

7. Multi-Model Engineering: Routing, Comparison, and Fallback

When there is more than one candidate model, it is recommended to treat “model selection” as a configuration rather than hardcoded logic:

1. Scenario-based routing: Route long document analysis to long-context models, coding tasks to strong programming models, and summarization or chitchat to cost-effective models.
2. Unified abstraction layer: Converge the Base URL, Model ID, timeout, and retry logic into a single client factory. Business code should only depend on the abstract interface.
3. Failure fallback chain: Automatically switch to a backup model upon rate limiting or timeout to avoid single points of failure.

FALLBACK_CHAIN = ["model-a", "model-b", "model-c"]
def chat(messages, timeout=30):
    last_err = None
    for model_id in FALLBACK_CHAIN:
        for _ in range(2):                # Max 2 retries per model
            try:
                return client.chat.completions.create(
                    model=model_id, messages=messages, timeout=timeout
                )
            except (RateLimitError, TimeoutError) as e:
                last_err = e
                continue
    raise RuntimeError(f"All candidate models unavailable: {last_err}")

This structure has an added benefit: when a new model is released, you can append it to the end of the chain or insert it at a comparison position. A gray release validation can be completed by changing just one line of configuration.

8. Common Pitfalls Checklist

1. Account phone verification or identity authentication is incomplete, causing API key calls to be rejected.
2. Model ID spelling does not match the platform, resulting in a “model not found” error. When a new model version is not yet listed, use the existing platform version and make the Model ID a configuration item so you can switch with one line once it is listed.
3. Free tier rate limits are strict. Evaluate quotas before stress testing or batch tasks, and request quota increases if necessary.
4. Long context request fees exceed expectations. Compare the cost of “chunking + retrieval” first before deciding whether to read the full text directly.
5. Ignoring temperature, max_tokens, and other parameter tuning. The same model performs significantly differently under different parameters; fix a parameter baseline during evaluation.
6. Accidentally committing keys to a code repository. Revoke and regenerate immediately and clean up the git history.

9. Conclusion

There is no one-size-fits-all answer to LLM selection. The industry’s iteration speed dictates that the half-life of the “optimal model” might only be a few months. The truly sustainable approach is: use unified protocols to turn models into pluggable components, use custom evaluation sets to continuously calibrate selection judgments, and use routing and fallback mechanisms to guarantee service stability. When the cost of switching models approaches zero, you no longer need to agonize over “which one to choose”—you can always use “the most suitable one right now.”