Large Language Model Selection and Inference Service Integration: A Practical Guide
As Chinese open-weight large models now iterate on a monthly or even weekly cadence, one-off selection decisions quickly go stale. This guide lays out a repeatable framework for evaluating models, choosing an inference delivery model, and integrating services without locking into a single vendor. It covers building your own golden evaluation set instead of relying on public leaderboards, comparing hosted APIs with private deployment and hybrid architectures, calculating true per-task cost rather than unit price, designing replaceable integrations with routing and fallback, and adding observability, cost attribution, security, and governance. A weighted scoring rubric and a suggested rollout schedule help engineering teams stay balanced on cost, capability, and stability through frequent iteration.
1. Background: The Model “Shelf” Refreshes Faster, So the Selection Logic Must Change
Over the past two years, the iteration cycle for Chinese large models has shrunk to months or even weeks. A typical pattern is that on mainstream cloud and inference platforms, new models often appear on the “shelf” before any official launch event or marketing campaign—weights are open-sourced and hosted endpoints are already callable, yet the official messaging is not yet consistent. For engineering teams, this means two things:
- Availability is no longer the bottleneck. With open weights and cloud-hosted endpoints running in parallel, a team can complete a real business validation within hours.
- The half-life of selection conclusions is shrinking. A choice that was optimal six months ago may only meet the baseline today. One-off selection is turning into routine, periodic re-evaluation.
So rather than arguing over “which model is strongest,” it is better to build a stable evaluation and integration framework: make model replacement a configuration item, not an architecture overhaul.
2. Separate Three Things First: Models, Inference Services, and Integration Platforms
Many selection debates come from conflating concepts. A clear three-layer breakdown:
| Layer | What It Really Is | What It Determines |
|---|---|---|
| Model | Weights and training recipe | Capability ceiling, knowledge boundary, context length |
| Inference service | Turns a model into a callable endpoint | Throughput, latency, concurrency ceiling, rate-limiting policy, stability |
| Integration platform | Unified entry, routing, quotas, billing, auditing | Migration cost, observability, governance |
Key takeaway: “A strong model” does not equal “an easy-to-use service.” The same model can behave differently across inference services because of quantization precision, batching strategy, cache implementation, and rate-limit thresholds—not the model itself. When evaluating, score these three layers separately.
3. Capability Evaluation: Use Your Own Golden Set Instead of Leaderboard Intuition
Public leaderboards (general-knowledge, code-generation, agent, and web-generation categories, among others) work as an initial screening signal, but they are not suitable as a direct basis for decisions, for three reasons: data contamination is hard to rule out, the question distribution does not match your business distribution, and leaderboards measure capability but not engineering metrics.
The recommended approach is to build your own golden set. It does not need to be large—200 to 500 items covering your real business distribution is enough—tagged along these dimensions:
- Basic understanding and reasoning: multi-hop QA, long-document summarization, and instruction following (especially format constraints and negative constraints).
- Code and tool calling: completion accuracy, function-call argument validity, and multi-turn tool-chain convergence rate.
- Long context: embed a fact in the middle of a long document to test effective recall, not just “can it fit.”
- Structured output: schema compliance rate for JSON/tables—the most easily overlooked hidden cost when integrating into business systems.
- Stability: variance across multiple samplings of the same prompt, which directly affects online experience consistency.
- Chinese and domain terminology: recognition accuracy for industry jargon, abbreviations, and proper nouns.
An evaluation skeleton (pseudocode, omitting the specific SDK):
def evaluate(client, cases, k=3):passed = fmt_ok = tool_ok = 0latencies = []for case in cases:ok_any = Falsefor _ in range(k): # multiple samples to observe stability, not single-shot luckresp = client.chat(case.messages, temperature=case.temperature)latencies.append(resp.latency_ms)if not check_schema(resp, case.schema):continue # format non-compliance counts as a fail, close to real integrationfmt_ok += 1if case.expect_tool and not tool_args_valid(resp, case.tools):continuetool_ok += 1if judge(resp, case.golden):ok_any = Truepassed += int(ok_any)return {"pass@k": passed / len(cases), # capability ceiling"format_pass_rate": fmt_ok / (len(cases) * k), # integration friendliness"tool_call_pass_rate": tool_ok / (len(cases) * k),"p95_latency_ms": percentile(latencies, 95), # engineering metric}
The value of these metrics is that they measure capability and integration cost at the same time. A model with slightly lower accuracy but near-100% format compliance often has a lower total cost than a slightly more accurate model that needs a lot of post-processing correction.
4. Inference Forms: Hosted API, Private Deployment, and Hybrid Architecture
| Form | Best-Fit Scenarios | Main Constraints | What to Watch |
|---|---|---|---|
| Public cloud hosted API | Fast validation, spiky traffic, no GPU ops capability | Data egress, rate-limit quotas, unit price that grows with volume | Rate-limiting policy, protocol compatibility, SLA and fallback plan |
| Private deployment | Sensitive data, steady high concurrency, deep customization needed | Upfront compute investment, ops complexity, lagging model updates | GPU memory and concurrency capacity planning, quantization precision loss, upgrade path |
| Hybrid architecture | Sensitive data processed locally, general tasks on the cloud | Routing rules and consistency governance complexity | Unified gateway, data classification, two-way observability |
Private-deployment capacity planning cannot look only at “does it run,” but at the latency curve under concurrency. Mainstream open-source inference frameworks generally already support continuous batching, paged KV-cache management, weight quantization, and speculative decoding; the impact of these optimizations on throughput usually exceeds the difference in model size. In practice:
- Load-test with the real request-length distribution (input/output token ratio), not with the official recommended config;
- Clearly separate time-to-first-token from total generation latency—the former determines the interactive feel;
- Quantization schemes (INT8/INT4) require a companion precision regression, using the Section 3 evaluation set to verify whether business red lines are crossed.
5. Cost: From “Looking at Unit Price” to “Calculating Per-Task Cost”
Unit price is the easiest metric to obtain and the easiest to be misled by. A formula closer to the financial view:
per-task cost = (input tokens × price_in + output tokens × price_out) / successful taskseffective cost ≈ nominal cost × (1 + retry amplification factor) × (1 − cache hit benefit)
Three often-overlooked multiplier terms:
- Retry amplification: repeated calls triggered by timeouts, rate limits, and format correction can multiply actual consumption; be sure to bring retry rate into monitoring.
- Cache benefit: prefix reuse (system prompt, few-shot, long documents) yields significant gains on services that support prefix caching; putting stable content up front is an important optimization.
- Token structure: output tokens usually cost more than input tokens. Making the model write less fluff and keep structured output lean saves money faster than switching models.
At the capacity level, reserve elasticity rather than keeping peak-sized resources resident: shape offline batch tasks (summarization, tagging, evaluation) through a message queue and isolate them from online interactive traffic, so they do not compete for each other’s quotas.
6. Integration Engineering: Design “Replaceability” into the Architecture
The first principle is not to depend directly on any single vendor’s proprietary SDK. Prefer services compatible with the de-facto industry-standard interface (chat/completions style), and insert an adapter layer between the business and the model:
class LLMClient(Protocol):def chat(self, messages, **kwargs) -> ChatResult: ...def stream(self, messages, **kwargs) -> Iterator[Chunk]: ...def call_with_policy(client, req, policy):for attempt in range(policy.max_attempts):try:return client.chat(req.messages, timeout=policy.timeout)except RateLimited:sleep(backoff(attempt)) # exponential backoff + jitterexcept TransientError:continuereturn fallback(req) # fall back to a smaller model or rule-based reply
Supporting engineering constraints:
- Routing strategy: split traffic to different backends by data-sensitivity level, task type, and cost budget; keep routing rules external as configuration, not hardcoded.
- Streaming output: interactive scenarios must support streaming, while setting both an overall timeout and an idle timeout to prevent hanging connections.
- Fault tolerance and fallback: circuit breaker plus a multi-level fallback chain (strong model → lightweight model → cached result → rule-based reply), so availability is not bound to a single backend.
- Prompt injection protection: treat external input as untrusted data, require secondary authorization for tool calls, and avoid letting model output directly drive high-privilege operations.
7. Observability and Governance: Make Model Calls an Operable Object
Traditional APM cannot directly cover model calls, so you need to fill in four types of data:
- Call logs: sampled request and response traces, token usage, and cache hits, written to a logging service for easy bad-case replay.
- Business metrics: success rate, rate-limit hit rate, retry rate, time-to-first-token and end-to-end P95, and per-task cost—with thresholds configured for monitoring alerts.
- Quality metrics: periodically run offline evaluations with the golden set to track whether version upgrades introduce capability regressions.
- Cost attribution: allocate token consumption by business line, tenant, and feature module, so “total cost went up but no one knows who drove it” does not happen.
If the relevant workloads run on Baidu AI Cloud, you can funnel model calls through a unified API gateway, with logs, metrics, datasets, and artifacts landing in general capabilities such as the logging service, monitoring and alerting, and object storage, while the private-deployment portion runs on a container platform—forming a unified governance view.
8. Security and Compliance Essentials
- Data classification: define which data can leave the boundary and which must be inferred locally; classification rules must be auditable.
- Masking and minimization: complete masking of sensitive fields such as phone numbers, ID numbers, and addresses before sending data to the model.
- Content safety: two-way review of input and output; keep a manual review switch before generated content enters the production pipeline.
- Permissions and quotas: issue separate credentials per caller and configure tenant-level quotas to prevent a single business from exhausting shared resources.
- Traceability: record version numbers (model + prompt + parameters) with each request to ensure issues can be reproduced.
9. Selection Scoring Rubric (Adjust Weights per Business)
| Dimension | Suggested Weight | Scoring Basis |
|---|---|---|
| Business task accuracy | 30% | Golden-set pass@k |
| Integration friendliness | 20% | Format compliance rate, tool-call success rate, protocol compatibility |
| Engineering performance | 15% | P95 latency, concurrency ceiling, rate-limiting policy |
| Cost | 20% | Per-task cost (including retry and cache correction) |
| Stability and compliance | 15% | SLA, data-egress policy, audit capability, upgrade continuity |
10. Rollout Rhythm and Common Pitfalls
Recommended rhythm: in the first week, use a 200-item golden set to compare 2 to 3 candidates side by side and produce a baseline report; in the second week, run a 5% to 10% canary on real traffic, focusing on retry rate and format failure rate; then move into periodic re-evaluation (suggested quarterly, or triggered whenever a candidate model has a major version update).
Five common pitfalls:
- Using leaderboard rankings to replace business validation;
- Comparing only input/output unit prices without counting retry and post-processing costs;
- Using short prompts in the demo stage, then hitting a long-context performance cliff after launch;
- Ignoring rate limits and concurrency quotas—load tests pass but production gets throttled;
- Hardcoding business logic onto a specific model’s proprietary SDK, resulting in extremely high migration cost.
Core conclusion: models will keep being replaced, but the golden set, adapter layer, cost model, and observability system are assets you can reuse for the long term. Treating selection as a pipeline rather than a one-time decision lets the team stay at a reasonable balance point among cost, capability, and stability through frequent iteration.
