Chinese LLM API Integration and Migration: A Practical Guide

1. The Core Issue: Integration Cost Is Not the Same as Call Cost

Many teams evaluate Chinese large language models by first looking at unit prices and benchmark leaderboard scores. After the real implementation starts, however, they often find that the largest cost lies elsewhere. A complete model migration usually consists of four parts:

1. Code changes: differences in request bodies, response bodies, authentication headers, and error-code semantics;
2. Toolchain adaptation: IDE plug-ins, terminal agents, and automation scripts in CI pipelines;
3. Evaluation regression: confirming that the new model has not “silently degraded” in a critical scenario;
4. Operations and observability: re-integrating rate limiting, retries, logging, and quota alerts.

The first item is often severely underestimated. If business code contains dozens of hardcoded endpoint addresses and custom parsing logic, migration becomes a cross-team refactor. Conversely, if the integration layer is protocol-agnostic from the beginning, switching model providers can be reduced to changing a few environment variables. This is the engineering leverage created by protocol compatibility.

2. The Reality of Three Coexisting API Styles

The current large-model service market shows an interesting phenomenon: several interface conventions have effectively become de facto standards. They can be grouped into three categories.

Chat Completions style
This style uses a messages array as its core. Roles are divided into system, user, assistant, and tool. Available functions are declared through the tools field, tool_choice controls the invocation strategy, and when stream: true, the response is returned incrementally through SSE. This is currently the most widely supported form. Most open-source frameworks, agent scaffolds, and SDKs support it natively.

Messages style
This style is designed for agent scenarios. The system prompt is elevated to an independent top-level parameter instead of being treated as a message. Tool definitions, tool execution results, and thinking blocks have explicit structured representations. Streaming events are divided into multiple types, such as message_start, content_block_delta, and message_delta. It is more expressive, but its fields do not map one-to-one to the first style.

Vendor-native styles
These expose proprietary capabilities, such as deep-thinking parameters that explicitly control inference budgets, built-in deep search with retrieval, stricter structured output constraints, and orchestration of multimodal inputs. In a compatibility layer, these capabilities are usually downgraded or dropped.

Major cloud AI platforms in China, such as Baidu AI Cloud, often unify authentication, quotas, request logs, and billing definitions at the model-service gateway layer, and provide multiple access patterns on top of that. The engineering judgment here is: compatibility layers handle “it works”; native APIs handle “it works well”; the two should coexist rather than be treated as either-or.

What a Compatibility Layer Really Maps

A compatibility layer is not just a URL change. At least the following six mappings matter:

Mapping item	Typical difference	Pitfall
Authentication header	Authorization: Bearer vs. x-api-key	401 errors, often with unhelpful messages
Endpoint path	/v1/chat/completions vs. /v1/messages	A 404 is mistaken for a quota problem
System prompt	An item in the message array vs. a top-level field	Changes in prompt weighting can shift output style
Tool calls	function_call vs. tool_use block	Agent loops can break immediately
Streaming events	choices[].delta vs. content_block_delta	Frontends may render blank content or lose characters
Finish reason	Different finish_reason enum sets	A truncated response is treated as normal completion

The last item is especially subtle. If a response is truncated because of length but stored as a normal completion, downstream parsers may crash when they try to parse half-written JSON.

3. Reference Implementation: A Protocol-Agnostic Client Wrapper

The right approach is to converge protocol differences into an adapter layer, while business code depends only on its own abstract interface.

import os, json, time, random, logging
import httpx
log = logging.getLogger("llm.client")
# 1) Endpoints and credentials must come from environment variables; do not hardcode them into the repository.
BASE_URL = os.environ["LLM_BASE_URL"].rstrip("/")
API_KEY = os.environ["LLM_API_KEY"]
MODEL_STRONG = os.environ.get("LLM_MODEL_STRONG", "reasoning-model")
MODEL_FAST = os.environ.get("LLM_MODEL_FAST", "lightweight-model")
RETRYABLE = {408, 409, 429, 500, 502, 503, 504}
def chat(messages, tools=None, model=None, max_tokens=4096,
         temperature=0.2, timeout=120, max_attempts=3):
    """Protocol-agnostic call entry point: the business layer only cares about messages, tools, and text results."""
    payload = {
        "model": model or MODEL_STRONG,
        "messages": messages,
        "max_tokens": max_tokens,
        "temperature": temperature,
    }
    if tools:
        payload["tools"] = tools
        payload["tool_choice"] = "auto"
    delay = 0.5
    for attempt in range(1, max_attempts + 1):
        try:
            resp = httpx.post(
                f"{BASE_URL}/v1/chat/completions",
                headers={"Authorization": f"Bearer {API_KEY}"},
                json=payload, timeout=timeout,
            )
        except (httpx.TimeoutException, httpx.TransportError) as exc:
            if attempt == max_attempts:
                raise
            log.warning("transport error, retry %s: %s", attempt, exc)
            time.sleep(delay + random.random() * 0.3)
            delay *= 2
            continue
        if resp.status_code in RETRYABLE:
            if attempt == max_attempts:
                resp.raise_for_status()
            # Prefer the retry guidance returned by the server when available.
            hint = resp.headers.get("retry-after")
            time.sleep(float(hint) if hint else delay + random.random() * 0.3)
            delay *= 2
            continue
        resp.raise_for_status()
        return normalize(resp.json())
    raise RuntimeError("unreachable")
def normalize(raw):
    """Collapse responses from different protocols into a unified structure and isolate upstream changes."""
    choice = raw["choices"][0]
    msg = choice["message"]
    calls = [
        {"id": c.get("id"), "name": c["function"]["name"],
         "arguments": c["function"].get("arguments") or "{}"}
        for c in (msg.get("tool_calls") or [])
    ]
    return {
        "text": msg.get("content") or "",
        "tool_calls": calls,
        "finish_reason": choice.get("finish_reason"),  # Important: do not discard this.
        "usage": raw.get("usage", {}),
    }

Three details are worth emphasizing:

• Retries must distinguish error types. Retrying authentication failures (401/403) or invalid parameters (400) is meaningless. Only rate limits and server-side errors should trigger backoff retries. Random jitter is also necessary; otherwise, multiple instances may wake up at the same time and create a secondary burst of load.
• finish_reason must be propagated upward. The business layer should use it to determine whether a response is complete, instead of relying on fragile inference such as “can this string be parsed?”
• The adapter should only transform structures, not make business decisions. Once business logic is inserted into the adapter layer, switching models will start to involve business code again.

4. Tool Calls and Streaming Responses: The Two Easiest Points to Get Wrong

Three Common Tool-Call Mismatches

1. Parameter types: Some conventions carry parameters as JSON strings, while others use already-parsed objects. If the same data is repeatedly passed through json.dumps across layers, the result may become an escaped string, causing the tool execution side to fail when parsing.
2. Parallel calls: A single response may return multiple tool-call requests. All of them must be executed in the returned order, and their results must be written back to the same user message by tool_call_id. Filling back only one of them is the most common implementation defect.
3. Call depth: Agent loops need maximum round limits and total token budget limits. Otherwise, the model may keep cycling through “call, fail, retry,” leading to uncontrolled costs.

# Tool execution results must be returned by id, and all results must be returned at once.
def run_tool_round(res, registry):
    tool_msgs = []
    for call in res["tool_calls"]:
        fn = registry[call["name"]]
        try:
            args = json.loads(call["arguments"] or "{}")
            output = fn(**args)
        except Exception as exc:                 # Return tool exceptions instead of interrupting the loop.
            output = f"error: {type(exc).__name__}: {exc}"
        tool_msgs.append({
            "role": "tool",
            "tool_call_id": call["id"],
            "content": str(output)[:20000],       # Prevent oversized output from overwhelming the context.
        })
    return tool_msgs

Streaming Alignment Points

• Incremental fragments may need to be aggregated by index. When text blocks and tool-call blocks appear interleaved, simple string concatenation is not enough.
• End events and error events have different semantics. A network interruption, such as a reset connection, may be retryable, while an explicit server-side error usually is not. These two cases must be handled separately.
• The frontend display and the data written to storage should use the same aggregated result, avoiding a mismatch between “what is seen” and “what is stored.”

5. Build an Internal Evaluation Set Instead of Blindly Trusting Leaderboards

Public benchmarks have reference value, but using them directly for selection decisions is risky: task distributions differ greatly from your own repositories, training-data contamination is possible, and scoring granularity is not uniform, for example, whether unit tests pass versus whether humans approve the result. A more reliable approach is to build a small but high-fidelity internal evaluation set.

How to build one:

1. Extract 50 to 150 real tasks from historical tickets, code review comments, and production bugs, covering your technical stack.
2. Define automatically checkable success criteria for each task, such as “all new unit tests pass,” “compilation succeeds with no errors and linting passes,” or “the generated configuration passes schema validation.”
3. Use stratified sampling: simple completion, single-file modification, cross-file refactoring, and terminal-command tasks should each occupy a reasonable proportion.
4. Run the evaluation set every time a model or prompt template changes, and keep historical results for regression comparison.

Dimensions to evaluate, in order of importance:

• Executable correctness: whether code compiles and tests pass; this is the hardest indicator;
• Repository-level understanding: cross-file dependencies, naming and style consistency, and whether unrelated code is modified without authorization;
• Long-context stability: whether early constraints are forgotten after large files or long conversations;
• Instruction following: whether clearly prohibited operations, such as “do not modify configuration files,” are respected;
• Safety: whether generated commands include destructive operations or introduce known risky dependencies;
• Visibility of failure modes: whether the model admits uncertainty instead of fabricating non-existent APIs.

6. Cost, Performance, and Risk Governance

Understand the billing model clearly. Input tokens, output tokens, cached tokens, and reasoning tokens usually have different unit prices. For reasoning models, the “thinking process” may also be billed. When estimating cost, do not look only at output unit price. Instead, evaluate the end-to-end cost of completing the same task by combining task pass rate, average retries, and average context length. The conclusion is often counterintuitive.

Cost optimization levers, ranked by benefit:

1. Increase prompt-cache hit rate: place stable system prompts and tool definitions at the beginning, and dynamic content at the end;
2. Improve context governance: use retrieval-based injection instead of full concatenation, and summarize or truncate tool outputs;
3. Route by model tier: use lightweight models for classification, extraction, and formatting, and use strong models for complex refactoring and architecture design;
4. Send batch and asynchronous tasks through offline channels to avoid peak-hour rate limits.

# Simple tiered routing: choose the model tier based on task type.
def route(task):
    if task.kind in {"classify", "extract", "format", "summarize"}:
        return MODEL_FAST
    if task.kind in {"refactor", "debug", "review"}:
        return MODEL_STRONG
    return MODEL_FAST

Performance and stability must be load-tested. Key metrics include time to first token (TTFT), which directly affects interactive experience, output throughput, queue time under concurrency, and rate-limit frequency. Load tests should cover both long contexts and large outputs, because many services behave very differently in these two extremes than in short requests.

Risk checklist:

• Vendor lock-in: keep the adapter layer, and externalize model names and endpoints into configuration;
• Canary releases and rollback: roll out a new model with a small traffic percentage first, compare online results using the same evaluation set, and keep a one-click rollback switch;
• Data compliance: define which data can leave the internal network, which data must be masked, how long logs are retained, and whether logs may be used for service improvement; cross-border transfer requires compliance assessment;
• Observability: connect call logs, token consumption, error rates, and latency distributions to unified logging, monitoring, and alerting systems, and split dashboards by model dimension;
• Key management: credentials should go through a secret management service rather than being written as plaintext environment variables into images; establish rotation mechanisms and least-privilege policies.

7. Deployment Checklist

• Are endpoints, model names, and credentials 100% externalized into configuration?
• Does the adapter layer isolate protocol differences so that business code does not need to be aware of them?
• Is finish_reason propagated upward and used to drive truncation handling?
• Do tool calls support parallelism, return all results by id, and enforce round and budget limits?
• Is the streaming aggregation result consistent with what is written to storage?
• Is there an internal evaluation set with at least 50 tasks that can be run automatically?
• Is billing evaluated by end-to-end task cost rather than unit price?
• Have long-context and high-concurrency load tests been completed?
• Are canary release, monitoring, alerting, and rollback capabilities in place?
• Has the data masking and retention policy been confirmed for compliance?

Conclusion: The difficulty of integrating Chinese large language models has never been “can it be called?” but “can it evolve at low cost after it can be called?” Converge protocol differences in the adapter layer, base quality judgment on an internal evaluation set, and switch the cost measure from unit price to end-to-end task cost. If these three things are done correctly, a model provider becomes less of a strategic decision requiring formal project review and more of a routine operations task that only requires changing a few lines of configuration.