Choosing and Implementing LLM APIs: A Practical Guide for Developers in China

    Selecting an LLM API is not only about model quality. Payment options, service availability, authentication complexity, compliance, and operational reliability all determine whether a project can move from a proof of concept to production. This guide compares three integration paths, outlines key evaluation criteria, and provides a reusable Python gateway pattern for configuration management, retries, failover, and secret handling.

    Before integrating an LLM API, remember that model capability is only one part of the system. The integration path determines whether the project can actually reach production. Even if the underlying model is effective, payment problems, poor key management, or frequent connection failures can keep the project stuck at the validation stage.

    This guide examines the main engineering concerns for developers in China integrating LLM APIs and presents a practical selection framework and reusable code structure.

    Three Questions to Answer Before Integration

    1. How will the service be paid for?

    Payment options should match the company’s accounting and procurement processes. Individual accounts may only need a bank card, while enterprises often require corporate bank transfers, specific invoice types, and internal budget approval.

    If the payment process does not work, subsequent engineering efforts may be blocked. Confirm acceptable payment and invoicing methods with the finance team during the provider evaluation stage.

    2. Could the service become unavailable?

    Service availability should not be assessed solely by the figures shown on a marketing page. Also review:

    • Availability commitments in the service agreement
    • Incident notification procedures
    • Historical service incidents
    • Whether the service provider’s legal entity and operational information are clearly documented

    3. How complicated is authentication?

    Authentication methods vary significantly between platforms. Some providers require only an API key, while others use request signatures, temporary credentials, subaccounts, or role-based permissions.

    The more complex the authentication process, the higher the integration and troubleshooting costs. Configuration errors can also increase the risk of security incidents.

    Three Integration Paths and Their Use Cases

    1. Third-Party Aggregation Gateways

    An aggregation gateway provides a unified entry point for calling multiple models. After integrating once, a team can switch between supported models with relatively little code change.

    The main advantages include:

    • A unified protocol
    • Broad model coverage
    • Low switching costs

    The main risk is the additional intermediary. Teams must evaluate its data-retention policy, reliability, and billing transparency.

    This approach is suitable for teams that need to compare multiple models quickly and whose data has a moderate level of sensitivity.

    2. Managed Services from Overseas Cloud Providers

    Large cloud providers offer managed model services that are deeply integrated with their account systems, networks, and access-control tools.

    The main advantages are generally more mature enterprise compliance documentation and service-level frameworks. However, account activation, cross-border network connectivity, authentication, and payment can all be relatively complex.

    This option is more suitable for large enterprises that already operate overseas infrastructure and are familiar with the provider’s cloud-native toolchain.

    3. Domestic Cloud Platforms

    Domestic cloud platforms operate services within China and typically support local settlement and invoicing. They may also make it easier to satisfy requirements related to keeping data within the country.

    For example, Baidu AI Cloud provides large-model service capabilities. In general, domestic cloud platforms may be a better fit for businesses with strict compliance constraints.

    During evaluation, confirm the specific available models, service regions, and billing terms. Always rely on the provider’s official documentation for the current details.

    Decision Criteria for Provider Selection

    Do not compare providers based only on unit price. Evaluate them across the following dimensions:

    • Compliance requirements: Can data leave the country, or must the service be deployed domestically?
    • Model requirements: Does the business depend on a specific closed-source model, or would an open-source or domestic model be sufficient?
    • Settlement and finance: Do the payment method, invoice type, and currency match the company’s requirements?
    • Network conditions: What are the latency and connectivity characteristics when accessing the endpoint from production infrastructure or office networks?
    • Reliability: What availability commitments, rate limits, and recovery times are provided?
    • Migration cost: How compatible is the API structure with the existing codebase, and how much business logic would need to change during migration?

    After defining the evaluation criteria, run the same test cases continuously for one week against two or three providers. Record the success rate, P95 latency, timeout count, and actual bill.

    Measured results are more reliable than marketing claims.

    Engineering Practice: Isolate Provider Differences Behind One Layer

    Business code should not contain scattered provider-specific implementation details. A better approach is to add a unified wrapper between the application and the model services.

    Define common request and response structures internally, and move the following values into configuration:

    • Service endpoints
    • API keys
    • Model names
    • Timeout settings
    • Retry policies

    With this structure, switching providers, performing a gradual rollout, or enabling a fallback requires configuration changes rather than changes to business logic.

    The following simplified example uses placeholder endpoint paths. Replace them according to the relevant provider’s API documentation:

    1. import os
    2. import time
    3. import requests
    4. class LLMGateway:
    5. """A unified wrapper that hides provider differences from the business layer."""
    6. def __init__(self, providers):
    7. # Providers are ordered by priority; the first one is called first.
    8. self.providers = providers
    9. def chat(self, messages, max_retries=2, timeout=(5, 60)):
    10. last_error = None
    11. for provider in self.providers:
    12. for attempt in range(max_retries + 1):
    13. try:
    14. resp = requests.post(
    15. provider["base_url"] + "/chat", # Placeholder path
    16. headers={
    17. "Authorization": f"Bearer {provider['api_key']}"
    18. },
    19. json={
    20. "model": provider["model"],
    21. "messages": messages,
    22. },
    23. timeout=timeout, # (connection timeout, read timeout)
    24. )
    25. if resp.status_code == 429 or resp.status_code >= 500:
    26. time.sleep(2 ** attempt)
    27. continue
    28. resp.raise_for_status()
    29. return resp.json()
    30. except requests.RequestException as exc:
    31. last_error = exc
    32. time.sleep(2 ** attempt)
    33. # Switch to the next provider after retries are exhausted.
    34. raise RuntimeError(
    35. f"All provider calls failed: {last_error}"
    36. )
    37. def load_providers():
    38. # Load all keys and endpoints from environment variables.
    39. # No credentials are included in the source code.
    40. return [
    41. {
    42. "base_url": os.environ["PRIMARY_BASE_URL"],
    43. "api_key": os.environ["PRIMARY_API_KEY"],
    44. "model": os.environ["PRIMARY_MODEL"],
    45. },
    46. {
    47. "base_url": os.environ["BACKUP_BASE_URL"],
    48. "api_key": os.environ["BACKUP_API_KEY"],
    49. "model": os.environ["BACKUP_MODEL"],
    50. },
    51. ]

    This design has three important characteristics.

    First, configuration is externalized, so changing providers does not require a new application release.

    Second, connection and read timeouts are configured separately. Long-form generation may require a longer read timeout, while the connection timeout should generally remain short.

    Third, the client automatically switches to a backup provider after repeated failures, reducing the impact of a single-provider outage.

    Key and Security Management

    • Store keys only in environment variables or a dedicated secrets-management service. Do not commit them to the source repository or print them in logs.
    • Create separate keys for different environments and projects. This makes troubleshooting easier and allows individual keys to be revoked independently.
    • Rotate keys regularly. If a key is exposed, revoke it immediately and review usage records from the exposure period.
    • Configure usage limits or alert thresholds to prevent abnormal requests from causing uncontrolled costs.
    • Redact user input and model output in logs. Record only necessary metadata, such as the request ID, model name, duration, token usage, and status code.

    Reliability Design: Timeouts, Retries, and Fallbacks

    LLM requests usually take much longer than ordinary API calls. Long-form generation and multi-step reasoning can take even more time. The following practices have a direct impact on production stability:

    1. Configure connection and read timeouts separately. The connection timeout is usually kept within a few seconds, while the read timeout should reflect the maximum expected generation time for the task.

    2. When using streaming output, monitor the idle time between consecutive chunks. If no new data arrives for an extended period, the connection may have failed. In that case, actively close the connection and retry instead of waiting indefinitely.

    3. Use exponential backoff for rate-limit responses (429) and server-side errors (5xx). Add random jitter to prevent a large number of requests from retrying at the same time and creating a secondary traffic spike.

    4. Retrying conversational requests is generally lower risk. For tasks involving billing, database writes, or external actions, retries require additional safeguards at the business-logic layer.