Choosing Domestic LLMs: An Engineering Framework for Latency, Cost, and Throughput

    Selecting a large language model is a constraint-optimization problem rather than a search for the single “best” model. This guide explains how to define task requirements, standardize latency and token metrics, model cache-aware costs, evaluate throughput, route requests across model tiers, and use offline testing, canary releases, observability, and governance to build a sustainable LLM strategy.

    1. Define the Problem First: Model Selection Is Constraint Optimization

    When discussing which domestic large language model to choose, teams often focus too heavily on benchmark rankings or the experience of a single conversation. From an engineering perspective, model selection is not about finding the absolutely strongest model. It is a constraint-optimization problem involving business SLAs, task quality, budget, compliance requirements, and operational complexity.

    Different tasks require different capabilities:

    • Classification and extraction prioritize format stability and low latency.
    • Retrieval-augmented generation (RAG) depends more on context utilization and factual consistency.
    • Code generation requires syntactic correctness and executable output.
    • Complex reasoning and long-document analysis rely more heavily on a model’s capability ceiling.

    The first step should therefore be defining the task inventory, not listing models. Clarify the task distribution, input and output lengths, peak concurrency, acceptable time to first token and end-to-end latency, retry policies, data sensitivity levels, and quality acceptance criteria.

    Only after these constraints are quantified can measurements such as processing time, token usage, cost units, and throughput be interpreted correctly. Lightweight models are generally faster and less expensive per unit, making them suitable for high-frequency tasks with short, structured outputs. Flagship models usually offer stronger capabilities but come with higher latency and cost, so they are better suited to difficult cases, high-value tasks, or scenarios that require complex reasoning.

    The practical answer is usually not to choose one model exclusively, but to use models in tiers.

    2. Standardize Measurement: From Latency to Tokens per Second

    A typical evaluation record may include model processing time, input tokens, output tokens, total tokens, cost units, cost per total token, and tokens per second. These metrics are useful only when their definitions are consistent.

    Latency should be separated into at least:

    • Time to first token
    • End-to-end latency
    • Queueing time

    Similarly, tokens per second should distinguish output decoding throughput from total processing throughput. Cost per total token is useful for coarse-grained comparisons, but it should not be compared directly across unrelated tasks. Output ratios, cache hit rates, and retry counts can significantly change the actual cost.

    During load testing, keep the following variables consistent:

    • Prompt templates
    • Sampling parameters
    • Concurrency levels
    • Warm and cold cache states
    • Timeout settings

    Record P50, P95, and P99 latency separately. For interactive applications, time to first token has a major impact on user experience. For offline batch processing, total throughput and cost per task are usually more important.

    A basic record structure might look like this:

    1. record = {
    2. "task_type": task,
    3. "input_tokens": count_input,
    4. "output_tokens": count_output,
    5. "cache_create_tokens": count_cache_create,
    6. "cache_hit_tokens": count_cache_hit,
    7. "latency_ms": end_to_end_ms,
    8. "first_token_ms": first_token_ms,
    9. "success": ok,
    10. "retry_count": retries
    11. }

    Only by recording successful tasks, failed retries, and cache states together can teams avoid the situation where a single request appears fast while production costs become difficult to control.

    3. Cost Modeling: Multipliers and Caching Are Hidden Variables

    Large-model costs should not be estimated from total tokens alone. A more general cost model is:

    1. Cost = Input Tokens × Input Price
    2. + Cache Creation Tokens × Cache Creation Price
    3. + Cache Hit Tokens × Cache Hit Price
    4. + Output Tokens × Output Price

    Output pricing is often higher than input pricing. Cache creation may introduce additional charges, while cache hits are usually less expensive. The exact rules depend on the service terms.

    Cache hit rates can have a significant effect on the cost of each task in scenarios involving long system prompts, fixed knowledge prefixes, multi-turn conversations, and retrieval-augmented generation. Comparing only cost per total token may hide the impact of output ratios and cache creation. More business-relevant metrics include:

    • Cost per successful task
    • Cost per unit of quality

    Plans and rate limits must also be included in the cost model. A lightweight plan may allow only several dozen requests per minute, which can be sufficient for low-frequency usage but inadequate for peak concurrency, retry amplification, or batch workloads.

    Rate limiting can cause queueing and timeouts, which may trigger retries. Retries then increase token consumption. Budgets should therefore account for failed retries, cache creation, evaluation traffic, and canary traffic rather than calculating only the tokens used by successful requests.

    4. Performance Modeling: Separate Latency from Throughput

    End-to-end latency can be divided into queueing, preprocessing, first-token generation, decoding, and network transmission.

    Streaming output can improve perceived responsiveness, but it does not reduce total processing time. A high output-token rate does not necessarily mean that time to first token is low. Short-output tasks may be influenced more by scheduling and network overhead, while long-output tasks are more affected by decoding throughput.

    As concurrency increases, throughput and P95 latency often deteriorate at the same time. Load testing must therefore cover multiple concurrency levels instead of measuring only a single request.

    Small models often have advantages in speed and cost for short-output, formatting, and classification tasks. Flagship models are generally more reliable for complex reasoning, long-context processing, and high-risk workloads.

    A practical routing strategy is “lightweight by default, upgrade difficult cases”:

    1. resp = light_model(prompt)
    2. if resp.confidence < threshold or resp.format_invalid:
    3. resp = general_model(prompt)
    4. if still_failed and task_is_critical:
    5. resp = flagship_model(prompt)

    Routing requires confidence evaluation, format validation, timeout circuit breakers, and fallback policies. These controls help prevent the application from becoming dependent on a single model. External interfaces should expose a unified abstraction so that models can be replaced later without changing business code.

    5. Task Tiers and Model Routing

    Tasks can be divided by business value and difficulty:

    • L0: Classification, extraction, and formatting
    • L1: Summarization, rewriting, and RAG-based question answering
    • L2: Code generation and multi-turn tool calls
    • L3: Complex reasoning, long-document analysis, and high-value decisions

    L0 and L1 tasks should generally prioritize lightweight models. L2 tasks should be assigned to general-purpose models according to the required quality threshold. L3 tasks should use flagship models on critical paths, together with human review where appropriate.

    Offline batch workloads can trade some latency for lower cost. Interactive applications should prioritize low time to first token. High-risk tasks should retain auditability and fallback mechanisms.

    6. Evaluation and Canary Releases: Offline, Online, and Shadow Traffic

    Offline evaluation sets should cover the real task distribution, difficult cases, safety requirements, and format compliance.

    During online deployment, use A/B testing, shadow traffic, and canary releases. Monitor:

    • P95 and P99 latency
    • Error rate
    • Rate-limit events
    • Cache hit rate
    • Cost per successful task
    • Human review rate

    Cost per successful task is generally more representative of business performance than cost per token. Regression testing should be completed before release to prevent model-version changes from causing format drift.

    7. Engineering Implementation: Caching, Batching, Streaming, and Observability

    Caching can be divided into exact-match caching and semantic caching. Both require appropriate expiration and consistency handling.

    Batch processing can increase throughput but adds latency. Streaming improves perceived responsiveness. Timeout retries should use backoff and a maximum retry limit. Concurrency pools and priority queues can help protect critical workloads.

    Logs should be anonymized and attributed to tenants, features, and task types. If a team already uses Baidu AI Cloud infrastructure, it can reuse general-purpose capabilities such as containers, logging, monitoring and alerting, and object storage to build evaluation and canary environments. The model-selection methodology itself should remain neutral and verifiable.

    8. Decision Checklist

    The final decision can be consolidated into the following checklist:

    • Is the task distribution clearly defined?
    • Are the SLAs quantified?
    • Can the quality threshold be tested and accepted?
    • Are input and output token patterns measured?
    • Has the cache hit rate been evaluated?
    • Have rate limits and peak concurrency been load-tested?
    • Does the cost model include retries?
    • Are routing and fallback mechanisms available?
    • Are canary releases and monitoring in place?
    • Do compliance and data-residency requirements meet expectations?

    The conclusion is usually not “choose one model.” A more sustainable strategy is to use lightweight models by default, reserve flagship models for difficult cases, reduce costs through caching and routing, and control risk through monitoring and canary releases.

    This approach produces a model strategy that is explainable, operationally manageable, and capable of evolving over time.