LLM Selection and Traffic Governance in the Agent Era

    Token-call volume is often used as a public signal of LLM adoption, but it can be misleading when teams evaluate models or plan production architecture. This article explains what call-volume data can and cannot show, then outlines a practical approach to model selection and traffic governance in multi-model, Agent-driven systems. It covers token metrics, task-model evaluation, unified model gateways, fallback and routing strategies, observability, cost governance, security, and selection checklists.

    1. Background: What a Weekly Call-Volume Report Reveals

    A third-party LLM aggregation and scheduling platform recently published weekly statistics. In the reported week, total global AI LLM call volume reached about 210 trillion tokens, up about 8.25% week over week. Call volume for models from Chinese vendors reached about 116.18 trillion tokens, up about 16.71%. In the same period, call volume for models from US vendors was about 13.98 trillion tokens, down about 32.4%. For 28 consecutive weeks, call volume for LLMs from Chinese vendors has been higher than that for models from US vendors, and it has continued to lead globally.

    Among the top five single models, four were from Chinese vendors. The top model handled about 56 trillion tokens in a week, roughly four times the volume of the leading US model in the same period. The other three models in the top four were also Chinese models, while the fifth was the main model from a leading US vendor.

    Earlier, the same platform reported that in the week of March 9–15, API call volume for models from Chinese vendors reached about 4.12 trillion tokens, exceeding about 2.94 trillion tokens for models from US vendors for the first time. Since then, LLMs from Chinese vendors have continued to maintain the lead.

    These figures only count API traffic forwarded through that aggregation platform. Traffic from vendor-owned APIs, cloud-hosted services, private deployments, and consumer-facing web or client applications is not included. Therefore, the data should not be read directly as global market share.

    For developers and technical managers, the data sends at least three signals:

    • Call volume is growing rapidly: global weekly calls have reached the level of hundreds of trillions of tokens and are still increasing.
    • Model supply is increasingly diverse: multiple vendors’ models are online at the same time, lowering the cost of switching and combining them.
    • Public rankings cannot fully reflect the industry: limited statistical scope means they are references, not conclusions.

    This article does not argue about “who won.” Instead, it focuses on two practical questions:

    1. How can teams evaluate and select models objectively?
    2. In a multi-model environment, how can call volume be controlled, measured, and used well?

    2. Understanding the Metrics: What Call Volume Can and Cannot Tell You

    2.1 Why Token Became the Core Unit

    Token is the basic unit for processing text in large language models. Models read input by tokens and generate output by tokens. The length of text represented by one token depends on the tokenizer. Under common tokenization schemes, one English token may correspond to part of a word, while one Chinese token may correspond to one or more Chinese characters. The exact ratio depends on the tokenizer used by the model.

    Industry conventions around tokens include:

    • Billing: input and output tokens are usually billed separately, with output tokens often priced higher than input tokens.
    • Capacity: the context window is measured in tokens and determines how much material a single request can carry.
    • Rate limiting: many services use tokens per minute (TPM) as a rate-limiting dimension.

    Because tokens connect cost, capacity, and stability, call volume has become one of the most direct public indicators for observing the LLM industry.

    2.2 Three Limitations of Call-Volume Metrics

    Limitation 1: Limited coverage

    This type of platform statistics only includes API traffic forwarded through the platform. Vendor-owned APIs, cloud platform managed services, private deployments, and consumer-facing web or client traffic are not included. The platform’s user mix, campaign cadence, and billing rules can also directly affect the numbers. Therefore, this metric is suitable for observing “call trends flowing through aggregation channels,” but should not be interpreted as “overall market share.”

    Limitation 2: Call volume is not the same as business value

    Dimension Call volume Business value
    Unit Tokens (input/output) Completed tasks, conversions, saved labor hours
    Influencing factors Context length, whether a reasoning model is used, cache hit rate, retries Business process design, model-task fit, user experience
    Meaning of growth More consumption; may also indicate lower efficiency Whether output per unit cost improves
    Use cases Trend monitoring, capacity planning, cost attribution Model selection, investment decisions

    For example, an agent application may require multiple reasoning steps and tool calls for one completed task. Both call volume and token consumption can rise significantly, but delivered business outcomes may not increase proportionally.

    Reasoning models also generate many internal reasoning tokens before producing a final answer, and these tokens count toward consumption. Switching the same business traffic to a reasoning model can increase token usage by several times.

    Weekly data can be affected by:

    • holidays and weekends;
    • concentrated trials after new model launches;
    • limited-time free quotas, promotions, and traffic subsidies;
    • sudden traffic caused by hot events;
    • batch changes to application-layer caching strategies or prompt templates.

    Only after smoothing over multiple weeks and observing alongside task-type composition can teams identify real trends.

    2.3 More Useful Derived Metrics

    Instead of obsessing over a single model’s weekly call volume, teams should watch these derived metrics:

    • Model switching frequency: how often an application switches models within a period, reflecting the degree of competitive supply.
    • Cost per completed task: average tokens and cost consumed to complete one full business task.
    • Output per token: effective tasks supported per million tokens.
    • Quality per unit cost: cost per task at an equivalent quality level, often described as “cost-performance ratio.”

    For engineering teams, model selection is essentially finding the best solution under constraints among capability, cost, and compliance, rather than chasing a ranking at a particular moment. The more sufficient the market supply becomes, the more selection should return to business evaluation and engineering governance.

    3. Selection and Evaluation: Use Business Data, Not Rankings

    3.1 Build a Task-Model Capability Matrix

    Different tasks place very different demands on model capability. Here are four common task types:

    Task type Core capability requirements Key metrics Common engineering practices
    Knowledge Q&A and customer service Instruction following, knowledge accuracy, response speed First-token latency (TTFT), answer accuracy, hallucination rate Retrieval-augmented generation, caching, fallback to smaller models
    Document structured extraction Long context, structured output, numerical calculation Field accuracy, format validity Output constraints, rule validation, manual sampling
    Code generation and completion Code correctness, context understanding, low latency Compilability, unit test pass rate, completion latency Code indexing, progressive completion, static checks
    Agent tool calling and planning Function-calling stability, multi-step reasoning, context retention Task success rate, average steps, interruption rate State-machine constraints, idempotent retries, checkpoint recovery

    The matrix should be used as follows: identify the P0 tasks in the business, then select 3 to 5 candidate models for each task, avoiding a single standard for all business scenarios.

    3.2 Build a Reproducible Evaluation System

    1. Evaluation set source: sample from real business data, covering typical cases, edge cases, and historical failure cases. Freeze versions so that different models are compared on the same data.
    2. Combine scoring methods:
      • Use rule-based assertions for deterministic tasks such as extraction, classification, and format validation.
      • Use “model review + manual sampling” for generation tasks, and regularly calibrate the consistency between the review model and human judgment.
    3. Test performance and stability separately: load-test TTFT, output throughput, behavior under long contexts, rate-limit trigger frequency, and error rates.
    4. Use small-traffic gray release: route a portion of real traffic to a new model according to rules, compare with online metrics, and gradually increase traffic.
    5. Run regular regression tests: model services continue to iterate, so evaluation sets must be updated as business changes. Also prevent evaluation-set data from leaking into prompt examples.

    3.3 Sample Code: Weighted Scoring Based on Task Profile

    1. # This is only for demonstrating the scoring idea. Weights and metrics should be configured based on real evaluation results.
    2. # dimensions: quality (evaluation set score), latency (TTFT pass rate), cost (normalized inverse of cost per task), compliance (whether redaction and audit capabilities meet requirements)
    3. TASK_WEIGHTS = {
    4. "chat": {"quality": 0.40, "latency": 0.25, "cost": 0.25, "compliance": 0.10},
    5. "extract": {"quality": 0.50, "latency": 0.10, "cost": 0.25, "compliance": 0.15},
    6. "code": {"quality": 0.45, "latency": 0.25, "cost": 0.20, "compliance": 0.10},
    7. "agent": {"quality": 0.40, "latency": 0.15, "cost": 0.25, "compliance": 0.20},
    8. }
    9. def score_model(model_metrics: dict, task: str) -> float:
    10. weights = TASK_WEIGHTS[task]
    11. total = 0.0
    12. for dim, weight in weights.items():
    13. val = model_metrics.get(dim, 0.0)
    14. val = min(max(val, 0.0), 1.0) # Normalize to [0, 1]; outliers are excluded
    15. total += weight * val
    16. return round(total, 4)

    Scoring is only a decision aid. For sensitive data or regulated business scenarios, compliance should be an admission threshold (a veto condition), not an ordinary weighted dimension.

    4. Architecture Practice: Use a Unified Model Gateway for Multi-Model Coexistence

    When multiple models are used in parallel, if every business module connects directly to each service, several problems arise: inconsistent API styles, scattered key management, difficulty enforcing unified rate limiting, and no fallback during failures. A more robust approach is to add a unified model gateway between applications and model services. The gateway handles protocol adaptation, routing decisions, flow control, and observability.

    4.1 Layered Design

    1. Access layer: serves business applications and handles authentication, tenant identification, request validation, and rate limiting.
    2. Routing layer: selects candidate models based on task type, input size, priority, and cost budget, and supports weighted gray release.
    3. Adapter layer: implements unified request and response conversion for each model service, hiding differences in parameter names and streaming response formats.
    4. Governance layer: handles logging, metrics, tracing, budget alerts, and auditing.
    1. Business applications
    2. |
    3. v
    4. +-----------------------------+
    5. | Unified model gateway |
    6. | Access -> Routing -> Adapter|
    7. +--------------+--------------+
    8. | | |
    9. v v v
    10. Primary Backup Lightweight
    11. model model model

    4.2 Routing Strategies

    • Rule-based routing: send deterministic tasks such as intent classification and document extraction to smaller models confirmed by evaluation. Send open-ended Q&A and complex reasoning to more capable models.
    • Cost-based routing: while meeting quality requirements, prefer candidates with lower cost per task, dynamically considering cache hit rate.
    • Tiered fallback: try models in the order of “primary model → same-tier backup model → lightweight fallback.” The fallback layer returns templated answers or transfers to human support, while logging fallback events for later review.
    • Gray release: new models first receive a small portion of traffic. After all metrics meet requirements, traffic is gradually increased.

    4.3 Fault-Tolerance Design Points

    • Set timeouts and maximum retries for each call. Retries should only target recoverable errors, such as network jitter, rate limiting, or temporary server unavailability.
    • Use idempotency keys on the business side to avoid duplicate writes or duplicate orders caused by retries.
    • For failures with unknown state, such as interrupted streaming output, keep checkpoints to support recovery from the interruption point.
    • Implement circuit breaking: when a service fails continuously, quickly switch to a backup candidate.

    4.4 Sample Code: Routing, Fallback, and Usage Reporting

    1. # This is only for demonstrating the routing and fallback flow. Field names and exception types should be adapted to the actual environment.
    2. import time
    3. def generate(task: str, ctx: dict):
    4. for model in route_candidates(task, ctx): # Return candidates by rule, weight, or cost
    5. start = time.monotonic()
    6. try:
    7. resp = call_with_retry(model, ctx, max_retries=2, timeout_s=30)
    8. record_usage(trace_id=ctx["trace_id"], task=task,
    9. model=model, latency_ms=(time.monotonic() - start) * 1000,
    10. resp=resp, status="success")
    11. return resp
    12. except (TimeoutError, RateLimitError, ServiceUnavailable) as e:
    13. # Recoverable error: log it and fall back to the next candidate
    14. log_fallback(trace_id=ctx["trace_id"], model=model, reason=type(e).__name__)
    15. continue
    16. # All candidates failed: return a fallback response and mark it as degraded
    17. return fallback_response(task, ctx)

    4.5 Multimodal and Streaming Scenarios

    • Multimodal requests: non-text inputs such as images and audio must first be converted into an encoding accepted by the model. Some services support passing image URLs or binary data directly. Engineering teams must also consider storage and transmission of large files. For example, object storage and CDN distribution can be used to serve original files, with only file references carried in requests, avoiding overly large request bodies.
    • Streaming output: providing streaming responses for chat products can significantly reduce the perceived waiting time caused by first-token latency. The gateway must forward incremental content chunk by chunk and handle cleanup logic after client disconnection.

    5. Observability: Make Every Token Consumption Traceable

    5.1 Minimum Logging Fields

    Field Purpose
    trace_id, session_id Correlate all calls in one task
    Tenant/team/business scenario Cost allocation and quota management
    Model identifier, version Effect comparison and regression analysis
    Input/output/cached token counts Usage accounting and cost calculation
    First-token latency, total latency Performance monitoring and capacity planning
    Status code, error type, retry count Stability analysis
    Whether fallback occurred Evaluate actual availability of candidate models

    5.2 Core Monitoring Metrics

    • Usage metrics: token consumption, request volume, cache hit rate, average calls per task.
    • Quality metrics: task success rate, automatic evaluation score, manual sampling pass rate.
    • Performance metrics: TTFT P50/P95/P99 percentiles, error rate, rate-limit trigger count.
    • Cost metrics: cost per task, daily/monthly accumulated cost, budget consumption progress.

    Alerts should use combined multi-dimensional conditions, such as “error rate keeps rising and fallback ratio exceeds a threshold.” This reduces false positives and helps detect service anomalies earlier.

    Logs can be written asynchronously to a logging service through a message queue. Metrics and alerts should be integrated into a unified monitoring and alerting system. Tracing data is used to reconstruct the complete call chain.

    5.3 Sample Code: Usage and Cost Estimation

    1. # Pricing and cache billing rules should follow the official documentation of the selected service. This only demonstrates the calculation logic.
    2. def estimate_cost(usage: dict, price: dict) -> float:
    3. cached = usage.get("cached_tokens", 0)
    4. uncached_in = max(usage.get("prompt_tokens", 0) - cached, 0)
    5. cost = uncached_in * price["input"] / 1_000_000
    6. cost += cached * price.get("cached_input", price["input"]) / 1_000_000
    7. cost += usage.get("completion_tokens", 0) * price["output"] / 1_000_000
    8. return round(cost, 6)

    6. Cost Governance: Balancing Quality and Cost

    1. Layered caching

      • Exact caching: use the hash of prompt content as the key and return historical results directly on a hit.
      • Prefix caching: fixed prefixes such as system prompts, tool definitions, and long documents can reuse computed results if the selected service supports context caching, reducing consumption for those inputs.
      • Semantic caching: match similar questions through vector similarity to greatly reduce duplicate calls. Set a reasonable similarity threshold and expiration/invalidation policies to avoid returning outdated or incorrect answers.
    2. Prompt slimming: remove redundant examples and irrelevant background, control context length, and avoid “stuffing the entire database into the window.”

    3. Cascaded calls: use a lightweight model first for simple requests, then escalate to a more capable model when confidence is insufficient. Routing by confidence is more economical than using a large model for every request.

    4. Batch and asynchronous processing: for offline extraction, batch labeling, and similar tasks, if the service provides asynchronous batch APIs, off-peak execution can usually be used to obtain lower unit prices or more stable loads.

    5. Budgets and quotas: set token quotas and cost budgets for each team and each business. Automatically alert when approaching thresholds and support adjusting routing strategies by priority.

    7. Security and Compliance: An Essential Part of Multi-Model Architecture

    • Data classification and desensitization: identify and desensitize sensitive fields such as personal information, account numbers, and identity document numbers before sending. For highly sensitive data, prioritize private deployment or a compliant dedicated network option.
    • Transmission and storage: use HTTPS end to end, centrally manage keys, and encrypt or truncate sensitive content in call logs.
    • Auditing and traceability: record input/output summaries, model versions, and operators for each call to meet traceability requirements.
    • Output validation: validate the format of structured outputs, require citation sources for key conclusions, and perform post-validation based on business rules.
    • Vendor management: evaluate whether data is used for training, data retention periods, and data residency terms. When necessary, use technical architecture to constrain data flows.

    8. Common Pitfalls and Selection Checklist

    8.1 Common Pitfalls

    1. Ranking obsession: looking only at call volume or benchmark scores without considering business scenario fit.
    2. Ignoring total cost of ownership: comparing only token prices without counting retries, manual validation, and operations costs.
    3. Single-point dependency: relying on one service for all business without a fallback plan.
    4. Cache failure: failing to monitor cache hit rate, causing costs to keep rising.
    5. Context inflation: repeatedly stuffing all historical conversations into the window, causing cost and latency to rise together.
    6. Missing evaluation: not running regression tests after model service updates, allowing quality to decline unnoticed.

    8.2 Selection Checklist

    • Are the P0 tasks and their capability requirements clearly defined?
    • Is there a version-frozen business evaluation set and an automated scoring process?
    • Have performance, stability, security, and compliance been verified?
    • Is there multi-candidate routing and fallback capability?
    • Is token usage and budget alerting implemented per tenant/business?
    • Have sensitive data desensitization and access controls been implemented?
    • Is there a regular regression evaluation mechanism?

    9. Conclusion

    Call-volume data reflects market activity and supply-side competition, but it cannot answer “how should enterprises use LLMs well?” For developers and enterprise users, a more valuable path is:

    • Start with business evaluation;
    • Use a unified model gateway as the hub;
    • Rely on an observability system for evidence;
    • Treat cost governance and security compliance as the baseline.

    In practice, teams can choose mature cloud model services based on business scenarios and use common cloud products such as object storage, message queues, container platforms, logging services, and monitoring/alerting systems to build infrastructure. For teams comparing architecture options, Baidu AI Cloud provides a relatively complete set of cloud products and tools around model invocation, application development, and runtime support.

    When call volume is no longer the only metric, but is broken down into measurable engineering questions such as task completion rate, cost per task, and quality per unit cost, teams can truly benefit from the “model competition” and turn technical capability into stable business value.