# LLM Inference Architecture Design and Optimization: Cutting Token Cost at Scale
Weekly token consumption across large language models has climbed into the trillions, with lightweight models now absorbing a large share of that load. This article explains what those numbers actually measure, why token consumption keeps rising, and how to design an inference architecture that lowers cost per token and raises throughput without sacrificing quality. It covers the metrics that matter (TTFT, TPOT, throughput per GPU, cost per business task), the six core inference-engine optimizations (continuous batching, paged KV cache, prefix caching, chunked prefill with phase disaggregation, quantization, speculative decoding), tiered routing by task complexity and budget, three layers of caching and cache-key design, context engineering, elastic scheduling signals and cold-start mitigation, plus observability, cost attribution, continuous evaluation, and a four-stage rollout roadmap.
Introduction: Once Call Volume Tops the Charts, the Real Work Begins
According to statistics from an international model aggregation platform, global LLM call volume reached roughly 188 trillion tokens in the week of October 19–25, 2026. Chinese LLMs accounted for about 93.54 trillion of those tokens — roughly half the total — and have held the global top spot for 26 consecutive weeks. Lightweight models stood out on the leaderboard: one lightweight variant recorded about 33.6 trillion tokens in a single week, ranking first globally with a week-over-week increase of 801%, while another lightweight model logged about 15 trillion tokens over the past seven days, placing second on the platform.
For developers, these figures are more informative than the ranking itself. They point to three things:
- Lightweight models are absorbing real, scaled production load. High-frequency tasks such as classification, extraction, rewriting, and code completion are sensitive to latency and cost, so their call volume concentrates on lightweight variants. This contradicts the intuition that “small models were obsoleted by the large-model era” and instead confirms the need for model tiering.
- Exponential call growth translates directly into cost pressure. Tokens are both consumed and billed by volume, so doubling call volume at least doubles cost per unit time. Without engineering optimization, faster business growth means a larger bill.
- Call volume is an outcome metric, not a capability metric. Platforms measure distribution-layer volume, which is influenced by model pricing, version releases, and traffic allocation. An 801% week-over-week jump more likely reflects traffic redistribution after a new version launch than a genuine eightfold increase in load for the same model.
Call volume therefore hands engineering teams one core problem: at equal service quality, how do you lower cost per token, raise throughput per unit of compute, and keep the service more stable? This article starts from metric definitions and then works through the design and optimization of inference architectures.
1. Understand Call Volume First: Metric Definitions and What They Mean
1.1 Tokens, Requests, and Call Volume Are Not the Same Thing
A token is the basic unit of text the model processes. Text is first split into a token sequence by a tokenizer, then fed into the model. The same Chinese content can produce noticeably different token counts under different tokenizers and vocabularies, so when comparing cost across models, price per million tokens alone is not enough.
Call volume reflects token throughput; request count reflects the number of interactions. The two can diverge sharply:
- A coding assistant with a 10,000-token system prompt may issue only one request while consuming tens of thousands of tokens.
- A Q&A service with effective prompt caching may see request counts multiply several times while billed tokens stay roughly flat.
1.2 The Core Metric Checklist
| Dimension | Key metric | Meaning and role |
|---|---|---|
| Latency | Time to first token (TTFT) | Time from request dispatch to the first output token; driven mainly by the prefill stage and central to the streaming experience |
| Latency | Time per output token (TPOT) | Average time to generate each output token during decoding; determines the smoothness of long answers |
| Throughput | Output tokens per second per GPU | Measures the output capability of a unit of compute |
| Throughput | Batch request completion rate | Measures overall system capacity under high concurrency |
| Cost | Blended cost per million tokens | Accounts for input, output, and cache hits |
| Cost | Cost per business task | Total cost of completing one user-visible task, including multi-turn and multi-step calls |
| Cost | Cache hit rate | Directly determines actual billed volume |
| Quality | Task success rate / human acceptance rate | Guards against “cheaper but worse” outcomes |
| Stability | Error rate, P95/P99 queueing latency, timeout rate | Reflects whether capacity is sufficient |
Cost per business task is the most easily overlooked metric, yet the one that matters most. A single code change in a coding assistant may trigger dozens of model calls, so differences in per-token price are multiplied by the number of calls.
1.3 Leave Room to Interpret the Aggregate Numbers
Aggregated platform data is useful as a reference, but it has clear limitations:
- Limited coverage. It mainly reflects calls distributed or forwarded through the platform and cannot represent the full consumption from enterprise self-owned APIs, private deployments, and other paths.
- Sensitive to traffic allocation. Platforms adjust routing based on price, speed, and available quota, so a sudden jump in one version’s call volume does not necessarily mean the underlying load spiked.
- Sensitive to measurement window. Weekly statistics are easily skewed by version launches, promotions, and similar events.
Enterprises should therefore treat the metering data from their own inference gateway as authoritative and use platform figures only as an industry-trend reference.
2. Why Token Consumption Keeps Climbing
Rising token consumption is not just “more users.” It reflects a deeper shift in calling patterns:
- From single-turn Q&A to agents. In agent mode, the model repeatedly cycles through think → call a tool → observe the result → think again, and each step re-runs inference with the full history attached. Token consumption can grow roughly linearly with the number of steps, and sometimes faster.
- Contexts keep getting longer. With a knowledge base connected, retrieved documents get concatenated into the prompt; in multi-turn conversations, history accumulates; coding assistants must also pull relevant code snippets from the repository into context. The longer the context, the heavier the prefill computation.
- Structured output and validation cause repeated consumption. When strict JSON, unit tests, or a compliant format is required, generation is typically followed by validation and sometimes retries, so one task can consume several times the tokens.
- Machine-initiated calls are a growing share. Batch data processing, automated evaluation, and automated repair in CI pipelines shift the source of calls from “a human clicking” to “a program looping.”
Token budgets therefore have to be governed like server capacity or storage: measurable, allocable, alertable, and optimizable.
3. The Inference Engine: From “It Runs” to “It’s Efficient”
3.1 The Two Phases of Autoregressive Generation
When an LLM generates text, it computes the token distribution at each position and samples an output. This happens in two phases with very different characteristics:
- Prefill: The model processes all input tokens in parallel. This is compute-intensive and mainly constrained by compute capacity. Longer input means higher TTFT.
- Decode: Only one new token is generated at a time, repeatedly reading model weights and the existing key-value cache (KV Cache). This is memory-bandwidth intensive and mainly constrained by VRAM bandwidth.
This difference dictates the direction of the two families of optimizations: prefill focuses on parallel computation and cache reuse, while decode focuses on memory efficiency and batching.
3.2 The Memory Ledger: Do the Math Before Optimizing
Consider an illustrative example: a model with 32 layers, where each layer’s attention keys (K) and values (V) each hold one cache entry, each entry requiring 2 numbers per token, with 2 bytes per number in half precision. That is 2 × 2 × 2 = 8 bytes per token per layer, and 256 bytes per token across 32 layers.
| Context length | KV cache per request | 1,000 concurrent requests |
|---|---|---|
| 4K tokens | 1 MiB | ~1 GiB |
| 32K tokens | 8 MiB | ~7.8 GiB |
| 128K tokens | 32 MiB | ~31.3 GiB |
That accounts only for the KV cache. Model weights, framework runtime overhead, and activations consume additional VRAM. The longer the context and the higher the concurrency, the more likely the KV cache becomes the throughput bottleneck. Nearly all inference-engine optimizations revolve around using memory well.
3.3 Six Key Optimization Techniques
1) Continuous batching. Traditional static batching must wait for a full batch of requests to arrive and for all of them to finish generating before resources are released, so short requests get held back by long ones, creating head-of-line blocking. Continuous batching schedules at decode-step granularity: as soon as a request finishes it exits immediately, and the freed slot is filled by a new request right away, substantially improving GPU utilization.
2) Paged KV cache management. Borrowing the idea of virtual-memory paging from operating systems, each request’s KV cache is split into fixed-size blocks, with a block table recording the mapping from logical to physical positions, allocated on demand and reclaimed after use. This reduces memory fragmentation and makes it easier for multiple requests to share cache blocks with identical content, such as the same system prompt.
3) Prefix cache reuse. System prompts, tool definitions, few-shot examples, and fixed long documents recur across a large number of requests. If the inference service supports prefix caching, the prefill result for an identical prefix can be reused, so subsequent requests only compute the changed part. Both TTFT and prefill cost drop significantly.
4) Chunked prefill and phase separation. Splitting the prefill of a very long input into multiple chunks and interleaving them with decode requests prevents a long request from monopolizing compute for an extended period and slowing down other users. At larger scale, prefill and decode can also be deployed into separate resource pools, each provisioned according to its own bottleneck.
5) Quantization. Lowering the numerical precision of weights and the KV cache (for example INT8 or FP8) reduces memory footprint and memory traffic, allowing larger batch sizes and higher throughput. The gains and the quality loss must be established through evaluation, and sensitive workloads should set a quality red line.
6) Speculative decoding. A lightweight “draft” model proposes several tokens in sequence, then the target model validates them all in parallel at once: accepted tokens are taken directly, and rejected positions are rolled back and recomputed. Because validation is parallel, decoding can often be sped up while keeping the output distribution essentially unchanged.
Beyond these, long-context scenarios can also use chunked retrieval, sliding windows, or sparse attention to control attention computation. Mixture-of-Experts (MoE) architectures let each token activate only a subset of experts as parameter count grows, reducing the computation per call.
All of these are general-purpose inference optimization directions in the industry. Actual implementation capabilities should be confirmed against the documentation of the chosen model service and inference framework, and validated through measurement.
4. Tiered Routing: Match Each Class of Request to the Right Model
The first-principles rule of engineering here is: do not use the same model tier for every request.
4.1 An Example of Task Tiers
| Tier | Typical tasks | Suggested model choice |
|---|---|---|
| Lightweight | Intent recognition, information extraction, format conversion, short-text rewriting | Small-parameter model or lightweight variant |
| General | Everyday Q&A, routine content generation, single-turn code completion | Mid-size general-purpose model |
| Advanced | Multi-step reasoning, long-document analysis, cross-file refactoring, algorithm design | Large-parameter model or deep-reasoning mode |
4.2 Dynamic Routing Strategies
- Complexity prediction: Assess difficulty with a lightweight model or small classifier as the request enters, or let business labels (code review, contract review) decide the route directly.
- Budget awareness: Set a cost budget per tenant or business line and automatically fall back to a more economical tier as the threshold approaches.
- Quality and consistency constraints: Tasks that require strictly consistent output (such as structured extraction) should pin the model and its parameters, avoiding routing drift that destabilizes results.
- Fallback degradation: When the large model is overloaded or times out, fall back to the general tier, or notify the user in the response that “fast mode has been switched on.”
4.3 Sample Code: Routing by Budget and Priority
from dataclasses import dataclass@dataclassclass RouteDecision:model_tier: str # "light" | "standard" | "advanced"max_tokens: int # upper bound on output tokenstemperature: floatdef decide_route(req, budget) -> RouteDecision:"""Illustrative logic: evaluate hard constraints (budget / priority),then task type, then complexity, and pick the lowest model tier thatsatisfies the constraints. Tune against evaluation results beforeproduction use."""# 1. Hard constraint: on the cost-saving lane or with the budget# exhausted, route everything to the lightweight modelif req.cost_saving or budget.remaining_tokens <= 0:return RouteDecision("light", max_tokens=512, temperature=0.2)# 2. Task type: deterministic tasks like extraction and classification# use low temperatureif req.task_type in ("extract", "classify", "format"):return RouteDecision("light", max_tokens=1024, temperature=0.0)# 3. Priority: high-priority interactive requests favor latency and use# the fast tierif req.priority == "high" and req.context_tokens <= 8192:return RouteDecision("standard", max_tokens=2048, temperature=0.7)# 4. Complexity: multi-step reasoning or very long contexts use the# advanced tierif req.complexity_score >= 3 or req.context_tokens > 32768:return RouteDecision("advanced", max_tokens=8192, temperature=0.3)return RouteDecision("standard", max_tokens=2048, temperature=0.7)
5. Cache Reuse: The Most Direct Cost Lever
5.1 Three Layers of Caching
- Result caching: For highly repetitive requests (high-frequency FAQs, fixed-template generation), cache the complete result at the application layer. Set a sensible expiration time and use this cautiously for personalized requests.
- Semantic caching: For questions that mean the same thing but are phrased differently, use vector retrieval to judge semantic similarity and return the historical answer on a hit. This suits higher-tolerance scenarios such as knowledge Q&A, but not numerical computation or real-time data lookups.
- Prefix KV caching: Handled by the inference engine or inference service, reusing the computation result of an identical prefix across requests. This is currently one of the most reliably beneficial optimizations and is especially suited to long system prompts and fixed knowledge-base documents.
5.2 Cache Key Design
A cache key must cover every factor that affects the output, otherwise you get cross-contamination or wrong hits:
import hashlibimport jsondef build_cache_key(system_prompt: str, messages: list, params: dict,model_version: str, knowledge_version: str) -> str:payload = {"system": system_prompt,"messages": messages,"params": params, # temperature, top_p, max_tokens, etc."model": model_version, # a model version change must# invalidate old cache entries"knowledge": knowledge_version, # a knowledge base update must too}raw = json.dumps(payload, ensure_ascii=False, sort_keys=True)return hashlib.sha256(raw.encode("utf-8")).hexdigest()
Note that parameters such as temperature change the output distribution, so caches for generation tasks must only be reused under identical parameters. Deterministically exact tasks may relax this constraint.
5.3 Context Engineering: Cut Tokens at the Source
- Put stable system prompts and tool definitions at the front and dynamic content at the end, which makes prefix cache hits easier.
- Truncate retrieved results by relevance instead of stuffing an entire document into the context.
- Summarize and compress early conversation history in multi-turn dialogs, keeping only key conclusions.
- Under token-based billing, a longer context means a higher bill, so context governance is also cost governance.
6. Elastic Scheduling: Handling Fluctuation and Peaks
6.1 The Tidal Nature of Load
Inference load usually shows pronounced peaks and troughs:
- Weekday daytime is the peak for office scenarios.
- Nights and weekends may be filled by batch jobs and automated evaluation.
- Coding assistants see burst peaks during release crunches.
Provisioning statically for the peak leaves resources heavily idle in the trough; provisioning for the trough means being overwhelmed at the peak. Elastic scheduling is the key to resolving this tension.
6.2 Choosing Scaling Signals
Rather than relying solely on CPU utilization or request count, autoscaling for inference services is better driven by a combination of signals:
- Queueing latency or P95 TTFT: reflects user experience directly and is the most sensitive scale-up signal.
- KV cache utilization: approaching the ceiling means concurrency capacity is nearly exhausted.
- GPU utilization and per-GPU throughput: reflect real compute pressure.
- Request queue length: a persistently growing queue is a clear scale-up signal.
Set reasonable cooldown periods and scaling step sizes to avoid oscillation.
6.3 Cold Starts and Resource Pooling
A typical pain point for inference services is the cold start: a new instance must load model weights of several GB up to hundreds of GB, so capacity often lags behind a traffic peak. Common mitigations include:
- Keep a baseline share of always-on instances to absorb steady traffic.
- Pre-warm model weights, and use shared or local model caches to speed up loading.
- Build multi-spec resource pools (GPUs of different compute tiers, CPU inference nodes), running lightweight models on CPUs or lower-compute cards so high-compute cards are freed for complex models.
- Stagger scheduling: during the overnight trough, shift part of the capacity to batch inference or offline evaluation, and switch it back to online serving automatically before the peak.
- Asynchronous task queues: non-real-time batch work (document summarization, bulk translation) enters a message queue for peak shaving and valley filling, with consumers processing it at a steady rate according to available capacity.
On cloud platforms, general capabilities such as container platforms, autoscaling, object storage, message queues, logging, and monitoring and alerting can be combined for this purpose. Baidu AI Cloud offers products in these categories and can serve as a reference for building such an elastic inference platform; check the official documentation for current availability and service coverage.
6.4 Stability Design
- Multiple replicas and fault isolation: group deployments by model and by workload so a single failure does not spread.
- Timeouts and a retry budget: set an end-to-end timeout per request and count retries against the total budget, so retry storms do not amplify failures.
- Rate limiting and quotas: enforce per-tenant, per-endpoint token-per-minute quotas so abnormal callers cannot crowd out resources.
- Circuit breaking and degradation: when a dependency misbehaves, switch quickly to a backup model or return a cached result.
7. Observability and Cost Governance
7.1 End-to-End Tracing
Tie the gateway, routing layer, inference engine, cache, and downstream tool calls together with a request ID to form a complete call chain. One business task may involve many model calls, and only end-to-end tracing can show whether the bottleneck sits in prefill, decode queueing, or waiting on a tool call.
7.2 Token Metering and Cost Attribution
The inference gateway should log, for every call: input tokens, output tokens, cache-hit tokens, model tier, business label, tenant identifier, and elapsed time. On that basis:
- Generate cost reports by business line, tenant, and scenario.
- Identify high-consumption, low-value calls (failed retries, uselessly long contexts) and optimize them specifically.
- Set budget alerts that notify owners automatically when thresholds are exceeded, or trigger a degradation policy.
7.3 Continuous Evaluation and a Closed Loop
Cost reduction must not come at the expense of quality. Build an offline evaluation set covering typical tasks, and run a regression on it before any model swap, quantization upgrade, routing change, or prompt edit. Focus metrics include task success rate, format compliance rate, and human acceptance rate; ship only after those targets are met, then roll out gradually.
Continuously feed production failure samples back into the evaluation set to close the loop: monitor → analyze → optimize → evaluate → ship.
8. A Phased Rollout Roadmap
| Phase | Suggested duration | Core work | Expected benefit |
|---|---|---|---|
| Phase 1: Make it visible | 1–2 weeks | Deploy an inference gateway, unify token metering and cost attribution, establish baselines for latency, throughput, and error rate | Understand the real call structure and cost distribution |
| Phase 2: Cut cost | 3–6 weeks | Prefix caching, context governance, tiered routing, result caching and semantic caching | Reduce repeated computation and calls to expensive models |
| Phase 3: Raise efficiency | 6–10 weeks | Continuous batching, paged KV cache, chunked prefill, speculative decoding, quantization | Increase per-GPU throughput, shorten TTFT |
| Phase 4: Strengthen elasticity | Ongoing | Autoscaling, multi-spec resource pools, asynchronous peak shaving, multi-model fallback | Absorb peaks smoothly and improve availability |
Two principles are worth following during implementation: first, every optimization should come with a measurable before-and-after comparison; second, load-test in a pre-production environment and only roll out gradually after it passes.
Conclusion
Chinese LLMs holding the global top spot in weekly call volume for 26 consecutive weeks marks the shift of LLM applications from experimentation to scaled production. The higher the call volume, the more important engineering capability becomes. For teams building or planning to build LLM applications, five points stand out:
- Measure call volume with the right metrics rather than being driven by leaderboards;
- Use tiered routing to match model capability to task difficulty;
- Use caching and context engineering to cut token consumption at the source;
- Use inference-engine optimizations to raise output per unit of compute;
- Use elastic scheduling and observability to keep cost and quality in balance over the long run.
Ultimately, a token is both the basic unit of information an LLM processes and a yardstick for the maturity of AI engineering. Whoever can complete the same task with fewer tokens at equal quality will go further in this era of scaled application.
