New Model Launches and Old Model Retirement: Inference Architecture Evolution and a Selection Guide
A practical engineering guide to what really happens when a provider launches a new model and retires an old one. It explains why cost structure has become a first-class decision variable on the inference side, how model supply is being tiered into light and heavy options, and the generational gap in inference infrastructure. The article breaks down the key latency and throughput metrics (TTFT, TPOT, single-request and system throughput), the architecture and infra techniques that improve them (continuous batching, chunked prefill, KV cache management, speculative and constrained decoding, low-precision execution, and prefill/decode disaggregation), and the token-cost impact of model "overthinking" along with ways to govern it. It closes with a selection methodology based on tiered routing, cascade escalation, and business eval sets, a phased low-risk model retirement process, and a pre-launch checklist.
1. Three signals you can read from a version switch
When a team publicly says it would rather give up part of its usage volume to shift traffic to a new version, it usually means the old model’s compute cost per request (memory footprint, GPU-hours, power, and interconnect resources) has risen above the value it delivers. Combined with the paired move of “new version launch + simultaneous old model retirement,” you can abstract three layers of information.
Signal one: cost structure has become a first-class decision variable on the inference side. Training cost is one-time; inference cost is continuously compounding. Once call volume grows to a certain scale, per-request cost enters the gross-margin model directly, and “can it keep running” more easily becomes the bottleneck than “does it run well.” Retiring the old model is essentially a capacity reallocation: pulling memory and GPU-hours back from a low cost-performance path and investing them in the new architecture and new business.
Signal two: model supply is becoming tiered. A product line usually doesn’t offer only a single tier; it splits into light/fast tiers and heavy inference tiers along “task complexity × latency budget × unit cost.” When the light tier is capable enough, routing users to it is a rational capacity strategy, not a product downgrade—many real business requests (classification, extraction, formatting, short code completion, simple Q&A) don’t need heavy inference.
Signal three: there is a generational gap in architecture and infrastructure. The same model capability, running on serving stacks and hardware of different generations, can differ several-fold in latency and throughput. If the new version can stably reach “first token in hundreds of milliseconds, hundreds of tokens per second per request” while the old one can’t, the gap comes more from the inference engine, caching systems, and parallelism strategy than from model structure changes alone.
Note that all the latency and throughput figures in this article are common industry target magnitudes and illustrations meant to explain engineering implications; they are not measured commitments for any specific product or platform. Actual performance varies significantly with hardware, input length, and concurrency load.
2. Get the metrics right first: the engineering meaning of TTFT and single-request throughput
Many “model speed” debates lose focus because they conflate several different metrics. At minimum, separate them:
| Metric | Definition | Main determining factors | Typical sensitive scenarios |
|---|---|---|---|
| TTFT (time to first token) | Time from sending the request to receiving the first output token | Prefill compute, input length, queue wait, scheduling policy | Interactive chat, AI coding, long-document Q&A |
| TPOT (time per output token) | Time between adjacent tokens after the first token | Decode-phase memory bandwidth, intra-batch concurrency, per-step overhead | Streaming reading experience |
| Single-request throughput | Output speed derived from TPOT (tokens/s) | Read bandwidth for weights and KV cache, whether each step is interrupted by communication | Long-form generation, code generation |
| System throughput | Total tokens produced by the whole cluster per second | Batching efficiency, memory capacity, degree of parallelism | Capacity planning and unit cost |
| End-to-end latency and P95/P99 | Full-chain latency distribution | Sum of the above plus the long tail | SLA commitments |
One key point: single-request speed and system throughput often pull against each other. Raising intra-batch concurrency significantly improves total throughput and dilutes unit cost, but enlarges the denominator for each request and worsens TPOT. So capacity planning must define two sets of SLOs simultaneously: a single-request latency target for user experience, and a total throughput/unit-cost target for finance. Squeeze only one, and the other will surely go out of control.
Add one quantity that’s easily overlooked: the KV cache is the main consumer of memory. Every decoding step reads all weights and all historical KV, so the longer the context and the higher the concurrency, the greater the pressure on memory bandwidth and capacity. Latency degradation in long-context scenarios is often not a matter of insufficient compute, but of the KV cache not fitting and causing frequent swap in/out.
3. To hit this kind of latency and throughput, what usually changes in the architecture and infra
The following techniques are common engineering practices in the industry and can be combined as needed.
Scheduling layer: continuous batching + chunked prefill. The traditional “compute the whole batch together, then swap in the next” causes severe head-of-line blocking: one very long prompt stalls every short request in the batch. Continuous batching lets sequences enter and leave the batch dynamically at the token level; chunked prefill splits a long prompt into small chunks and interleaves them with decoding steps, thereby pushing down the TTFT tail.
KV cache system: paged management + prefix reuse + quantization. Use paged memory management to reduce fragmentation and reservation waste; reuse prefix caching for requests sharing the same prefix (same system prompt, same batch of few-shot examples, same repository context) to significantly cut repeated prefill compute, with especially notable gains for “long fixed prefix + short variable suffix” workloads like AI coding; quantize KV to a lower bit-width when necessary, trading a little precision for capacity.
Parallelism and topology: combining several parallelism modes + compute-communication overlap. Tensor parallelism splits a single layer’s compute, pipeline parallelism splits layers, expert parallelism splits expert modules, and data parallelism scales replicas; pair this with high-speed interconnect to amortize collective communication overhead, and overlap communication with computation. Sparsely activated structures (where each token passes through only part of the parameters) can keep per-token compute low while total parameters stay large—a common reason newer models are “bigger but faster.”
Decoding acceleration: speculative decoding and constrained decoding. Use a small draft model or multi-candidate prediction to guess several tokens, then verify them in one parallel pass with the main model, saving steps when they’re accepted; for strongly structured output like JSON, regex, or syntax trees, use constrained decoding to cut invalid tokens at the source, which both speeds things up and reduces retries.
Numerics and operators: low precision + graph execution + efficient attention kernels. Use low-precision formats like FP8/INT8 to reduce memory and bandwidth pressure (while keeping higher precision on sensitive layers); use static graph capture to eliminate per-operator launch overhead; use fused, efficient attention operators instead of naive implementations.
Deployment shape: disaggregated prefill and decode. The two phases have completely different resource profiles—prefill is compute-heavy, bursty, and latency-sensitive; decode is bandwidth-heavy, steady-state, and capacity-sensitive. Splitting them into two service groups that scale and allocate hardware independently makes it easier to meet TTFT and TPOT targets at the same time than co-locating them.
Gateway and observability layer. Streaming responses, timeouts and retries, circuit breaking and degradation (degrade to the light tier when you can’t keep up rather than returning errors outright), per-tenant priority queues and quotas, and instrumenting/reporting TTFT, TPOT, queue time, and thinking-token ratio. Without this layer, the effects of the previous six items can’t be attributed, and problems can’t be quickly located during incidents.
4. The “overthinking” problem: the second growth curve of token cost
Newer reasoning-oriented models generally have long internal thinking processes. The benefit is real—on tasks like math, complex code refactoring, and multi-step troubleshooting, letting the model write a few extra draft steps genuinely improves accuracy. But it brings two side effects:
- Latency is stretched: thinking content also consumes decoding steps, slowing end-to-end response.
- Cost is amplified: under most platforms’ billing, thinking content is metered on the output side, and the output-side unit price is usually clearly higher than the input side. In other words, thinking tokens are the item with the longest leverage on spending—check the billing description of the specific platform you use for the exact terms.
The governance approach is “think on demand,” not “turn thinking off entirely”:
- Explicit budget: if the service you use supports controls on thinking length or thinking effort, treat it as a tunable parameter and set caps by task tier.
- Routing by tier: turn thinking off directly for simple tasks (see section 5).
- Prompt-level constraints: require “conclusion first, then rationale,” specify a step limit, and forbid restating the prompt and self-introduction.
- Caching and reuse: prefix-cache repeated system prompts and repository context; result-cache idempotent query-type requests, but always set TTL and invalidation policies to avoid returning stale data.
- Asynchronous degradation: put latency-insensitive batch tasks into an offline queue and run them at a lower resource tier.
- Observability and attribution: make “thinking tokens / total output tokens” a core dashboard metric with threshold alerts, and attribute cost to teams and business lines.
The sample code below shows how to decompose latency and token composition for a call:
# Illustration only: decomposing latency and cost of a single streaming call (API names are pseudocode)def profile_stream(messages):t0 = time.perf_counter()ttft_ms = Nonethink_tokens = out_tokens = 0for chunk in stream(messages): # consume chunk by chunk to get the first token earlyif ttft_ms is None:ttft_ms = (time.perf_counter() - t0) * 1000usage = getattr(chunk, "usage", None) # the usage field name varies by platformif usage:think_tokens = getattr(usage, "reasoning_tokens", 0) or 0out_tokens = usage.output_tokense2e_ms = (time.perf_counter() - t0) * 1000tps = out_tokens / max((time.perf_counter() - t0) - ttft_ms / 1000, 1e-6)return {"ttft_ms": ttft_ms, "tps": tps, "e2e_ms": e2e_ms,"think_tokens": think_tokens, "out_tokens": out_tokens,"think_ratio": think_tokens / max(out_tokens, 1)}
5. A selection methodology: tiering, cascading, and hard-case escalation
Instead of arguing over “which version is stronger,” turn the question into an engineering problem: given the task’s complexity distribution and latency budget, how do you minimize total cost while keeping an acceptable failure rate.
Step one: tiered routing. Give default tiers and thinking budgets by task type:
# Illustration only: model tiering and budget control by task difficultyROUTES = {"extract": {"tier": "light", "thinking": 0, "timeout_s": 5}, # extraction/classification/formatting"chat": {"tier": "light", "thinking": 512, "timeout_s": 20},"code": {"tier": "light", "thinking": 2048, "timeout_s": 60}, # completion/single-file edits"reason": {"tier": "heavy", "thinking": 8192, "timeout_s": 180}, # multi-step reasoning/large refactors}def pick(task):if task["prompt_tokens"] > 32_000 or task.get("need_deep_reason"):return ROUTES["reason"]return ROUTES.get(task["kind"], ROUTES["chat"])
Step two: cascade escalation. Let the light tier answer first, then use an automatically determinable verifier to decide whether to escalate to the heavy tier. The verifier can be whether unit tests pass, whether compilation succeeds, whether a schema is valid, whether rules conflict, or whether confidence falls below a threshold. This is much cheaper than “always use the heavy tier” and safer than “always use the light tier”:
# Illustration only: light tier first + hard-case escalation (cap escalations to prevent runaway cost)def solve(task, verify, max_escalate=1):tier, out = "light", Nonefor _ in range(max_escalate + 1):out = call(pick_route(tier, task), task)if verify(out):return out, tiertier = "heavy"return out, tier
Step three: decide with a business eval set, not leaderboards. Leaderboards reflect average capability; your payoff depends on the difficulty distribution of your own business. Build a golden set covering the real traffic distribution (a few hundred to a few thousand items is enough to start), with metrics including at least: task pass rate, human-sampled acceptance rate, P95 end-to-end latency, and unit cost per thousand successful tasks. These four form a Pareto frontier, and selection is finding the point on that frontier that fits your business constraints.
6. The right way to retire a model: don’t let retirement become an incident
Retiring an old model is a high-risk change because it alters capability, latency, billing, and quotas at the same time. The proper approach is a phased rollout:
- Announcement and deprecation period: give the retirement time window and migration guidance in advance, and clarify the differences between the old and new versions.
- Aliases and dual-track: use a stable alias that points to the new version so callers don’t hard-code a specific version number, leaving room for future switches.
- Shadow traffic comparison: mirror real online requests to the new version, comparing without returning, and observe differences in pass rate and latency distribution.
- Canary migration: switch in batches by tenant, by traffic percentage, and by business line, keeping a one-click rollback switch.
- Full rollout and capacity reclamation: confirm there are no long-tail dependencies before taking the old model down, and prioritize reclaimed memory and GPU-hours for the new version’s capacity and peak-hour elasticity.
Migration checklist (the compatibility issues most easily stepped on):
- Changes to output format and structured-output stability.
- Whether tool/function-calling fields and multi-turn protocols stay consistent.
- Changes to parameter limits such as context window, max output length, and thinking budget.
- Changes to rate-limit quotas and burst capacity.
- Changes to billing terms (especially whether thinking tokens count on the output side).
- Changes to error codes and retry semantics—avoid misjudging a “successful degradation” as a failure and retrying wildly.
- Dependency governance: regularly scan code and config for hard-coded version numbers.
What about long-tail users. There will always be a few heavy users insensitive to the new version’s gains and already dependent on the old version’s behavior. The reasonable approach is to provide quota caps, queue priority, or a dedicated resource pool, not simply cut them off—these users’ contributed revenue is often disproportionate to their compute consumption and should be handled with a quota strategy rather than a capability strategy.
7. Pre-launch checklist
- Define latency SLOs (TTFT P95, TPOT P95) and throughput/cost SLOs separately, and alert on each separately.
- Load testing covers four scenario types: long prompts, very long context, burst concurrency, and slow nodes with partial failures.
- Thinking budgets are tiered by task, and thinking-token ratio is on the dashboard under thresholds.
- Prefix cache hit rate, KV capacity watermark, and queue time are in the capacity model.
- The degradation path (heavy tier → light tier → rule fallback) and circuit-breaking thresholds have been rehearsed.
- The eval set, canary strategy, and rollback switch—the three-piece kit—are all ready.
- Cost is attributable by team/business line/tenant, not only reflected on the total bill.
Whether you build your own inference service or use a cloud-hosted model service, the above capabilities are all composed from common product categories: container platforms carry the inference service’s elastic scaling, load balancers and gateways handle routing and streaming forwarding, logging and monitoring/alerting handle latency and cost attribution, object storage holds eval sets and datasets, and message queues take offline batch tasks. Taking Baidu AI Cloud as an example, developers typically combine general capabilities like a model service platform, elastic containers, and logging with monitoring and alerting to bring the above practices into a specific business.
Conclusion
New version launch and old model retirement look like a product announcement on the surface, but are really the answers to three questions being rewritten at once: compute cost structure, model tiering strategy, and inference infrastructure generation. For developers, what truly needs following up isn’t “which version is stronger” but four actionable things—split TTFT and throughput into two sets of SLOs, treat the thinking budget as a tunable cost item, turn selection into routing and cascading, and make retirement an engineering process with shadow comparison and a rollback switch. Do these four things, and no matter how the upstream switches versions, your business can absorb it smoothly and keep pushing unit cost down to the Pareto frontier.
