Understanding LLM Inference Performance: A Guide to TTFT, TPS, and Model Selection

1. Compute Constraints: The New Normal

Recently, many developers have encountered “subscription pauses” or “rate limiting” on popular large language model services. While these appear to be operational strategies, they fundamentally reflect a supply-demand imbalance:

• Inference is a resource-intensive workload: Large parameter models typically require multi-GPU parallelism. VRAM capacity and memory bandwidth form hard constraints, causing inference costs to scale almost linearly with concurrency.
• Demand outpaces scaling: GPU procurement, datacenter construction, and cluster tuning have objective lead times, whereas user growth can surge within weeks.
• Capacity management is inevitable: Providers must balance “existing user experience” against “new paying users.” Rate limiting, halting sales, and raising access thresholds are common operational tactics.

For developers, this yields two key takeaways. First, no single model service should be treated as a 100% reliable dependency; architectures must reserve redundancy and fallback paths. Second, rather than chasing the “latest and greatest” models, it is better to establish a selection methodology based on performance metrics and task matching—which is the core focus of this article.

2. Two Core Metrics: TTFT and TPS

To discuss whether inference is “fast,” we must first decompose the vague concept of “slow” into measurable metrics.

2.1 TTFT (Time to First Token)

Definition: The time elapsed from when a client sends a request to when it receives the first output token. It generally consists of three parts:

• Queuing time: When server load is high, requests wait in the scheduling queue.
• Prefill time: The process where the model performs a forward pass on the input prompt to generate the KV Cache. Longer inputs and tighter compute resources increase this duration.
• Network and scheduling overhead: Fixed costs like gateway authentication, routing, and connection establishment.

TTFT directly determines “perceived responsiveness.” In interactive scenarios (AI coding assistants, chatbots), users are far more sensitive to first-token latency than total duration. Receiving the first token in 1 second followed by a steady stream of text provides a much better experience than staring at a blank screen for 10 seconds before receiving the entire output at once.

2.2 TPS (Tokens Per Second)

Sometimes expressed as its inverse, TPOT (Time Per Output Token). It primarily depends on:

• Model parameter size and memory bandwidth: The decoding phase is a classic memory-bound workload. Bandwidth is often the bottleneck; larger parameters result in lower output per unit of time.
• Batching strategy: To improve overall throughput, servers use continuous batching. Larger batches reduce the generation speed allocated to individual requests.
• Output length and sampling configuration.

TPS dictates the “wait time for long tasks.” When generating large code blocks or long documents, the weight of TPS in the overall experience increases dramatically.

2.3 Approximating End-to-End Latency

A practical estimation formula:

Total Latency ≈ TTFT + (Output Token Count / TPS)

Let’s look at the magnitude of difference using illustrative numbers. Assume Model A has a TTFT of 1s and a TPS of 60; Model B (with deep thinking) has a TTFT of 5s and a TPS of 20. Both generating 800 tokens of code:

• Model A: 1 + 800/60 ≈ 14 seconds
• Model B: 5 + 800/20 ≈ 45 seconds

If the thinking process additionally generates hundreds or thousands of hidden tokens, the gap widens further. This explains a common phenomenon: larger, higher-ranking models can feel so slow in actual coding experiences that users want to switch back.

3. Thinking Modes and the “Overthinking” Problem

3.1 Design Logic of Dual Modes

Many next-generation inference models offer both “deep thinking” and “fast response” modes. Essentially, these assign different thinking budgets to the model:

• Thinking mode: Generates a longer chain-of-thought internally (invisible or minimally exposed to the user) before outputting the final answer, trading time for accuracy.
• Fast mode: Compresses or skips the chain-of-thought to output results directly, significantly reducing latency.

3.2 What is “Overthinking”?

Overthinking occurs when a model generates a verbose chain-of-thought even for simple tasks. For example, if asked to rename a variable or add a comment, it might still “think deeply” for hundreds of tokens. This has three consequences:

• Token consumption multiplies, increasing costs in usage-based billing.
• End-to-end latency extends significantly, degrading the interactive experience.
• Server compute is occupied by low-value inference, exacerbating overall load pressure—which is another driver of compute shortages.

It is worth noting that offering dual modes only “mitigates” overthinking; it does not change the model’s inherent generation speed limits. A low TPS or slow TTFT are architectural and resource-level constraints that mode switching cannot fundamentally resolve.

3.3 Coping Strategies

• Mode routing: Route simple tasks to fast mode; enable thinking mode only for complex tasks.
• Prompt constraints: Explicitly requiring the model to “give the conclusion directly” or “do not expand on derivation steps” can effectively suppress redundant thinking.
• Selection evaluation criteria: Include whether a model service allows disabling or tightening the thinking budget, or can automatically judge task difficulty, as a key evaluation dimension.

4. Model Selection: Task Matching Over Leaderboard Rankings

4.1 Limitations of Benchmark Scores

Public evaluation sets mostly consist of static questions and are repeatedly used for “score grinding” by various models. They rarely reflect the pass rates, latency, and stability under real-world workloads. Leading a leaderboard does not mean leading on your specific tasks.

4.2 Tiering Coding Tasks

• Lightweight tasks: Inline completion, small function generation, comments and naming, formatting, and refactoring. Mature mid-tier code models or fast modes are more than sufficient and offer distinct speed advantages.
• Medium tasks: Cross-file modifications, scaffolding for common frameworks, unit test completion. Mid-tier models combined with good context engineering are usually adequate.
• High-difficulty tasks: Tricky bug localization, concurrency and boundary condition design, algorithmic optimization. These are the only cases worth routing to flagship models or deep thinking modes.

In practice, the vast majority of daily requests fall into the first two categories. This means the engineering intuition that “a mature code model is sufficient for most tasks at hand” is accurate. There is no need to make all requests bear the latency and cost of a flagship model just to accommodate a few high-difficulty tasks.

4.3 Establish Internal Evaluations and Routing Tables

It is recommended to maintain a “Task Type → Model/Mode” routing table and continuously calibrate it with your own data:

Task Type	Recommended Configuration	Key Metrics
Completion/Renaming/Comments	Fast mode/Lightweight model	TPS, Suggestion Acceptance Rate
Feature Implementation/Refactoring	Standard mode	First-Time Pass Rate
Tricky Debugging/Design	Thinking mode/Flagship model	Accuracy, Explainability

Evaluation sets should be derived from real historical tasks, tracking first-time pass rates, average TTFT, TPS, and cost per request, rather than just looking at demo effects.

5. Engineering Optimization Checklist for Compute-Constrained Environments

1. Streaming output: Always enable streaming APIs to let first-token time dominate perceived responsiveness. Pair this with progressive frontend rendering to alleviate waiting anxiety.
2. Prompt slimming and stable prefixes: Place fixed content like system prompts and tool descriptions in prefixes. This allows the server to hit the KV Cache, directly reducing Prefill time.
3. Context management: Inject only the code snippets relevant to the current task to control input length; TTFT will improve accordingly.
4. Result caching: Cache and reuse idempotent generation requests (fixed documents, template code) to reduce redundant inference.
5. Multi-model redundancy and fallback: Automatically switch to a backup model/mode when the primary model is unavailable or its timeout rate exceeds thresholds, avoiding single-point dependency.
6. Asynchronous processing: Shift non-real-time tasks like code review summaries and batch document generation to asynchronous queues to avoid peak hours.
7. Metrics monitoring: Build dashboards and alerts for TTFT, TPS, timeout rates, and error rates. Use data to drive the iteration of routing strategies.

Mainstream cloud platforms’ model services generally provide streaming APIs, usage monitoring, and multi-model access capabilities (for example, Baidu AI Cloud and other cloud providers offer corresponding categories). You should prioritize reusing these foundational capabilities to build the mechanisms above, reducing custom development costs.

6. Conclusion

“Compute shortages and sales pauses” remind us that both the capability boundaries and supply stability of model services are dynamic variables. Rather than agonizing over whether to buy the most expensive new model, it is better to return to engineering fundamentals. Decompose “slow” into TTFT and TPS; decompose “strong” into first-time pass rates on real tasks. Then, combine them using routing, caching, streaming, and fallback strategies. When a new model is released, use the same evaluation process to quickly answer three questions: Does it deliver genuine quality gains on your tasks, at an acceptable latency and cost? Once you clarify these three questions, model selection becomes effortless.