Cutting LLM Inference Costs: Inference Engine Optimization and Performance Evaluation (TTFT & TPS)
This article explains why LLM inference prices and free-credit offers vary so widely across providers, and how to make rational engineering decisions beyond the headline numbers. It walks through the economics of generous free quotas, then details the core inference-engine techniques that drive unit cost down—continuous batching, KV cache management, quantization, speculative decoding, and prefix caching with chunked prefill. It also compares self-hosting against API forwarding, and introduces the key performance metrics you need to evaluate a service: TTFT (time to first token) and TPS (tokens per second), along with inter-token latency, throughput, and goodput. The article closes with practical guidance on model selection, tiering by task, avoiding the "quota illusion," and building your own reproducible evaluation baseline.
1. The Economics Behind Generous Free Credits
Lately, “credit wars” have become common around LLM inference services: some AI coding assistant platforms repeatedly raise free or discounted quotas within a short window, using highly attractive pricing to win over developers. On the surface, this looks like generous giveaways to users. But if you understand the cost structure of inference, there is a clear commercial and technical logic behind these quota adjustments.
First, be clear on one point: the cost of LLM inference is mainly determined by how long GPU compute is occupied. The more tokens processed per unit of time, and the less the GPU sits idle, the lower the provider’s marginal cost. So when a provider can substantially lower its unit inference cost, it has the confidence to offer higher quotas. Conversely, if a discounted quota stays above the actual compute cost for a long time, it usually means a “strategic loss” during a customer-acquisition or marketing phase—typically with a defined time limit, and not something to base long-term planning on.
For developers, the right attitude is not to be drawn in by the quota number itself, but to ask whether that quota is actually sufficient under real business load. The same nominal quota can support conversations that differ by an order of magnitude depending on the model, concurrency, and context length. The following sections break down how cost and performance interact from a technical standpoint.
2. Inference Engine Optimization: The Real Driver of Lower Cost
Why can the same class of model be priced so differently by different providers? The core difference usually lies not in the model itself, but in the inference engine. An inference engine that has been deeply optimized for a specific model architecture can multiply throughput several times over while barely sacrificing output quality, pushing unit cost very low. Here are several optimization techniques commonly used across the industry.
2.1 Continuous Batching and Iteration-Level Scheduling
Traditional static batching requires a batch of requests to start and finish together. As long as one request in the batch is still generating, the entire batch’s resources stay occupied, leaving GPU utilization low. Continuous batching (also called iteration-level scheduling) refines the scheduling granularity down to each decoding step: finished requests exit immediately, and waiting new requests slot in at any time. This keeps the GPU from sitting idle and significantly boosts throughput.
2.2 KV Cache Management
Autoregressive generation needs to cache the Key/Value tensors of historical tokens. With contiguous memory allocation, long sequences easily create fragmentation, wasting GPU memory and limiting concurrency. Paged cache management borrows the paging mechanism from an operating system’s virtual memory: it slices the KV cache into fixed-size blocks allocated on demand, greatly reducing fragmentation and increasing the number of concurrent requests that can be held. For high-concurrency online services, this is often a major source of cost difference.
2.3 Quantization
Compressing model weights and activations from FP16/BF16 down to lower precision such as INT8 or INT4 reduces memory usage and speeds up both memory access and computation. Quantization falls into two categories: post-training quantization (PTQ) and quantization-aware training (QAT). The former is fast to deploy and low in cost; the latter better preserves accuracy under extreme low-bit settings. Quantization is a typical case of “trading a small loss in precision for a significant reduction in cost.”
2.4 Speculative Decoding
A lightweight draft model quickly generates several candidate tokens, which the large model then verifies in a single parallel pass. If they match, they are accepted directly; if not, the result falls back. Because verifying multiple tokens is far cheaper than generating them serially one by one, speculative decoding can reduce latency and improve throughput while barely affecting the output distribution.
2.5 Prefix Caching and Chunked Prefill
Many scenarios (such as conversations with a fixed system prompt or fixed context) share a common prefix. Prefix caching can reuse the already-computed KV for this portion, avoiding redundant computation. Meanwhile, chunked prefill splits the prefill of a long prompt into small chunks and interleaves them with decoding, easing the impact of long inputs on time to first token.
These techniques are usually combined. The true difference in a mature inference engine lies precisely in whether these optimizations can be tailored to the specific model architecture.
3. Self-Hosting vs. API Forwarding
The quotas and discounts a provider offers are closely tied to whether it self-hosts its infrastructure.
- API forwarding mode: The provider forwards requests to the upstream official API and only earns a margin or provides a wrapper layer. This model is fast to launch and light to operate, but its costs are constrained by upstream pricing and discount policies. Once the upstream tightens discounts, the provider’s room to offer deals narrows quickly—which also explains why some quotas are conservative in the early stage.
- Self-hosting mode: The provider builds and operates its own inference cluster. The upfront investment is large, and it must handle engineering challenges like scaling, load balancing, and fault tolerance. But once it completes optimization for the target model, it controls its own cost and can offer more aggressive discounts.
One caution: self-hosted services need extra attention on stability. A self-built cluster may experience request queuing, latency jitter, and even intermittent failures during traffic spikes, GPU memory pressure, or backend anomalies. For production environments, whether automatic retries, timeout fallback, and backup-model switching are in place often matters more than the size of the quota.
4. How to Evaluate Inference Performance: TTFT and TPS
Beyond quotas, what truly affects the user experience is performance. When evaluating an inference service, you should look at at least two core metrics:
- TTFT (Time To First Token): The time from sending a request to receiving the first token. It is mainly determined by queuing time and prefill computation, and is especially critical for interactive conversations, coding assistance, and agent scenarios—if the first token takes too long, users feel the service has “frozen.”
- TPS (Tokens Per Second): The generation speed during decoding, which determines how fast text streams out and affects the reading of long text and overall smoothness.
You can also look at inter-token latency (ITL/TPOT), overall system throughput (which depends on concurrency), and goodput (the real output after excluding requests that time out or fail).
It is worth stressing that any performance figure is strongly correlated with the specific context length, concurrency level, and input/output ratio. Claiming that “a certain model is faster” without these conditions is not rigorous. To illustrate, under identical load the comparison looks like this (the following is a conceptual example to aid understanding, not measured data from a specific product):
| Scenario | TTFT | TPS | Perceived Experience |
|---|---|---|---|
| Short input, low concurrency | Low | High | Smooth interaction |
| Long input, high concurrency | High | Medium | Noticeable wait for first token |
The truly scientific approach is to run a load test with your own typical prompts, real concurrency, and target context length, and measure the distribution of TTFT and TPS. Do not look only at averages—P95/P99 often better reflect the real experience during peak periods.
5. Practical Advice on Model Selection
Back to engineering decisions: when facing multiple models that differ in quota, speed, and capability, how do you choose?
- Tier selection by task. For latency-sensitive, interaction-heavy scenarios like coding and agents, prioritize models with low TTFT and good stability; for throughput-sensitive tasks like offline batch processing and long-form generation, focus on TPS and unit cost.
- Watch out for the “quota illusion.” Convert the nominal quota into “actual supportable conversations/requests” before comparing across options, so the numbers don’t mislead you.
- Consider capability boundaries. If your business needs multimodal abilities such as image-and-text understanding, you can’t look only at text-only models; the rework cost caused by missing capabilities often far exceeds the price difference.
- Have a stability safety net. For critical paths, preparing backup models plus automatic switching, retries, and fallback strategies matters more than chasing a single low price.
- Build your own evaluation baseline. Use a fixed evaluation set and load-test scripts to regularly measure TTFT, TPS, success rate, and output quality, forming a reproducible basis for selection instead of relying on subjective impressions.
Conclusion
Large free-credit offers are an effective way to attract users, but they are neither a long-term cost commitment nor a guarantee of a better experience. Understanding how an inference engine drives down unit cost through continuous batching, cache management, quantization, and speculative decoding, seeing clearly the tradeoffs between self-hosting and API forwarding, and using metrics like TTFT and TPS to build your own evaluation baseline—this is the foundation for making rational decisions in a fast-moving LLM service market. If you need inference capabilities that are stable, controllable, and elastically scalable, deploying a self-hosted inference service on a general-purpose container platform, paired with logging and monitoring/alerting for continuous observability, is a technical path worth serious evaluation.
