LLM Inference Speed and Cost Optimization: A Technical Breakdown and Engineering Playbook

1. First, align on the metrics: what does “faster output” actually mean?

When a frontier model is refreshed, the marketing typically highlights two numbers: content produced per unit of time goes up, and the cost per token goes down. For engineering teams, though, neither of these is a single metric. Conflating them leads directly to wrong capacity planning and cost budgets.

Four quantities to keep separate:

• TTFT (Time To First Token): the time from request submission to receiving the first meaningful content increment, determined mainly by the prefill stage (processing the input context) and queue wait time. It drives the “clicked but nothing happened” feel of interactive applications.
• TPOT / ITL (Time Per Output Token / Inter-Token Latency): the average interval between adjacent tokens during decoding, which determines “how fast the text pops out.”
• End-to-end latency: approximately TTFT + output length × TPOT. Long-form generation is extremely sensitive to TPOT; short Q&A is extremely sensitive to TTFT.
• Aggregate throughput: the total tokens produced per unit time across a machine or a cluster, which determines unit cost. This is an independent optimization goal from single-user perceived speed, and under high-concurrency load tests the two often trade off against each other.

So “faster generation” might come from decoding kernel optimization, or from batch scheduling and cache reuse — and these carry completely different engineering implications. The first thing to do is always to establish your own baseline rather than accept an external conclusion. Below is a minimal benchmark script that depends only on the standard library and is gateway-agnostic:

import json, time, urllib.request
URL = "https://<your-gateway>/v1/chat/completions"   # self-hosted gateway address
def bench(prompt, max_tokens=512, runs=5):
    ttfts, tpots, e2e = [], [], []
    for _ in range(runs):
        body = json.dumps({
            "model": "<model-id>",
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
            "stream": True,
        }).encode()
        req = urllib.request.Request(
            URL, data=body,
            headers={"Content-Type": "application/json",
                     "Authorization": "Bearer <API_KEY>"},
        )
        t0 = time.perf_counter()
        first, last, n = None, t0, 0
        with urllib.request.urlopen(req) as resp:
            for raw in resp:
                line = raw.decode("utf-8", "ignore").strip()
                if not line.startswith("data:"):
                    continue
                payload = line[5:].strip()
                if payload == "[DONE]":
                    break
                now = time.perf_counter()
                delta = json.loads(payload)["choices"][0].get("delta", {})
                if delta.get("content"):
                    if first is None:
                        first = now
                    last, n = now, n + 1
        t_end = time.perf_counter()
        if first is None or n < 2:
            continue
        ttfts.append(first - t0)
        tpots.append((last - first) / (n - 1))
        e2e.append(t_end - t0)
    def stats(xs):
        xs = sorted(xs)
        return {"p50": round(xs[len(xs) // 2], 3),
                "p95": round(xs[max(int(len(xs) * 0.95) - 1, 0)], 3)}
    return {"ttft": stats(ttfts), "tpot": stats(tpots), "e2e": stats(e2e),
            "tokens_per_sec_p50": round(1 / tpots[len(tpots)//2], 1) if tpots else None}
if __name__ == "__main__":
    print(json.dumps(bench("Summarize the consistency strategies of distributed caching in 300 words"), ensure_ascii=False, indent=2))

Two implementation points: first, you must report percentiles rather than averages — P95/P99 reflect the real user experience; second, concurrency must be measured separately from single requests — TPOT at concurrency 1 and TPOT at concurrency 64 are usually two orders of magnitude apart.

2. Why decoding can be sped up: from compute characteristics to scheduling efficiency

2.1 The bottleneck in decoding is memory access, not compute

Autoregressive decoding generates only one token per step, at which point matrix multiplication degenerates into vector-matrix multiplication with very low arithmetic intensity — a classic memory-bound workload. This explains two things: why simply piling on more compute yields limited gains, and why “trade one memory access for more tokens” is the main line of speedup. All mainstream optimizations reduce to this one principle.

2.2 Sparse activation: trading total parameter count for per-forward cost

Mixture-of-experts architectures have a very large total parameter count, but each token activates only a subset of the experts, reducing the weights and compute that must be moved in a single forward pass. The tradeoffs are equally clear: higher memory footprint, uneven expert-routing load, and extra communication from cross-device expert parallelism. When evaluating these models, watch “memory per card” and “tail latency” together.

2.3 The decoding-algorithm layer: verify multiple tokens per forward pass

Speculative decoding uses a lightweight draft model (or the model’s own multi-token prediction heads) to guess several tokens, then has the main model verify them in parallel; on acceptance it advances multiple steps at once. It amortizes the serial memory-access bottleneck across multiple tokens, and the actual speedup depends on draft hit rate and average acceptance length — both highly dependent on your business corpus distribution, so someone else’s speedup ratio cannot be applied directly to your scenario. Tree-based candidate verification is an extension of the same idea.

2.4 Batching and memory management

• Continuous batching: request-level dynamic swap in/out, eliminating the head-of-line blocking where “short requests wait for long requests” in static batches; the most significant improvement to throughput.
• Paged KV cache management: slicing the KV cache into fixed blocks allocated on demand, greatly reducing fragmentation and reservation waste, which is equivalent to raising the number of concurrent requests you can hold.
• Chunked prefill: splitting the prefill of long inputs into chunks interleaved with decode, preventing a single very-long-context request from stalling the whole batch’s decode; the key technique for improving TTFT P99.

2.5 Prefix reuse

System prompts, few-shot examples, and frequently hit document fragments in RAG are often repeated verbatim across many requests. Once the prefix cache hits, these tokens don’t need to be re-prefilled — a certain-gain, very-low-risk optimization. The engineering task is to move stable content to the front of the prompt and keep it byte-for-byte identical.

2.6 Quantization and compilation optimization

Low-bit weights and FP8 compute, KV cache quantization, operator fusion and graph-level compilation, using graph capture to reduce kernel launch overhead, and overlapping communication and compute across tensor/pipeline/expert parallelism are all general techniques for raising effective compute. It must be stressed: the impact of these optimizations on output quality has to be measured empirically — quantization error can be amplified in multi-step reasoning, code generation, and numerical tasks, and cannot be replaced by a generic leaderboard in place of a business-specific evaluation.

3. Cost reduction requires two accounts

3.1 Pricing structure: why input and output are priced separately

Per-token billing has become the industry norm, and it typically tiers input and output separately, with the input unit price noticeably lower than the output unit price. The reason lies in the cost structure itself: the prefill stage can be massively parallelized with high compute utilization, whereas the decode stage is serial, memory-bound, and requires holding memory for a long time, so its marginal cost is higher. On top of this, cache-hit discounts, batch/async call discounts, and context-length tiers are also common pricing dimensions. For a specific service’s unit prices and discount rules, refer to the official billing documentation; this article does not cite any specific numbers as fact.

3.2 The unit cost formula and adjustable levers

For self-hosted or managed inference, the cost per token can be approximated as:

unit cost per token ≈ (accelerator hourly cost × number of cards × runtime) / effective tokens produced

The word “effective” in the denominator is the key. The available improvement levers, roughly ordered by return on investment, are: raising the cache hit rate > raising the effective concurrent batch size > trimming redundant context > reducing retries and failure rate > quantization and compilation optimization > switching to a more suitable model size.

3.3 A worked example: a unit-price reduction is not the same as a bill reduction

IN_PRICE, OUT_PRICE = 8.0, 40.0     # illustrative pricing assumptions, for demonstrating the calculation only
CACHE_RATE = 0.1                    # cached portion billed at 10% (a common industry pattern)
OLD_FACTOR = 1.25                   # assume the previous generation was 25% more expensive
def cost(in_tok, out_tok, unit_in=IN_PRICE, unit_out=OUT_PRICE, hit_ratio=0.0):
    billable_in = in_tok * (1 - hit_ratio) + in_tok * hit_ratio * CACHE_RATE
    return billable_in / 1e6 * unit_in + out_tok / 1e6 * unit_out
# Scenario A: customer-service Q&A, long input short output, system prompt and knowledge-base fragments highly repetitive
in_a, out_a = 4000, 300
old_a = cost(in_a, out_a, unit_in=IN_PRICE*OLD_FACTOR, unit_out=OUT_PRICE*OLD_FACTOR)
new_a = cost(in_a, out_a, hit_ratio=0.6)
# Scenario B: copywriting, short input long output, almost no reusable prefix
in_b, out_b = 300, 3000
old_b = cost(in_b, out_b, unit_in=IN_PRICE*OLD_FACTOR, unit_out=OUT_PRICE*OLD_FACTOR)
new_b = cost(in_b, out_b, hit_ratio=0.0)
for name, o, n in (("A: long input, short output", old_a, new_a), ("B: short input, long output", old_b, new_b)):
    print(f"{name}: {o:.6f} -> {n:.6f} USD per call, reduction {(1-n/o)*100:.1f}%")

The point this example is meant to make: the same “unit-price reduction” can produce a bill reduction that differs by more than 2× across different input/output ratios. For long-input businesses with a high cache hit rate, the actual reduction will clearly exceed the unit-price reduction; for output-dominated businesses, the reduction basically tracks the unit price. So cost budgeting must be modeled on your own real token distribution, not derived by applying a unit-price percentage.

Three easily missed cost items: token loss from failed and retried calls, context bloat from un-trimmed conversation history, and putting tasks that should be batch-processed asynchronously on the real-time path and thereby forgoing the discount.

4. Turning a model upgrade into reproducible gains: a deployment checklist

4.1 Inference gateway abstraction layer

Inserting a self-built gateway between the business and the model pays off far more than the single upgrade itself:

• Unify request/response structures and streaming protocols, hiding interface differences across inference backends;
• Model registry + routing policy, selecting the model size by task type, context length, priority, and remaining quota;
• Circuit breakers, timeouts, degradation, and fallback chains, ensuring a single model’s failure doesn’t amplify into a business failure;
• Canary releases and shadow traffic, comparing new vs. old model quality and cost with real traffic.

4.2 Tiered caching strategy

Exact caching keyed by the hash of “normalized prompt + sampling parameters” is safe and controllable; semantic caching matches historical Q&A via vector similarity, with a higher hit rate but a risk of false hits. On deployment you must pair it with three constraints: a conservatively chosen similarity threshold, a reasonable TTL, and cache isolation by tenant and permission — otherwise you’ll return user A’s answer to user B. For scenarios involving personalization, financial, or medical information, it’s recommended to enable only exact caching.

4.3 Context engineering

The most direct way to cut cost is often to reduce the input rather than switch models: trim redundancy from system prompts, change few-shot from “many and complete” to “precise and accurate,” have RAG rerank and deduplicate and keep only high-scoring fragments, and summarize multi-turn conversation history instead of concatenating it in full. It’s advisable to make context-length distribution a monitored metric and to specifically address P99 over-long requests.

4.4 Full-chain observability

At minimum you need to cover: QPS and concurrency, TTFT P50/P95, TPOT P50/P95, error rate and retry rate, cache hit rate, and per-request token consumption with converted cost. On supporting capabilities, you can build this system out of common cloud product categories — for example, use a logging service to centrally collect and search structured request logs (be sure to desensitize), use monitoring and alerting to set thresholds for cost anomalies and latency degradation, use object storage to archive desensitized samples for problem replay and evaluation-set iteration, use a message queue to smooth out batch offline jobs and get async pricing, and use a container platform to host inference services with elastic scaling. On platforms such as Baidu AI Cloud, these capabilities are standard, general-purpose building blocks that are decoupled from the specific model choice, which makes later replacement easier.

4.5 Evaluation methodology: don’t let “speedup” come at the cost of quality

1. Offline evaluation set: sampled from real traffic, augmented with hard-case mining and human-annotated preference pairs; metrics should include at least task success rate, format compliance rate, factual consistency (hallucination rate), and human win rate.
2. Online A/B: bucket by user or request, observing both quality metrics and cost/latency metrics at the same time to avoid single-objective optimization.
3. Freeze SLOs: write latency and cost as rollback-able thresholds — for example, “TTFT P95 must not exceed 1.2× the previous version, and task success rate must not drop by more than 1 percentage point” — with automatic rollback on breach.
4. Pareto view: choose the lowest cost at the same quality level, and the highest quality at the same cost budget, rather than looking at a single leaderboard score.

5. Common pitfalls and risks

• Looking only at the token unit price, not the input/output ratio, causing budgets to deviate seriously from measurements.
• Reporting latency with averages, hiding the tail-latency degradation caused by long-context requests.
• Equating “decode speedup” with end-to-end gains: gateway forwarding, authentication, network transfer, frontend rendering, and business orchestration can all be the real bottleneck — segment the timing before optimizing.
• Ignoring vendor lock-in: sticking with a gateway abstraction + your own evaluation set is the prerequisite for preserving bargaining power and migration feasibility.
• Privacy and compliance: desensitize request and response logs by default, define data retention periods and access permissions explicitly, and prefer private or dedicated deployment forms for sensitive scenarios.
• Cache consistency: after a document is updated, old answers may still hit the cache — establish version numbers for knowledge-type content and include them in the cache key.

6. Conclusion

The iteration of frontier models in decode efficiency and unit cost is continuous, but the dividend is only realized by an engineering system that is “observable, measurable, and rollback-able.” Start with three small things: establish a baseline for your business’s TTFT/TPOT/cost, deploy an inference gateway abstraction layer, and run an A/B once with dual-metric gating. After these three steps, the next model upgrade becomes just a configuration change rather than a rewrite.