LLM Selection Guide: Balancing Inference Performance and Cost

    As large language models (LLMs) move into production at scale, inference performance and cost per call directly shape both product experience and unit economics. This guide explains how to read TTFT (time to first token) and TPS (tokens per second), and why real performance should be measured under your own business workloads rather than taken from theoretical peaks. It also walks through tiered routing between lightweight and flagship models, the architecture and infrastructure techniques behind low latency and high throughput, and the hidden cost of reasoning tokens in "thinking" models. The guide closes with four practices for balancing speed, quality, and cost in production.

    1. Why Inference Performance and Cost Need to Be Revisited

    As generative AI moves into large-scale production, large language models are no longer black boxes that only need to run. They have become core elements that directly determine both product experience and the unit economics of an application. A model that responds slowly, delivers insufficient throughput, or costs too much per call can quickly erode the competitiveness of an otherwise excellent application.

    When selecting a model, developers repeatedly weigh three dimensions: response speed, output quality, and cost per call. These three are not in a simple, linear, positive correlation. Understanding how they interact and constrain one another is the prerequisite for sound architecture decisions.

    This is especially true in high-concurrency online scenarios. Every additional few hundred milliseconds of latency can visibly hurt user retention and conversion. Inference cost, meanwhile, is a fixed, ongoing operational expense that directly erodes profit margins.

    2. Understanding Inference Performance: What TTFT and TPS Actually Mean

    Two metrics are the most critical for measuring LLM inference performance:

    • TTFT (Time To First Token): The time from when a request is sent to when the first output token arrives. TTFT determines how long users wait before they see the model start responding, which is critical for interactive scenarios such as chat and code completion.
    • TPS (Tokens Per Second): The speed at which the model streams tokens once the first token has been returned. TPS determines how long it takes to generate a full response, which shapes the experience in long-text and long-answer scenarios.

    An ideal service achieves low TTFT plus high TPS. For example, it keeps first-token latency in the hundreds of milliseconds and sustains throughput of hundreds of tokens per second, so users barely notice any lag. However, pushing both metrics down at the same time is not easy. It depends heavily on coordinated optimization across the model architecture and the inference infrastructure.

    The sample code below can be used for load testing. It helps teams quantify real-world performance under their own business workloads:

    1. import time
    2. def measure_latency(client, messages):
    3. start = time.perf_counter()
    4. first_token_time = None
    5. token_count = 0
    6. stream = client.chat.completions.create(
    7. model="your-model-name",
    8. messages=messages,
    9. stream=True,
    10. )
    11. for chunk in stream:
    12. if chunk.choices and chunk.choices[0].delta.content:
    13. if first_token_time is None:
    14. first_token_time = time.perf_counter()
    15. token_count += 1
    16. end = time.perf_counter()
    17. ttft = (first_token_time - start) * 1000 if first_token_time else None # ms
    18. total = end - first_token_time if first_token_time else 0
    19. tps = token_count / total if total > 0 else 0
    20. return {"ttft_ms": ttft, "tps": round(tps, 1), "tokens": token_count}

    One point deserves emphasis: performance figures should be measured against your own business workloads, not taken from the theoretical peak numbers a vendor claims. Input length, concurrency level, and batching strategy can all significantly change the final results.

    3. Tiered Selection: Lightweight Fast Models and High-Performance Models Have Distinct Roles

    LLM services typically offer model variants in different size tiers, forming a clear spectrum:

    • Lightweight fast models: Strong in inference speed and cost efficiency. They suit high-frequency, low-complexity, latency-sensitive tasks such as intent recognition, simple Q&A, text summarization, and classification and tagging.
    • High-performance flagship models: Strong in inference quality and complex-task handling. They suit high-value scenarios such as long-chain reasoning, complex code generation, multi-step planning, and in-depth domain-specific analysis.

    A mature selection strategy has the two tiers work together rather than choosing one or the other. When lightweight models already cover most requests, sending all traffic to an expensive flagship model wastes budget. It also compromises service stability under high load. A more sensible approach is tiered routing: lightweight models handle the mainstream traffic, and only genuinely complex requests are escalated to the flagship model.

    1. def route_request(request, lightweight_model, heavy_model):
    2. # Coarse-grained classification based on task length, intent complexity, etc.
    3. complexity = estimate_complexity(request)
    4. if complexity < 0.4:
    5. return lightweight_model # Simple tasks go to the lightweight model
    6. return heavy_model # Complex tasks are escalated to the high-performance model

    What makes this tiered architecture work is the accuracy of the complexity estimate. A threshold set too low degrades output quality. A threshold set too high fails to deliver real savings. It therefore needs continuous tuning against annotated business data.

    4. Architecture and Infrastructure: The Foundation of High Throughput and Low Latency

    Reaching hundreds of milliseconds to first token and hundreds of tokens per second sustained usually takes more than the model alone. It requires close coordination between the inference architecture and the underlying infrastructure. Common industry techniques for accelerating inference include:

    • KV Cache and request scheduling: Reuse attention caches to avoid redundant computation. Combine this with continuous batching to improve GPU utilization.
    • Model quantization and operator optimization: Use lower-precision inference, operator fusion, and similar techniques to gain throughput while keeping quality loss under control.
    • Distributed inference and elastic scaling: Use container platforms to scale inference services horizontally. Use monitoring and alerting together with automatic scaling to handle traffic peaks and troughs.
    • Cache-layer reuse: Cache results for high-frequency identical requests to reduce redundant inference overhead.

    An older model version that has fallen clearly behind architectural advances can have its cost-efficiency overtaken by a newer lightweight model, even with a larger parameter count. The efficiency gains from a generational gap in architecture can sometimes outweigh a pure advantage in model scale.

    5. An Overlooked Cost Variable: The “Thinking” Overhead of Reasoning Models

    The “reasoning” or “thinking” models that have emerged in recent years generate large amounts of intermediate reasoning (reasoning tokens) before producing their final answer. This depth of thinking genuinely improves accuracy on complex math, logic, and coding tasks. However, it also creates a cost trap that is easy to overlook: excessive thinking inflates token consumption per call, which can offset the experience gains from higher speed.

    When estimating cost, you therefore cannot look only at “unit price × output length.” Tokens consumed during the thinking phase must be counted toward total spend as well. In practice, several strategies can help:

    • Toggle thinking mode dynamically based on task difficulty: Disable deep reasoning for simple tasks and enable it only for complex ones.
    • Set an upper limit on the thinking budget: Cap the maximum number of thinking tokens to prevent endless self-deliberation.
    • Build a cost monitoring dashboard: Measure TTFT, TPS, and input, output, and thinking token counts separately to create an attributable view of cost.

    6. Practical Recommendations for Your Business

    Overall, model selection should not be a single-choice question of “which model is the strongest.” It is an engineering methodology built on four practices:

    1. Establish metric baselines: Use load tests with real business workloads to measure TTFT and TPS, rather than relying on theoretical peak numbers.
    2. Deploy in tiers: Route requests between lightweight and high-performance models so that user experience and cost are balanced.
    3. Quantify thinking cost: Factor reasoning tokens into your budget so that faster responses do not come at a disproportionate price.
    4. Monitor and iterate continuously: Use logging services and monitoring and alerting to track changes in quality and spend, and dynamically return to the optimal configuration.

    On platforms such as Baidu AI Cloud, organizations can leverage common capabilities such as a model service platform, object storage, message queues, and monitoring and alerting. These let them quickly build a complete pipeline from load testing and canary rollout to elastic scaling. This turns the balance of “speed, quality, and cost” from a slogan into measurable engineering practice.

    Ultimately, the best model is not the one with the largest parameter count. It is the one that can consistently deliver an acceptable experience within your business constraints while keeping unit cost under control.