LLM Selection and Inference Optimization Under Compute Constraints
This article explores practical engineering approaches for selecting and optimizing Large Language Models (LLMs) when compute resources are constrained. It highlights the risks of relying on a single high-capability model and emphasizes the importance of balancing throughput, latency, concurrency, and cost. Key topics include separating TTFT and TPS metrics, implementing model tiering and routing, leveraging high/low reasoning modes, and designing effective capacity planning and degradation strategies to maintain system stability.
Recently, some LLM services have faced subscription pauses, quotas, or queues due to tight compute supply. For developers, this is not just about “unavailability” but exposes the risk of relying on a single high-capability model. While teams often default to the most powerful, parameter-heavy models, most real-world tasks do not require peak inference capability continuously. Instead, user experience and system stability are determined by a combination of throughput, Time to First Token (TTFT), concurrency capacity, task success rate, and cost. This article discusses LLM selection and inference optimization from an engineering perspective under compute constraints.
1. The Nature of Compute Constraints: Amplification Along the Call Chain
Compute tension first appears as restricted subscriptions or peak-time queues, but the real business impact is the chain reaction along the call chain: queueing increases TTFT, slower generation extends total task time, and retries/timeouts amplify concurrent load. This creates a vicious cycle where “slower responses lead to more retries, and more retries lead to slower responses.” Instead of simply blaming the model for being slow, we must address this systematically across capacity, task grading, calling strategies, and degradation mechanisms.
Architecturally, model calls should be abstracted into a replaceable capability layer, preventing business code from binding directly to a specific model or tier. Expose unified capability interfaces (e.g., generation, summarization, code completion, extraction) to the upper layer. A routing layer then selects the appropriate model based on task type, input length, latency targets, current load, and failure rates. This allows smooth degradation to lower-cost tiers when capacity is tight, rather than causing total business failure.
2. Key Metrics: Separating TPS and TTFT
When evaluating inference experience, TTFT (Time to First Token) and TPS (Tokens Per Second) must be evaluated separately. TTFT dictates the perceived “start of response” time, which is critical in streaming scenarios. TPS determines the total duration of long output tasks and affects the number of requests that can be served per unit time. High TTFT with acceptable TPS makes users feel they “wait forever to start,” while low TTFT with low TPS makes them feel it “starts fast but gets slower and slower.”
Engineering teams should set separate SLOs: interactive completion focuses on TTFT P95, long-form generation on total time and TPS, and batch processing on throughput and completion time. Monitoring metrics should at least include TTFT P50/P95/P99, generation speed, error rate, timeout rate, retry rate, token consumption per task, and task success rate. Separating these metrics is the only way to determine if issues stem from queueing, inference speed, excessive output, or retry amplification.
3. Model Tiering: Matching Tasks to Capabilities
Under tight compute constraints, model tiering is one of the most effective strategies. Tasks can be categorized into lightweight, standard, and deep types: Lightweight tasks (formatting, classification, short summaries, simple completions) prioritize low-latency, low-cost models. Standard tasks (routine code generation, document rewriting, information extraction) use a balanced tier. Deep tasks (complex reasoning, long-chain planning, cross-file understanding) are routed to the high-capability tier. Tiering is not just about price; it is a comprehensive judgment based on task risk, output length, failure cost, and latency targets.
A practical routing pseudocode looks like this:
def choose_model(task, sla, load):# First, check if the task is suitable for a lightweight modelif task.type in ["classify", "short_summary", "format"]:return "fast_model"# Interactive code completion is sensitive to first-token latencyif task.type == "code_completion" and sla.ttft_ms <= 800:return "fast_model"# Actively degrade under high load to avoid queue amplificationif load.utilization > 0.8:return "balanced_model_fast_mode"# Only complex tasks enter the deep tierif task.risk == "high" and task.expected_tokens > 2000:return "deep_model"return "balanced_model"
The routing layer should also support timeouts and retry limits. For instance, interactive requests should not retry indefinitely upon first-token timeout; they should quickly degrade or return an explainable prompt. Batch tasks can enter asynchronous queues, trading longer wait times for higher success rates. Caching is also crucial: for repeated template-based requests, fixed knowledge Q&A, and code snippet explanations, result caching significantly reduces compute consumption.
4. High/Low Reasoning Tiers: Mitigating Overthinking with Quality Trade-offs
Some services offer high/low reasoning tiers or fast/deep modes. The low reasoning tier typically reduces reasoning chain length, sampling budget, or internal thinking steps to lower latency and compute consumption, while the high reasoning tier improves the success rate of complex tasks. In practice, the low tier can mitigate the “overthinking” problem, allowing simple tasks to return faster, but it cannot fundamentally change inherently low generation speeds or high TTFT. Therefore, tier switching should be paired with task grading: simple tasks default to the low tier, while complex tasks explicitly upgrade. Upgrades should log the reason and success rate to prevent all requests from hitting the high tier.
A quality regression mechanism should also be established. After adjusting tiers, routing ratios, or timeout strategies, use a fixed evaluation set to compare task success rates, human preferences, and latency distributions. Do not just look at average latency; P95 and P99 better reflect user experience. For high-risk tasks, low-tier outputs should undergo rule validation or secondary confirmation; for low-risk tasks, a certain quality loss can be accepted in exchange for throughput.
5. Capacity Planning and Degradation: Turning “Unavailable” into a Manageable State
Under compute constraints, capacity planning must shift from “peak subscription” to “business-level allocation.” Identify core and non-core links, reserve throughput and quota for core links, and set up queueing, peak shaving, and asynchronous processing for non-core links. Use rate limiting and token buckets for sudden traffic to prevent retry storms. Set maximum token limits and segmented generation for long text outputs to reduce single-request occupancy time. Use window scheduling for batch tasks, executing them during off-peak hours.
Degradation strategies must be designed in advance: automatically switch to a balanced tier when the high-capability tier is unavailable; shorten prompts or reduce output length when TTFT exceeds the threshold; disable non-core features and retain core Q&A when error rates rise. Degradation is not a failure; it is pulling the system from an uncontrollable queueing state back to a serviceable state. All degradation actions should be logged with metrics for post-incident review.
6. Observability and Organizational Processes: Continuous Optimization
Model selection and inference optimization are not one-time tasks. Establish a unified observability dashboard to monitor TTFT, TPS, success rates, token consumption, and costs by task type, model tier, input length, and tenant. Implement canary releases for routing strategies, validating on a small traffic volume before scaling. Set up alerts for quota and subscription changes to spot capacity risks early. Business and platform teams should jointly maintain task grading standards and SLOs to prevent all requests from being flagged as the highest priority.
Ultimately, compute tension and subscription limits remind us that LLM applications require capacity, latency, and degradation design just like traditional distributed systems. Through task grading, fast/slow model routing, high/low reasoning tiers, timeout and retry control, caching, asynchronous processing, and comprehensive observability, an adjustable balance can be established among quality, latency, and cost, reducing the business risks associated with relying on a single model.
