Why the Same Inference Hardware Makes Some LLMs Profitable and Others Not
Running two large language models on identical inference hardware can produce very different unit costs, but a low cost does not guarantee a profit. This article explains why benchmark throughput and business profitability answer two different questions, and how to build a fair, production-like evaluation. It covers controlling for model scale while verifying task quality, separating input processing from output generation, measuring sustainable throughput under representative traffic, validating optimizations one variable at a time, and calculating profit and loss through effective task cost and break-even analysis. Readers will learn how to move from a test report to a defensible deployment decision based on quality, latency, and cost under real traffic.
1. Performance Testing and Cost Analysis Answer Different Questions
It is not surprising that running two models on the same inference hardware produces different unit costs. Differences in scale, architecture, and how context is handled can change how much compute, GPU memory, and inter-device communication each inference call consumes. But several conditions sit between “running faster” and “making money”: whether the output actually solves the problem, whether the service can meet its latency commitments under peak traffic, and whether real revenue covers long-term operating costs.
If a test publishes throughput at a single concurrency level without stating the model configuration, input/output lengths, quality threshold, latency constraints, and cost basis, you cannot use it to judge whether the deployment is profitable. Even when a configuration measures as expensive under the current workload, that only shows the configuration is at a disadvantage for that given load; it does not prove that an entire class of models is unsuitable for deployment. A verifiable conclusion should state explicitly: under what task, what traffic, and what service constraints one deployment approach has a lower delivery cost.
2. When Comparing Models, Control for Scale and Verify Task Quality
Parameter scale helps shortlist candidate models, but it does not by itself explain inference cost. For dense models, weight size is usually closely tied to compute and memory requirements. For models that activate only part of their experts per input, total parameter count better reflects the pressure of keeping weights resident, while the parameters actually activated per call are only a starting point for estimating compute. How experts are distributed across devices, whether request routing is balanced, and how often cross-device communication occurs all affect final throughput. If an approach adds extra memory components, their storage footprint, access latency, and maintenance cost should be included in the test rather than comparing only the main model’s parameter count.
The test objects should be determined by the decision goal. If the goal is to study hardware execution efficiency, you can use the same model as a baseline and then observe changes across precision, batching, and parallelism strategies through controlled experiments. If the goal is to select a model for production, you should include candidate approaches that meet delivery requirements—including smaller versions—rather than comparing only the most capable version. In this case each approach may use a configuration suited to itself, but all must obey the same quality floor, latency targets, and resource budget constraints.
For example, in a document field-extraction scenario, what the business actually needs is correct fields, compliant formatting, and data that can be written directly into downstream systems. If a lightweight approach generates quickly but frequently misses required fields, retries and manual data entry can wipe out the savings on the inference side. Conversely, if two approaches both reliably meet field-accuracy and formatting requirements, there is no reason to pay extra compute for capabilities you will never use. When selecting for business needs, what is comparable is not model size but the delivered result.
3. Turn Load Testing into a Near-Production Experiment
3.1 Separate Input Processing from Output Generation
When a request enters the system, the model usually processes the input context first, then generates output step by step. The former stage affects how long a user waits to see the first generated unit; the latter affects the speed of subsequent output and how long the whole request occupies resources. A document-extraction task with long input and short output can hit a different performance bottleneck than a content-generation task with short input and long output—even when the total token counts are close.
A report should record first-token latency, inter-token interval, end-to-end latency, and steady-state throughput separately. Latency should also report high percentiles, such as the 95th and 99th, not just the average. A fast average response in one load test does not mean that requests stuck at the back of the queue during peak hours will finish on time.
3.2 Measure Sustainable Throughput with Representative Traffic
Start by extracting non-sensitive workload characteristics from real business traffic, including the distribution of input/output lengths, the share of long requests, the pattern of concurrent arrivals, and the proportion of shared prefixes. Then test separately at low load, normal daily load, traffic spikes, and sustained high load. If the business has clear day/night swings, you should also simulate the trough rather than calculating unit cost only at full load.
During testing, gradually increase the request arrival rate while watching timeout rate, failure rate, queue growth, and latency percentiles. The key to identifying the capacity limit is not finding the highest throughput the hardware ever reached, but finding the throughput the system can sustain over the long term while the queue does not keep growing and latency and error rates still meet targets. That number is what belongs in production capacity planning.
3.3 Fix Control Conditions and Validate Optimization Gains Individually
If the model, hardware, and runtime parameters all change at once, even a cost reduction leaves you unsure of the cause. It is better to first establish a baseline configuration, then adjust one factor at a time—compute precision, batching strategy, parallelism, and caching. Each time, record GPU memory usage, device utilization, cross-device communication, request execution results, and task quality. After lowering precision, check both memory and throughput changes and re-test the business metrics; after enabling prefix caching, estimate the gain by the actual hit rate, and avoid overly optimistic conclusions drawn from a flood of repeated requests.
If you need to cross-validate cost-effectiveness across different inference environments, you should also unify the task set, quality threshold, and latency target, and note whether resources are dedicated or shared and whether the statistics reflect cold start or steady state. Public throughput data across environments that lack these conditions is, at most, suitable for initial screening and should not be used directly for procurement or capacity commitments.
4. Use Effective Task Cost to Determine Profit and Loss
Cost per million tokens is fine for describing resource consumption, but it does not necessarily explain the business bill. A model may generate longer answers for the same task, or trigger retries because it fails format or accuracy requirements. In that case, the approach with the lower per-token price can actually cost more per delivery. A more direct measure is:
Cost per effective task = Total service cost over the period ÷ Number of effective tasks in the same period
Here, “effective task” should be defined in advance—for example, returned within the required time, passed quality checks, and accepted by the downstream system. Compute consumed by failed requests counts toward cost but not toward the number of effective tasks; if a task is delivered successfully after multiple retries, it still counts as only one effective delivery. Total service cost should cover what the evaluated approach actually bears: resource rental fees in a rented setup, depreciation and energy in a self-hosted setup, and, as needed, network, storage, monitoring, operations, and disaster-recovery redundancy. Items already included in the rental fee should not be double-counted. If an approach requires more manual review, that should also be listed separately in the total business delivery cost.
For the same approach, you should also estimate cost under typical, low, and peak traffic. At low traffic, always-on instances and redundancy capacity can push up unit cost; at peak, adding replicas, handling timeouts, and triggering retries erode returns. So the configuration with the lowest cost at full load does not necessarily have the lowest actual monthly cost.
For services that earn revenue per effective task, you can estimate the break-even point within an approximately linear cost range:
Break-even task volume = Fixed cost per period ÷ (Net revenue per effective task − Variable cost per effective task)
This requires net revenue per task to exceed variable cost per task; otherwise, increasing volume will never cover fixed costs. If scaling requires adding devices in whole groups, cost rises in steps, so you should calculate by capacity range separately. For an internal enterprise service, it is better not to treat call volume directly as revenue; instead, compare manual processing time, rework cost, and delivery efficiency before and after deployment, and validate through a small pilot whether the savings actually materialize.
5. From Test Report to Deployment Decision
Selection can be done in three steps: first, use real task samples to rule out approaches that cannot meet the quality floor; then, use representative load to measure each approach’s capacity while meeting service targets; finally, put capacity, traffic forecasts, and cost into the same calculation sheet. The report should retain enough configuration and measurement basis to reproduce the results, and note which data comes from real measurement and which from cost assumptions. Before going live, select a small portion of real traffic to validate quality, latency, and failure rate, focusing on the two figures most easily overestimated in load testing: cache gains and sustained resource utilization.
If the business has both a large number of simple requests and a small number of complex ones, you can also let a low-cost model handle suitable tasks first, then route tasks that fail validation or genuinely need stronger capability to a higher-cost model. However, routing is only worthwhile when the judgment is reliable enough: misclassification, repeated processing, and added latency from serial calls all need to be included in the effective task cost. If routing does not save enough resources, a single stable, easy-to-maintain deployment is the better choice.
In the end, a single benchmark run is good for spotting performance differences, but it cannot alone decide procurement or deployment. What should be prioritized is: the approach that reliably meets quality and latency requirements under real traffic and completes more effective tasks at an acceptable cost.
