Domestic AI Chips for LLM Inference: How to Evaluate Performance and Economics Scientifically

    A telecom operator recently benchmarked two LLM inference services on domestic AI accelerators and concluded that one model was profitable while the other lost money. That kind of result spreads fast in engineering circles but is easy to misread, because profit is never a property of a chip alone — it emerges from model scale, inference architecture, service load, and hardware utilization together. This article breaks down why such tests mislead, how to make comparisons fair by matching model configurations, how to read unit economics per million tokens, which metrics a rigorous inference benchmark should cover, and how to move from single-card performance to total cost of ownership — plus practical advice for technical teams building their own reproducible baselines.

    1. Where the Problem Comes From: Why Tests Produce “One Wins, One Loses” Conclusions

    A telecom operator recently ran a round of performance and cost testing on two large-model inference services using domestic AI accelerator cards. The conclusion was that one model turned a profit and the other was operating at a loss. Results like these spread quickly in engineering circles, yet they are very easy to misread. Profit or loss is never a measure of a chip’s capability in isolation; it is the combined outcome of model scale, inference architecture, service workload, and hardware utilization. To interpret such tests correctly, you first need to separate three questions that are usually tangled together: how fast the hardware actually runs, whether the model configurations being compared are equivalent, and whether the cost per unit of inference covers the investment.

    2. The First Requirement for a Fair Comparison: Model Configurations Must Be Equivalent

    Many comparisons turn out to be nearly meaningless because the two models under test are not in the same weight class at all. Scale differences between large models can reach several times or even an order of magnitude. Change any one of total parameter count, effective activated parameters, context window, or quantization precision, and the compute and memory cost of inference shifts significantly. Putting a larger model head-to-head with a smaller one makes a difference in speed almost inevitable — and that difference says nothing about the real capability of the chip.

    A meaningful comparison therefore starts by making the model configurations equivalent:

    • Match scale: keep parameter count, effective activated parameters (for sparse architectures), layer count, and hidden dimension as close as possible;
    • Match precision: use the same quantization scheme (both FP16, both FP8, or both INT8), otherwise precision differences alone will change throughput;
    • Match context: use identical input and output lengths so that prefill overhead does not skew the results;
    • Match serving shape: same batching strategy, concurrency level, and SLA target.

    Only when “model A” and “model B” sit in the same scale bracket is a performance gap between chips actually comparable. For a real deployment, you should choose a model configuration whose scale matches the reference model (including all of its structural components) before comparing — otherwise you end up comparing apples to oranges.

    3. What “Profitable or Not” Actually Means: Breaking Down Inference Economics

    “Profitable” or “loss-making” is fundamentally a unit-economics question: is the cost of serving one million tokens lower than the price the business is willing to pay? Inference cost has three main components:

    • Hardware amortization: the fixed hourly cost of accelerator cards and servers spread over their depreciation period;
    • Power and datacenter: electricity, cooling, rack space, and other operating costs;
    • Effective throughput loss: hardware is not running at full load all the time — batching gaps, memory bandwidth bottlenecks, and KV Cache pressure all discount the nominal compute.

    A minimal estimation formula makes the relationship clear:

    1. # Single-node inference economics estimate (illustrative, not measured)
    2. tokens_per_second = throughput(concurrency, batch_size) # measured effective throughput
    3. node_hourly_cost = hardware_amortization + power + ops # total hourly cost
    4. cost_per_million_tokens = node_hourly_cost / (tokens_per_second * 3600) * 1_000_000

    The formula shows the mechanism directly: the larger the model, the less efficient the quantization, and the worse the concurrency utilization, the lower the effective throughput and the higher the cost per token. A larger model that “loses money” usually does so not because the chip is weak, but because it inherently consumes more compute and memory. A smaller model that “makes money” may simply be producing more output per unit of compute. If you do not isolate the scale factor and instead attribute profit and loss to the chip, the conclusion is guaranteed to be distorted. Sparse activation architectures (the MoE family) are another reason costs differ so widely between architectures: they activate only a subset of experts, so actual compute consumption does not scale with total parameter count.

    4. Which Metrics Matter: Don’t Fixate on a Single Number

    Looking only at “peak compute per card” or a single throughput figure is the fastest route to a one-sided conclusion. A rigorous inference benchmark should cover at least:

    • Throughput: tokens processed per unit of time, reflecting overall serving capacity;
    • Time to first token (TTFT): time from request submission to the first token emitted, which determines how responsive interactions feel;
    • Time per output token (TPOT/ITL): the interval between adjacent tokens during decoding, which affects the experience of generating long text;
    • Concurrency and stability: how latency and throughput curves behave as concurrency rises, and whether noticeable jitter appears;
    • Resource utilization: compute utilization, memory bandwidth consumption, and memory capacity headroom, used to locate bottlenecks.

    It is essential to distinguish the two phases: prefill is compute-intensive and determines TTFT; decode is memory-bandwidth-intensive and determines TPOT and throughput. Many arguments about whether “the chip is fast” actually conflate these two phases — the same hardware may perform well on prefill while being constrained by memory bandwidth during decode. A test must fix and disclose the workload, sequence length, and concurrency level, or the results cannot be reproduced.

    5. Doing Cost-Performance Right: From Single-Card Performance to Total Cost of Ownership

    Chip selection ultimately comes down to cost-performance rather than peak performance in isolation. A three-step approach works well:

    1. Define a realistic workload profile: measure the actual distribution of input and output lengths, peak concurrency, and SLA requirements on the business side instead of assuming an idealized fixed load;
    2. Compute end-to-end unit cost: use the formula above to derive cost per million tokens, broken down by service quality tier;
    3. Factor in total cost of ownership (TCO): include procurement, deployment, operations, power, and software stack maturity — not just hardware unit price.

    This also explains why cost-performance claims are more credible from independent, standardized evaluations. Vendor-run tests tend to pick the models and workloads that flatter the hardware, whereas a neutral third party using a uniform benchmark and configuration produces results you can trust more. The ideal practice is to build a reproducible benchmark pipeline that fixes the model configuration, quantization, concurrency, and sequence length, establishing a reference baseline.

    6. Practical Recommendations for Technical Teams

    • Align first, compare second: for any chip or model comparison, confirm that the subjects are equivalent in scale, precision, and context; otherwise leave them out of the conclusions;
    • Report the full metric set: give throughput, TTFT, TPOT, and utilization together, rather than a single benchmark score;
    • Let unit cost do the talking: convert performance into cost per million tokens and tie it to a specific SLA, so it can be connected to business pricing;
    • Build your own baseline: maintain an internal standardized benchmark and dataset so every new piece of hardware or new model can be re-tested quickly and fairly;
    • Watch the software stack: operator and framework adaptation maturity, quantization support, and batching/scheduling behavior often affect real throughput more than the bare hardware does.

    As domestic accelerator cards iterate quickly, platforms such as Baidu AI Cloud — with containerized deployment, monitoring and alerting, and log analysis capabilities — can help teams make benchmark testing more standardized and visible. But whatever hardware and software combination you use, sticking to the principles of “equivalent configuration + full metric set + unit cost” will keep you away from surface-level conclusions like “one model profits, one loses,” and let you see what domestic chips actually deliver in inference performance and economics under real business workloads.