Don't Just Look at LLM API Prices: A Cost Calculation Guide and Troubleshooting Checklist

    LLM API unit prices are falling, but the cost of each call depends on model architecture, billing rules, and how the API is used. This guide explains why mixture-of-experts models can lower per-call inference costs, which billing items to compare when choosing a model, how to implement streaming with SSE, and how to troubleshoot rate limits, missing usage data, and truncated responses.

    1. Why Trillion-Parameter Models Can Lower Per-Call Inference Costs

    Parameter count and inference cost are two different metrics. When a dense model generates each token, it uses nearly all of its parameters. A mixture-of-experts (MoE) model contains many expert networks, but each token passes through only a few of them. A model’s total parameter count can therefore be very large, while the number of parameters actually used in a single inference pass (the activated parameters) is much smaller. As a result, the compute per token can approach that of a smaller dense model.

    This conclusion has conditions. If expert load is unbalanced, or if the serving system must keep the weights of all experts loaded at once, memory and bandwidth costs still rise. Fewer activated parameters do not guarantee a cheaper call. The deployment method also matters.

    Serving systems also rely on two common optimizations:

    1. Continuous batching: The inference service dynamically merges requests into the same batch, which reduces GPU idle time.
    2. Memory optimization: Techniques such as PagedAttention manage the KV cache in pages. This reduces memory fragmentation, so a single GPU can hold more concurrent requests.

    Elastic scheduling is a third factor. The provider adds or removes GPU instances based on request volume, scaling down during low-traffic periods and up during peaks. This keeps hardware utilization high, and fixed costs are spread across more calls. However, elastic scheduling has clear limits:

    • Workloads with steady traffic gain little, because there are no low-traffic periods to scale down in.
    • Scaling takes time. Sudden traffic spikes cause queuing, which increases both time to first token and tail latency.
    • Scheduling policies are internal to the provider. Users cannot see them and can only judge them indirectly, through SLA commitments and measured latency.

    The optimizations above happen on the provider side, but users still have some indirect controls. You can merge short requests, place fixed system instructions at the beginning of the prompt so they can hit the cache, set a reasonable max_tokens value, move non-real-time tasks to batch interfaces, and use smaller model tiers for simple tasks.

    Whether prices fall depends on the specific provider, model version, and time. The phrase “trillion parameters” alone does not tell you the price. Check the official price list when choosing a model.

    2. Compare More Than Unit Prices When Choosing a Model

    Many teams compare only the price per million tokens, and their budgets end up off target. At a minimum, compare the following dimensions:

    Dimension Description Why It Matters
    Input token price Billing rate for prompts, system instructions, and retrieved passages Usually the largest cost in long-context scenarios
    Output token price Billing rate for generated content, usually higher than input Has a significant impact when generating long text or code
    Reasoning tokens Some reasoning models generate a thinking process that is not shown to users. These tokens are usually billed as output. Billed output tokens can far exceed the visible text
    Cache read and write prices Repeated prefixes that hit the cache are billed at a lower rate. Cache writes may be billed separately. Fixed system instructions can significantly reduce input costs
    Batch discounts Discounted prices for offline or asynchronous batch jobs, usually with a completion deadline Effective for non-real-time tasks
    Rate limits (RPM/TPM) Maximum requests per minute and maximum tokens per minute Determines whether your service can handle peak traffic
    Context window Maximum number of tokens accepted in a single request Determines whether a long document can be sent in full
    SLA and latency Availability commitments, time to first token, and total response time Directly affects user experience

    Providers define differently which models and calling modes count reasoning content as output. Check the provider’s documentation and billing details. Do not assume that every output token is visible text.

    The basic formula is:

    Cost per call = input tokens / 1,000,000 × input price + output tokens / 1,000,000 × output price

    This formula applies only when there is no caching, no batch discount, and all output tokens are visible text. When caching or reasoning is enabled, calculate each component separately:

    Cost per call = [uncached input tokens × input price + cache-hit tokens × cache read price + cache-write tokens × cache write price + (visible output tokens + reasoning tokens) × output price] / 1,000,000

    All prices in the formula are the provider’s rates per million tokens. When you use a batch interface, multiply the result by the discount factor the provider specifies. Other billable items, such as tool calls, images, and web search, are not included in this formula. Add them separately using the same method.

    Here is a calculation with assumed numbers. Assume the price is CNY 2 per million input tokens, CNY 8 per million output tokens, and CNY 0.2 per million cache-hit tokens. Cache writes are not billed separately. These numbers are for illustration only and do not represent any provider’s pricing.

    In one call, the input is 2,000 tokens: 1,500 hit the cache and 500 do not. The visible output is 500 tokens, and reasoning adds 1,000 tokens.

    • Uncached input cost: 500 / 1,000,000 × 2 = CNY 0.001
    • Cache-hit cost: 1,500 / 1,000,000 × 0.2 = CNY 0.0003
    • Output cost (visible plus reasoning, 1,500 tokens in total): 1,500 / 1,000,000 × 8 = CNY 0.012
    • Total per call: CNY 0.0133

    If you ignore caching and reasoning tokens and price the same call as 2,000 input tokens and 500 output tokens, the cost is CNY 0.008 per call. The two figures differ by about 66%. At 100,000 calls per day, the first scenario costs about CNY 1,330 per day, or about CNY 39,900 over 30 days. The second scenario costs about CNY 800 per day, or about CNY 24,000 per month.

    Cache and reasoning token accounting have a large effect on the budget. Replace the assumed values with the input, output, cache-hit, and reasoning token counts from your real logs before you estimate costs.

    3. Engineering Practices for Integration

    1. Streaming Calls

    A streaming interface returns the model’s output in segments, so users see the first words sooner. Most APIs compatible with the OpenAI format use the SSE (Server-Sent Events) protocol. SSE runs over regular HTTP. The server keeps pushing data, each event starts with “data: “ and events are separated by blank lines, and the stream ends with “data: [DONE]”.

    SSE is one-way push, which suits model output. WebSocket is a two-way persistent connection, which suits interactive systems where the client also needs to send messages in real time. If you only receive model output, SSE is simpler to implement and passes through ordinary HTTP gateways more easily.

    The Python example below depends on the requests library (install with pip install requests). It reads the endpoint and API key from environment variables, so they are not written into the code. Before running it, set them with export LLM_BASE_URL=<provider endpoint> and export LLM_API_KEY=<your API key>.

    1. import json
    2. import os
    3. import time
    4. import requests
    5. BASE_URL = os.environ["LLM_BASE_URL"] # e.g. https://api.example.com/v1
    6. API_KEY = os.environ["LLM_API_KEY"] # Read from environment variable, never hardcode
    7. def stream_chat(prompt: str, model: str = "your-model-name", max_retries: int = 3) -> str:
    8. url = f"{BASE_URL}/chat/completions"
    9. headers = {
    10. "Authorization": f"Bearer {API_KEY}",
    11. "Content-Type": "application/json",
    12. "Accept": "text/event-stream",
    13. }
    14. payload = {
    15. "model": model,
    16. "messages": [{"role": "user", "content": prompt}],
    17. "stream": True,
    18. "stream_options": {"include_usage": True}, # Supported by some providers; usage is returned at the end
    19. "max_tokens": 1024,
    20. }
    21. for attempt in range(1, max_retries + 1):
    22. try:
    23. with requests.post(url, headers=headers, json=payload,
    24. stream=True, timeout=(10, 120)) as resp:
    25. if resp.status_code == 429 or resp.status_code >= 500:
    26. retry_after = resp.headers.get("Retry-After", "")
    27. wait = int(retry_after) if retry_after.isdigit() else 2 ** attempt
    28. if attempt == max_retries:
    29. raise RuntimeError(f"Retries exhausted, last status code {resp.status_code}")
    30. time.sleep(wait)
    31. continue
    32. if resp.status_code != 200:
    33. raise RuntimeError(f"Request failed, HTTP {resp.status_code}: {resp.text[:200]}")
    34. parts, usage, finish_reason, done = [], None, None, False
    35. for raw_line in resp.iter_lines(decode_unicode=True):
    36. if not raw_line or not raw_line.startswith("data:"):
    37. continue
    38. data = raw_line[5:].strip()
    39. if data == "[DONE]":
    40. done = True
    41. break
    42. chunk = json.loads(data)
    43. if chunk.get("usage"):
    44. usage = chunk["usage"]
    45. for choice in chunk.get("choices", []):
    46. text = choice.get("delta", {}).get("content")
    47. if text:
    48. print(text, end="", flush=True)
    49. parts.append(text)
    50. if choice.get("finish_reason"):
    51. finish_reason = choice["finish_reason"]
    52. print()
    53. if not done:
    54. print("Warning: stream ended without [DONE]. The content may be incomplete.")
    55. print(f"finish_reason={finish_reason}, usage={usage}")
    56. return "".join(parts)
    57. except (requests.ConnectionError, requests.Timeout):
    58. if attempt == max_retries:
    59. raise
    60. time.sleep(2 ** attempt)
    61. raise RuntimeError("Retry attempts exhausted")

    Keep these points in mind when you read the code:

    • The code processes only lines that start with “data:” and ignores blank lines and other fields.
    • The stream ends normally only after “[DONE]” is received. If the connection drops early, the code prints a warning that the content may be incomplete.
    • Status codes 429 and 5xx are retried after the Retry-After delay. Other 4xx errors are raised immediately, so authentication failures are not retried pointlessly.
    • Usage is returned only at the end of the stream, and only if the provider supports stream_options. If it is not supported, usage is empty.

    Proxies and gateways can also affect streamed output. Nginx enables response buffering by default. It collects the model’s output into batches before sending it, so users see everything appear at once. Turn off buffering on reverse proxies and increase the read timeout, for example:

    1. location /v1/ {
    2. proxy_pass https://upstream.example.com/v1/;
    3. proxy_buffering off;
    4. proxy_read_timeout 300s;
    5. }

    4. Troubleshooting Common Errors

    1. Rate Limiting (HTTP 429)

    Symptom: The request returns 429 and reports that the rate limit has been exceeded.

    Diagnosis: First determine whether you hit the RPM (requests per minute) limit or the TPM (tokens per minute) limit. Count the requests and total tokens for each minute. A single long-context request can use up the TPM quota even when the request count is low. Some providers return remaining quota in response headers. Check the relevant API documentation for the exact field names.

    Fixes:

    • Wait for the duration given in Retry-After, and use exponential backoff for retries, as shown in the example above.
    • Apply token bucket rate limiting on the client to smooth peaks within your quota.
    • Reduce unnecessary long context, and use cached prefixes to lower repeated input costs.
    • If your business has genuine peak demand, request a higher quota from the provider.

    2. Token Usage Shows Zero

    Symptom: The usage field in your logs is 0 or empty, but the bill still shows charges.

    Common causes:

    • In a non-streaming call, the code reads choices but does not read the usage field in the response body, or the field name is misspelled.
    • In a streaming call, usage appears only in the last data chunk. If the code exits early or reads only the first chunk, it never gets the usage data.
    • A proxy or gateway drops the final data chunk, or the connection times out.
    • The provider or compatible implementation does not return usage.

    Diagnosis: Send a request directly to the provider’s endpoint with curl, then send the same request through your proxy, and compare the final data lines of both responses. Save the last few raw data lines and confirm whether usage is present.

    Fixes:

    • In the streaming loop, keep the last data chunk that contains usage, as shown in the example code.
    • Record the request ID for each call, and reconcile it against the bill entry by entry.
    • If usage truly cannot be obtained, estimate tokens with the tokenizer the provider supplies, and label the values as estimates in your logs.

    3. Truncated Responses

    Symptom: The response stops mid-sentence, or the visible content is empty.

    Common causes:

    • finish_reason is “length”, meaning the output reached the max_tokens limit. For reasoning models, reasoning tokens also count against this limit, so the visible content may be very short or empty.
    • The input plus the expected output exceeds the context window. Some providers return an error in this case. Others truncate the input. Check the documentation and test to confirm the behavior.
    • The streaming connection drops partway through, with no [DONE] and no finish_reason. This is usually caused by a proxy timeout or buffering configuration.
    • finish_reason is “content_filter”, meaning the content was blocked by a safety policy.

    Diagnosis: Log the finish_reason, input token count, output token count, and reasoning token count for every call. When a response is truncated, check finish_reason first, then check whether the output token count is close to max_tokens.

    Fixes:

    • If finish_reason is “length”, increase max_tokens, then recalculate the cost per call.
    • For reasoning models, reserve enough output budget so that the reasoning process does not use up the entire limit.
    • If the input is too long, compress it, or split it into segments and merge the results afterward.
    • If the connection is interrupted, increase the proxy read timeout and check the buffering settings.

    5. Conclusion and Checklist

    Before you choose a model or set a budget, confirm the following:

    • You have obtained the input, output, cache read, cache write, and batch discount prices from the official price list, and recorded the version and date.
    • You have confirmed how reasoning tokens are billed, and measured their share of total tokens using real logs.
    • You have replaced the assumed values with real input, output, and cache-hit data, and calculated daily and monthly costs using the full formula.
    • You have checked the RPM and TPM limits and confirmed they can cover peak demand.
    • You have confirmed that the context window can hold the longest input plus the expected output.
    • You have confirmed that the SLA, time to first token, and total response time meet your user experience requirements.