How to Control LLM API Costs: Model Selection and Billing Pitfalls to Avoid

3 hr ago0 views

This article explains how to manage large language model API expenses by understanding token-based billing, selecting models by task complexity, estimating costs reliably, and avoiding common billing traps such as unbounded conversation history, oversized contexts, unnecessary retries, and runaway output lengths. It provides practical formulas, configuration examples, and retry logic to help teams optimize spending without sacrificing quality.

1. First, Understand Where Your Money Goes

Large model API costs are typically metered by tokens. A token is the basic unit the model uses to process text. Input content (prompt, context, attached text) and output content (the model’s generated answer) are billed separately, and output unit prices are generally higher than input prices. The cost of a single request is determined by three variables: input length, output length, and the respective unit prices.

Most teams overspend not because a single call is particularly expensive, but because of a combination of these three patterns:

All tasks default to the strongest model.
Multi-turn conversations resend the entire history on every turn.
Failed retries and uncapped long outputs quietly inflate usage.

Below we discuss model selection, understanding billing, common pitfalls, and usage monitoring.

2. Model Selection: Tier by Task Complexity

Different model tiers differ significantly in capability and price. Vendors use different names for their tiers, but most can be grouped into three categories:

Light tier: Fast response, low unit price. Suitable for tasks with clear structure and well-defined rules, such as completing comments, format conversion, simple Q&A, and drafting initial test cases.
Balanced tier: A compromise between capability and cost. This is the workhorse for daily development, suitable for feature implementation, bug localization, code reading, and moderately complex refactoring.
Premium tier: Strongest reasoning ability, highest unit price. Use only for tasks that truly require deep reasoning, such as multi-file coordinated changes, hard-to-reproduce issues, and system architecture design.

A common waste is sending all requests to the premium tier. In practice, most routine tasks achieve the same result quality on the balanced tier at a fraction of the premium price.

When choosing a model, follow this decision order:

Can the task be handled by rules or scripts? If yes, do not call a model.
Is the task structure clear and the output format fixed? If yes, prefer the light tier.
Does the task involve multi-step reasoning or cross-context dependencies? If yes, consider the balanced tier.
Only when results from the previous tiers fail to meet quality requirements should you upgrade to the premium tier.

Don’t judge whether a model is “good enough” by intuition. Sample 20–50 real task cases, run them through different tiers, and compare quality via human review or automated scoring. Then decide the default tier for each task type. This gives conclusions based on your own data.

Example routing configuration:

TASK_TIER = {
    "comment_gen": "light",
    "format_convert": "light",
    "unit_test_draft": "light",
    "feature_impl": "balanced",
    "bug_fix": "balanced",
    "code_review": "balanced",
    "multi_file_refactor": "premium",
    "arch_design": "premium",
}
MODEL_BY_TIER = {
    "light": "<light-tier model ID>",
    "balanced": "<balanced-tier model ID>",
    "premium": "<premium-tier model ID>",
}
def pick_model(task_type: str) -> str:
    # Unknown tasks default to balanced tier to avoid blindly using high-cost models
    tier = TASK_TIER.get(task_type, "balanced")
    return MODEL_BY_TIER[tier]

Keep the tier mapping centralized. When models or prices change, you only update the configuration, not every business call site.

3. Understand Billing: Estimation Beats Guesswork

When understanding billing rules, keep three key points in mind:

The billing unit is tokens, not characters or words. The same text in Chinese and English often has different token counts. Always rely on the provider’s counting method or the usage fields returned in the response.
Input and output are priced separately, and output is usually more expensive. Therefore, limiting output length often reduces cost faster than compressing input.
Some platforms offer discounts for cached prefix content or batch calls. Refer to the official documentation for exact rules and prices; don’t rely on memory when designing a cost model.

The cost of a single request can be estimated with this formula:

Single request cost = input tokens × input price + output tokens × output price

(Prices are per million tokens)

def estimate_cost(input_tokens, output_tokens, in_price_per_m, out_price_per_m):
    """Prices are per million tokens (in your currency)."""
    return (input_tokens * in_price_per_m + output_tokens * out_price_per_m) / 1_000_000

Example calculation. Assume input price is 10 currency units per million tokens and output price is 40 currency units per million tokens (for demonstration only, not actual prices). For a request with 6,000 input tokens and 1,500 output tokens, the cost is:

6000 × 10 / 1,000,000 + 1500 × 40 / 1,000,000 = 0.06 + 0.06 = 0.12

If this feature is called 2,000 times per day, the daily cost is about 240 currency units. Estimating single-request cost and daily call volume first helps you decide which factor to optimize, which is more effective than waiting for the bill.

4. Avoid Common Billing Pitfalls

4.1 Unbounded Conversation History

Each round resends the entire history, so input tokens accumulate as turns increase. Mitigations include:

Set a history window, keeping only the most recent turns.
Compress older content into summaries instead of full text.
Store confirmed conclusions in a structured format and reference only the results in later turns, avoiding the full derivation.

4.2 Stuffing the Entire Code Repository into Context

Most requests only need a specific function, a few dozen lines of configuration, or a single error message. First locate the relevant file or search for the scope, then include only the relevant snippet. This significantly reduces input size and often improves the relevance of responses.

4.3 Retries Amplifying Cost

Network timeouts, rate limiting, and temporary server errors are appropriate for retries. Parameter errors, authentication failures, and similar issues are not retry-worthy—they only incur repeated charges. A sound retry policy should:

Only retry retryable errors.
Use exponential backoff with random jitter to avoid triggering rate limits during retries.
Set a maximum retry count and log usage for each retry.

import random, time
RETRYABLE = {"timeout", "rate_limited", "server_error"}
def call_with_retry(send, max_retries=3):
    for attempt in range(max_retries + 1):
        ok, result, err_kind = send()
        if ok:
            return result
        if err_kind not in RETRYABLE or attempt == max_retries:
            raise RuntimeError(f"Request failed: {err_kind}")
        time.sleep(min(2 ** attempt, 16) + random.uniform(0, 1))

4.4 Uncontrolled Output Length

Without a set limit, the model may output long explanatory text. Set a maximum output length and include explicit instructions like “return only JSON” or “provide only the modified code” to restrict output to what is truly needed. Note that truncated output may be incomplete, so combine with format validation and fallback logic.

4.5 Verbose System Prompts Repeated Every Time

Fixed role definitions and