# LLM Inference Cost and Model Selection: How to Read Growing Free Token Credits
Free credit grants on model platforms keep getting larger, but credit size is a marketing decision rather than a capacity decision. This article shows how to convert a credit grant into a real unit cost by combining token price, output volume, first-pass success rate, and retry rounds; explains where steep per-token discounts actually come from on the inference engine side (continuous batching, prefix caching, quantization, speculative decoding, tiered serving); breaks end-to-end latency into TTFT and TPOT with a provider-agnostic measurement script; and turns model selection into a scored, multi-tier routing problem with health checks and fallback. A practical checklist and a simple decision rule close the article.
1. First, Drop One Intuition: Credits Are a Marketing Variable, Not a Capacity Variable
A rising credit allowance is usually an operational rhythm, not a technical one. To read a “doubled allowance” correctly, you need to separate three independent quantities:
- Billed credit: the platform’s accounting unit. It tells you how much ledger-level resource you can spend.
- Unit price: the input and output price per million tokens. It decides how many real tokens a given credit balance buys.
- Effective output: the token volume implied by the credit, multiplied by the task’s first-pass success rate and divided by the average number of retry rounds.
The relationship reduces to a very plain formula:
available_calls = credits / (input_price × avg_input_tokens + output_price × avg_output_tokens)effective_output = available_calls × first_pass_success_rate / avg_retry_roundseffective_cost = credits / effective_output
This explains a common misconception. When an allowance goes from 100 to 500 and unit prices stay in the same order of magnitude, the ledger balance is five times larger but the number of calls you can buy remains in the same order of magnitude. And if that tier happens to cost an order of magnitude more than the platform’s lightweight tier, its “effective water level” can actually end up below the cheaper one. “They handed out 500 and nobody used it” is not a contradiction in the math — people were never comparing allowances. They were comparing effective_cost.
2. Where the Discounts Come From: Cost Engineering on the Inference Engine Side
A model tier priced at 50% or even 10% of list rate usually gets there by one of two paths: the engine side genuinely lowers the compute cost of a single request, or the operator subsidizes the difference out of budget. The second is not sustainable. The first is the part worth studying. In practice, the optimization work falls into five categories.
1. Batching and scheduling
Static batching forces short requests to wait alongside long ones. Continuous batching evicts completed sequences as soon as each decode step ends and immediately packs requests from the waiting queue into the freed slots; chunked prefill keeps a long prompt from monopolizing an entire scheduling cycle. Together they typically produce a substantial gain in GPU utilization. The scheduler also needs priorities and preemption: interactive requests ahead of offline batch jobs, with long-running tasks allowed to be preempted briefly.
2. KV cache memory efficiency and prefix reuse
GPU memory is the hard constraint in inference serving. Paged KV management reduces fragmentation, while prefix caching reuses the KV of repeated prompt prefixes (system prompts, repository context, tool definitions) and skips the redundant prefill. The payoff is especially large in coding workloads: the same repository context gets carried by thousands upon thousands of requests, and once the hit rate climbs, input-side cost and time to first token drop together.
3. Quantization and sparsification
Weight quantization (low-precision formats such as INT8 or FP8), activation quantization, and structured sparsity all trade precision for throughput. One boundary worth calling out: quantization is noticeably more sensitive on long-chain reasoning and strict-syntax output than on casual chat. Code generation has to close brackets correctly and respect the type system, so precision loss surfaces as syntax errors concentrated in one place. A quantized tier must be validated against your own regression set, not judged on general benchmark scores alone.
4. Speculative decoding and parallelism strategy
Speculative decoding has a small draft model propose several tokens in a row, then has the large model verify them in a single pass, which raises per-user output rate during decoding. The parallelism choice shapes the latency profile directly: raising tensor parallelism usually lowers time to first token but reduces total throughput per card, while expert parallelism suits very large sparse models. Before calling something “fast” or “slow”, you have to say whether you mean time to first token or decode rate.
5. Tiered serving
Route requests by value: intent detection, file summarization, and query rewriting go to the lightweight tier; only architecture design and hard defect localization get the deep-reasoning tier. This is the only mechanism that lowers average price without sacrificing quality on the tasks that matter.
A pragmatic judgment: without a self-hosted loop where you control batching and prefix caching, it is very hard to produce a genuine discount of 5x to 10x. So when you see an extreme discount, “do they control their own inference stack?” is a more useful question than “are they burning money?”
3. Splitting Apart “Fast” and “Slow”: Latency Differences Between Forwarding and Self-Hosting
“ Tier A is slower than tier B” is too coarse a conclusion. End-to-end latency for one request decomposes as:
e2e_latency ≈ TTFT + (output_tokens - 1) × TPOTTTFT ≈ queue_wait + prefill_compute + network_round_trips + first_chunk_assemblyoutput_rate ≈ 1000 / TPOT(ms)
- Forwarded through a public API: an extra gateway hop and network round trips add tens to hundreds of milliseconds of fixed overhead to TTFT, and queueing behavior is out of your control.
- Self-hosted or private line: a shorter path and a queue you manage yourself, but operations cost, elastic scaling difficulty, and failure domains all move onto your side.
The right way to observe these differences is to collect percentiles with instrumentation rather than rely on gut feeling. Below is a vendor-neutral minimal skeleton; provider is supplied by whatever adapters you write, as long as the interface stays the same:
import time, statisticsfrom dataclasses import dataclass, field@dataclassclass Sample:ttft_ms: floattpot_ms: floatout_tokens: intok: bool = Truedef probe(provider, prompt: str, max_tokens: int = 256) -> Sample:t0 = time.perf_counter()ttft = Noneprev = t0gaps = []out = 0try:for chunk in provider.stream(prompt, max_tokens=max_tokens):now = time.perf_counter()if ttft is None:ttft = (now - t0) * 1000else:gaps.append((now - prev) * 1000)prev = nowout += 1return Sample(ttft or 0, statistics.median(gaps) if gaps else 0, out)except Exception:return Sample(ttft or 0, 0, out, ok=False)def report(samples):ok = [s for s in samples if s.ok]ttft = sorted(s.ttft_ms for s in ok)tpot = sorted(s.tpot_ms for s in ok)p = lambda a, q: a[min(len(a) - 1, int(len(a) * q))]return {"success_rate": len(ok) / max(1, len(samples)),"TTFT_P50": round(p(ttft, 0.50), 1),"TTFT_P95": round(p(ttft, 0.95), 1),"TPOT_P50": round(p(tpot, 0.50), 2),"decode_tps": round(1000 / p(tpot, 0.50), 1) if tpot else None,}
When sampling, always fix the prompt length, fix the output length, cover different times of day, and record input and output token counts together with cache hit rates. Otherwise the numbers cannot be compared across tiers.
4. Why the High-Capability Tier Often Loses to the Vendor’s Own Lightweight Tier
This is not an anomaly. It is a predictable structural outcome, for four reasons:
- Traffic concentrates on short requests. Code completion, single-file edits, and test writing make up the overwhelming majority of calls. They demand little reasoning depth and are extremely sensitive to price and latency. Once the default tier is the lightweight one, the deep-reasoning tier structurally only receives leftover traffic.
- The capability gap does not translate into a gap in task success rate. Benchmark differences are often a few points to a dozen points, but on tasks like “change this function” or “add a test case”, the first-pass success rates of the two tiers can be very close. The extra money buys no matching improvement in delivery.
- Multimodal input is a hard gate. Screenshots, design mockups, error dialogs, and log images disqualify any tier without multimodal support from the candidate set, no matter how strong its text capability is.
- Interactive scenarios pay a latency tax. A two- or three-fold difference in time to first token is amplified in token-by-token completion. Users do not run rigorous comparisons; they switch to whichever tier feels more responsive.
Conclusion: the correct position for a high-capability tier is “a dedicated tier for a small number of high-value tasks”, not the default. Make it the default and both your bill and your perceived experience get worse.
5. Turning Selection into a Computable Problem: Multi-Tier Routing with Fallback
Scoring each candidate tier and routing by task type is the lowest-friction engineering approach. The score should include four dimensions at once: quality, latency, throughput, and cost.
# Illustrative: quality scores come from your own regression set,# latency and cost from measurements and billing datadef score(m, weights=(0.45, 0.25, 0.15, 0.15)):wq, wl, wt, wc = weightsreturn (wq * m.quality # 0-1, first-pass success rate on your own task set+ wl * (1.0 / (1.0 + m.ttft_p95 / 1000)) # shorter time to first token is better+ wt * min(1.0, m.decode_tps / 200) # decode rate, normalized- wc * m.unit_cost) # unit cost, normalized and negatedTASK_PROFILE = {"completion": ("light", 0.80), # completion: prefer cheap and low latency"edit": ("standard", 0.85), # single-file edit: balance quality and cost"architect": ("deep", 0.70), # architecture / hard localization: cost and latency allowed"multimodal": ("vision", 0.90), # image input: only the multimodal tier qualifies}def select(task, candidates, min_score=0.0):kind, threshold = TASK_PROFILE[task]pool = [c for c in candidatesif c.health and c.supports(kind) and c.quality >= threshold]pool.sort(key=score, reverse=True)return pool[0] if pool else Noneasync def call_with_fallback(task, payload, candidates):pool = sorted((c for c in candidates if c.supports(task)), key=score, reverse=True)last_err = Nonefor c in pool:try:return await c.invoke(payload, timeout=c.slo_timeout) # trip the breaker and switch on timeoutexcept Exception as e:last_err, c.health = e, Falseawait c.cooldown(30) # brief cooldown so traffic stops hitting the failed noderaise RuntimeError(f"All candidate tiers are unavailable: {last_err}")
Four operational practices matter more than the code itself:
- Health probing and circuit breaking: set thresholds on P95 latency and error rate, cool down automatically once tripped, and ramp back with a small share of traffic first.
- Heterogeneous standbys: backups should come from different engine implementations or different regions, otherwise a single failure domain takes out primary and secondary at once.
- A cost ledger: record input, output, and cache-hit tokens and spend per task dimension, and review
effective_costweekly. - Your own regression set: cover at least code completion, cross-file edits, strict-syntax output, and long-context retrieval. Run it before any quantization change or version upgrade.
6. Implementation Checklist
- Do not make decisions from public leaderboards. Their prompt distribution is not your repository’s distribution. A small-sample regression over 30 to 50 of your own real tasks is the best value for money.
- Keep prompt prefixes fixed. Build the system prompt, tool definitions, and repository summary into a stable prefix to raise the prefix cache hit rate and simultaneously cut input-side cost and time to first token.
- Set SLOs explicitly. Define separate TTFT P95 and timeout thresholds per tier, and route interactive traffic and offline batch jobs through different queues.
- Establish tiered defaults. Lightweight tier for completion and single-file edits, escalate to the deep tier only for architecture and hard localization, and send multimodal input to a dedicated vision tier.
- Treat free credit as a load-testing budget, not a capacity budget. The period when credit is plentiful is exactly the time to collect percentiles, run the regression set, and validate the fallback path. Those assets keep working after the credit shrinks.
- Watch observability, not brand names. Whether you run your own inference cluster or use managed inference services on cloud platforms such as Baidu AI Cloud, the experience comes down to the same set of metrics: success rate, TTFT percentiles, decode rate, output per unit cost, and multimodal coverage.
7. Conclusion
Changes in the credit number are market behavior. They can be read as a signal of how intense competition is, but not as a basis for selection. Translated into engineering language, it comes down to one sentence: look at credits / (unit_price × real_token_consumption) to get the effective unit cost, then see how much advantage is left after multiplying it by the inverse of the first-pass success rate.
Whether a tier can sustain a 50% to 10% price depends on whether it controls its own inference stack — continuous batching, prefix caching, quantization, and speculative decoding only turn into price when you hold them in your own hands. Whether a tier belongs in your default configuration depends on whether it simultaneously satisfies a low enough cost, a short enough TTFT, and the input modalities you actually need. Missing any one of the three, and it belongs in the “high-value task dedicated tier” instead. Work out this arithmetic once and the coming and going of free-credit promotions will no longer move your cost structure.
