Model Selection and Migration After API Pricing Model Changes

1. Background: A Higher List Price Does Not Mean Higher Costs

The pricing structure of large language model APIs is becoming more complex. As is common in the industry, some platforms have begun differentiating peak and off-peak prices for the same model, and pricing cache hits and different token types (such as standard input, cached input, and output) separately. This means two things:

First, there is no linear relationship between a nominal price increase and an actual bill increase. A task dominated by short input and long output, and a task dominated by long document input and short answers, have completely different cost structures under the same price sheet.

Second, the definition of a “drop-in replacement” must return to per-task cost. A drop-in replacement is not about finding a model with a lower list price; it is about replacing full-volume calls with a lower cost “per successfully completed task” while meeting the quality bar on real tasks. A low-priced call that fails the quality bar is a negative cost—it brings rework, retries, and user churn.

Therefore, the correct starting point for model selection and migration is not the price sheet, but the combination of task profile + quality baseline + cost accounting model.

2. Cost Accounting: Converting Unit Prices into Per-Task Cost

2.1 Three Variables That Must Be Included

For a given call, let the number of input tokens be $I$ (of which the cache-hit portion is $Ic$), and the number of output tokens be $O$. If the input unit price is $p_i$, the cached input unit price is $p{ic}$, and the output unit price is $p_o$, then the cost of a single call is:

cost = (I - I_c) * p_i + I_c * p_ic + O * p_o

This formula reveals three facts that are easily overlooked:

• Cache hit rate directly determines input-side cost. Under some platforms’ pricing systems, the price gap between a cache miss and a cache hit can be very large; raising the hit rate from 0 to 60% can cut input-side cost by more than half. Whether system prompts, fixed knowledge snippets, and few-shot examples are placed in a cacheable prefix position affects the bill more than which model vendor you choose.
• Output tokens usually carry far more weight than input. Output-heavy tasks such as summarization, rewriting, and code generation are far more sensitive to the output unit price than to the input unit price.
• The peak/off-peak price gap is a scheduling problem, not a selection problem. If the business allows batch tasks to be moved to off-peak hours, the effective unit price can be directly halved—this often yields a larger benefit than switching models, with no quality risk.

2.2 Using “Cost per Thousand Successful Tasks” for Cross-Comparison

It is advisable to establish a unified comparison standard:

unit cost = (average cost per call × average number of calls) / task pass rate

Here, “average number of calls” includes retries, multi-turn supplementation, and second requests after validation failures; “task pass rate” is given by the quality baseline evaluation. This standard puts “cheap but requires three retries” and “expensive but passes on the first try” on the same ruler. Many options that appear to differ by several times in price will see the gap narrow significantly, or even reverse, after conversion.

2.3 An Executable Accounting Script Skeleton

def unit_cost(avg_input, cache_hit_ratio, avg_output,
              p_in, p_cache, p_out,
              retries=1.0, pass_rate=1.0):
    miss = avg_input * (1 - cache_hit_ratio)
    hit  = avg_input * cache_hit_ratio
    per_call = miss * p_in + hit * p_cache + avg_output * p_out
    return (per_call * retries) / max(pass_rate, 1e-6)
# When comparing two candidate options, always use the same set of avg_input/avg_output/retries/pass_rate

The script itself is not important; what matters is that the parameters must come from real traffic sampling, not from rough estimates. It is recommended to compute, from logs and by task type, the input/output length distribution (P50/P90), cache hit rate, retry count, and human intervention rate.

3. Quality Baseline: A Prerequisite for Drop-In Replacement

A cost comparison without a quality baseline is meaningless. It is advisable to build a small and stable evaluation set for each task type:

• Size: 50–200 items per task type, covering typical inputs, boundary inputs, and known error-prone inputs.
• Metrics: accuracy or field-level F1 for structured tasks; human ratings or similarity to reference outputs for generative tasks; executable pass rate for code tasks.
• Acceptance threshold: pre-set an “acceptable lower bound,” for example, accuracy no lower than 98% of the baseline, or a human rating no more than 0.3 points lower. Candidate models below the threshold are eliminated directly and do not enter the cost comparison stage.

The value of this step is turning “feels about the same” into an auditable judgment. When migrating to a new pricing model, the evaluation set also serves as a regression test, preventing low-priced options from systematically degrading on long-tail inputs.

4. Tiered Routing: Not Every Request Needs the Same Model

Once the quality baseline is clear, the most effective cost-reduction lever is usually not a global replacement, but tiering by request difficulty.

4.1 Tiering Strategy

Tier	Request Characteristics	Handling
L0	Repeated questions that hit the cache	Return the cached result directly, zero model calls
L1	Short input, templated, low risk	Lightweight model
L2	Moderate complexity, requires some reasoning	Primary model
L3	Long context, high value, low error tolerance	High-capability model, higher unit price allowed

The basis for tiering can be rules (input length, presence of structured fields, business type) or a lightweight classifier. The key constraint is: the tiering logic must be observable—the call volume, pass rate, and unit cost of each tier must be separately measurable; otherwise, cost anomalies cannot be located.

4.2 Semantic Caching and Prompt Structure Optimization

Caching is not only a platform-side capability; the application side can also do it:

• Exact caching: hash the normalized input, and return directly on a hit. Suitable for FAQ and configuration query scenarios.
• Semantic caching: vectorize the input, reuse the answer when similarity exceeds a threshold, and attach a confidence judgment. Note that for time-sensitive tasks (such as price, inventory, and status queries), semantic caching should be disabled or its TTL shortened.
• Prompt prefix stabilization: placing system instructions, fixed examples, and knowledge snippets at the front of the request and keeping them byte-stable can significantly improve the platform-side cache hit rate. Any practice of inserting timestamps or random IDs into the prefix will break the cache.

5. Migration Plan: A Rollback-Safe Canary Path

What matters most during a pricing adjustment window is “controllability.” It is advisable to proceed in the following order:

Step 1: Shadow traffic comparison. Copy production traffic and send it to the candidate option, without returning results externally, recording only quality metrics and cost. Run for one to two weeks, covering a complete business cycle (including peak and off-peak periods).

Step 2: Canary rollout by task type. Start switching traffic from a single task type with the lowest risk and the most mature evaluation set, starting at 5% and increasing in steps of 1%–5%. At each stage, observe three metrics: task pass rate, P90 latency, and unit cost.

Step 3: Establish automatic rollback conditions. Clearly define the thresholds that trigger a rollback, for example, “pass rate below baseline by 2 percentage points for 30 consecutive minutes” or “unit cost 20% above expectation.” A rollback should be a configuration change, not a re-release.

Step 4: Integrate peak/off-peak scheduling. After the migration stabilizes, schedule non-real-time tasks (offline summarization, batch labeling, report generation) to off-peak hours. This step does not change model selection, only execution time—lowest risk and certain benefit.

# Routing configuration example: declare candidates and rollback thresholds by task type
routes:
  doc_summary:
    primary: model_a
    fallback: model_b
    batch_window: off_peak        # Allowed to defer to off-peak execution
    rollback:
      pass_rate_drop: 0.02
      unit_cost_increase: 0.20
  code_assist:
    primary: model_b
    fallback: model_a
    batch_window: none            # Real-time task, no deferral

Step 5: Cost attribution dashboard. After the migration is complete, break down costs along four dimensions: “task type × model × time period × cache hit.” Most teams will find at this step that the real cost concentration is often in a few task types—and these types are not necessarily the ones with the highest call volume.

6. Common Pitfalls

• Looking only at unit price and not at token structure: output-heavy and input-heavy tasks differ enormously in sensitivity to the same price sheet; cross-comparison must be weighted by real distributions.
• Ignoring retry costs: a higher retry rate for a low-priced model eats up the entire price gap, or even makes it more expensive.
• One-size-fits-all replacement: a full switch makes quality degradation on long-tail inputs hard to locate; canary rollout and tiering are necessary costs.
• Treating caching as free: semantic caching carries a risk of false hits and requires confidence thresholds and manual spot-check mechanisms, especially in scenarios involving factual content.
• Overlooking latency as a hidden cost: a rise in P90 latency changes user behavior (retries, abandonment) and indirectly pushes up per-task cost.

7. Summary

The more complex the pricing model, the more model selection should return to engineering methods: use real traffic sampling to build a cost model, use evaluation sets to protect the quality baseline, use tiered routing and caching to reduce ineffective calls, use canary rollout and automatic rollback to control migration risk, and use peak/off-peak scheduling to obtain certain benefits. This approach does not depend on any specific platform’s price sheet, and it applies equally to the next round of pricing adjustments.