Why Are AI Model Costs Falling So Fast? Inference Cost Trends and Model Selection Strategies
AI costs are not falling uniformly, but splitting in two directions. Legacy models have dropped by orders of magnitude and can cost a fraction of a cent per million tokens. New flagship models decline slowly because demand for their capabilities stays strong. This article explains what drives both trends and shows how to keep spending low with task tiering: cheap or cached responses for high-volume simple requests, and flagship models only where they matter. It also covers routing strategies and practical ways to cut inference costs further.
1. Has AI Gotten Cheaper? Two Kinds of “Getting Cheaper”
Claims about AI costs point in opposite directions. Some say costs have “plunged hundreds of times,” while others say they’re “still sky-high.” Neither is wrong: the two statements describe two different generations of models.
The per-unit cost of AI in general, and inference costs in particular, has fallen steadily over the long term. The drop has not been a slow, linear decline but a shift of nearly an order of magnitude. A widely observed rule of thumb is that the inference cost of a model with comparable capabilities falls by almost 10x each year. This trend will most likely continue, with room for inference costs to drop by another order of magnitude.
The key to understanding this is to separate the cost of older models from the cost of newer ones.
2. Legacy Models: Where the Dirt-Cheap Prices Come From
The previous generation of general-purpose models has seen the steepest cost declines. One common example in the industry is an early general-purpose model whose per-unit API cost fell from roughly $20 per million tokens shortly after launch to the order of a fraction of a cent per million tokens. That amounts to a drop of several hundred times. These models are now extremely cheap, so cheap that calling them a “bargain” is hardly an exaggeration.
Several factors drive this dramatic decline:
- Mature inference techniques. Engineering methods such as quantization, distillation, sparsification, batched scheduling, and operator-level optimization let models of the same accuracy run on less compute.
- More efficient hardware. Newer inference chips deliver higher throughput at a lower price per unit of compute, pushing down the cost per token directly.
- Scale and competition. The larger the inference cluster and the higher its utilization, the lower the per-unit cost. Meanwhile, competition drives prices down quickly.
- Repricing of demand and capability. Once newer models appear, older ones no longer handle the most cutting-edge workloads. Their value shifts to the “good enough and cheap” range.
The implication is straightforward: if your workloads don’t push against the capability ceiling of these older models, choosing them means very low costs, often far lower than you might expect.
3. New Models: Why They Cost More (and Why It’s Justified)
Prices for the latest flagship models fall much more slowly. The reason is equally clear: demand for their capabilities is strong, so supply and demand will keep their prices high for a considerable time.
These models are expensive, but businesses still pay for them because the quality difference shows up on difficult tasks such as complex reasoning, long-context understanding, code generation, and multimodal work. For many high-stakes use cases, the gains from better output far outweigh the higher API costs.
An apt analogy is fuel grades. Some cars run fine on regular (95 octane), while others need premium (98 octane) to deliver their full performance. AI models are tiered in much the same way: the more capable and cutting-edge a model is, the higher its cost tier. There’s a reason for the premium. The only question is whether your task really needs a “high-octane” model.
4. Two Key Takeaways on Cost
Put these two halves together, and the conflicting claims about AI costs make sense:
- When you hear that “costs have plunged hundreds of times,” the work is likely running on an older model.
- When you hear that “costs remain high,” it’s most likely a newer model.
It’s similar to building a PC. Going with a latest-generation CPU instead of a previous-generation one can make the cost of the same machine vastly different. Model choice has a comparable impact on the cost of an AI application. So there is no single answer to “when will AI costs come down.” It depends on which generation of models you use.
5. Engineering for Lower Costs: Task Tiering and On-Demand Model Selection
For most tasks that don’t demand the most extreme responsiveness or quality, the most practical approach isn’t to agonize over a single do-it-all model. Instead, tier your tasks and call models accordingly. The core idea is to spend compute where it matters: handle the bulk of simple requests with cheap models and call expensive flagship models only at the steps that truly need them.
A typical tiered architecture looks like this:
Request entry│├── Rules/cache layer: if a cache hit or rule matches, return directly at zero model cost│├── Small/legacy model layer: high-volume simple tasks such as classification, extraction, formatting, and basic Q&A│└── Flagship model layer: a small number of critical tasks such as complex reasoning, long-form writing, and demanding coding work
On top of this layering, a routing policy can decide dynamically which tier handles each request:
def route_request(task):# First assess task difficulty and complexityif is_cacheable(task) or is_simple(task):# Route to cache or small model for the lowest costreturn small_model_handler(task)if requires_deep_reasoning(task):# Only a few critical tasks go to the flagship modelreturn flagship_model_handler(task)# Route tasks of medium complexity to a standard modelreturn standard_model_handler(task)
You can combine the following techniques to reduce overall costs further:
- Cache reuse: Cache results for identical or highly similar requests to avoid repeated inference.
- Batching and concurrent scheduling: Increase inference resource utilization to spread costs across more requests.
- Model composition: Use a small model for a first pass and escalate only hard cases to a large model, reducing the volume of high-cost calls.
- Observability-driven optimization: Use logging and monitoring services with alerting to track success rates, latency, and costs across tiers, and let that data guide model selection and tuning.
For example, in Baidu AI Cloud’s AI capability lineup, combining object storage for caching, message queues to smooth out load spikes, container platforms to host inference services, and monitoring and alerting for cost observability is a broadly accepted set of general engineering practices. This kind of combination can help you put a tiering strategy into practice.
6. Summary: A Cost Problem Is Really a Model Selection Problem
Back to the original question: when will AI costs come down? They’re already declining, and different model generations are moving in different directions. Older models have entered an order-of-magnitude decline and may drop by another order of magnitude. New models will stay expensive for a long time because demand is strong.
What you really need isn’t to wait for some universal price cut. It’s to build a clear tiering strategy for model selection: use older or small models to absorb the bulk of simple requests, and concentrate flagship model compute on the few high-value tasks. Get this right and your AI application’s costs are often much lower than expected. As inference costs keep falling, the architecture will keep delivering additional savings over time.
