Why AI Services Charge: Cost Structure and Model Selection
This article explains why AI services have shifted from free access to usage-based pricing, how to estimate the requirements and limits of local deployment, how token-based billing works, why prices can change, and how to evaluate model options and reduce costs in production.
1. Why AI Services Have Moved from Free Access to Usage-Based Billing
Traditional desktop software and mobile apps usually run on users’ own devices. After the provider has built the product, adding another user adds almost no runtime cost, so a free model can be sustained for a long time.
Large model inference works differently. Models have large parameter counts, and every request consumes accelerator memory and compute capacity. The computation usually takes place in a remote data center. The provider’s costs mainly include the following:
- Hardware purchase and depreciation: accelerators, servers, and network equipment must be amortized over several years.
- Power and cooling: under high load, electricity and cooling costs are significantly higher than those of ordinary servers.
- Data center space and bandwidth: facility space, network egress, and data transfer are all billed.
- Operations and R&D: model iteration, system monitoring, and security teams require ongoing investment.
Free quotas are usually a customer acquisition tool. As the user base grows and total compute capacity remains limited, tighter free quotas are almost inevitable. Understanding this helps you judge which service prices are reasonable.
2. How to Estimate the Barrier to Local Deployment
Many developers consider deploying models on their own machines to reduce long-term costs or protect data. Before deployment, you need to estimate memory requirements.
The memory occupied by model weights can be roughly estimated as follows:
Weight memory ≈ number of parameters × bytes per parameter
At FP16 precision, each parameter takes 2 bytes. After INT4 quantization, each parameter takes about 0.5 bytes. Actual runtime also requires additional space for KV cache, activations, and runtime framework overhead. In long-context scenarios, this overhead can be very large.
The following code quickly estimates weight memory for models of different sizes:
def estimate_weight_memory_gb(params_billion: float, bytes_per_param: float) -> float:"""Estimate GPU memory used by model weights in GB, excluding KV cache and runtime overhead."""return params_billion * 1e9 * bytes_per_param / (1024 ** 3)for name, params in [("7B", 7), ("70B", 70)]:for dtype, nbytes in [("FP16", 2), ("INT4", 0.5)]:print(f"{name} {dtype}: {estimate_weight_memory_gb(params, nbytes):.1f} GB")
The results are approximately: a 7B model needs about 13 GB at FP16 and about 3.3 GB at INT4; a 70B model needs about 130 GB at FP16 and about 33 GB at INT4.
This covers only the weights themselves. An online service for multiple users must also consider concurrent requests, throughput requirements, and high-availability redundancy. Some models with a mixture-of-experts architecture activate only part of their parameters for a single inference, but all weights still need to be loaded into memory or GPU memory. As a result, local deployment of very large models usually requires multiple high-end accelerators, large memory capacity, and a stable power supply. The one-time investment is often far beyond the budget of an individual or small team.
3. The Core of Usage-Based Billing: Tokens and Pricing
Most AI services use tokens as the billing unit. A token is the basic unit used when a model processes text. One token may correspond to a character, part of a word, or a piece of punctuation, depending on the model’s tokenizer.
A common billing method prices input and output separately, and output is usually more expensive because the model must generate content token by token. The cost of a single call can be expressed as:
Cost per call = input tokens × input price + output tokens × output price
Actual prices are determined by the provider’s published rates and vary significantly across models and service tiers.
Hidden Costs in Multi-Turn Conversations
In a multi-turn conversation, each request usually carries previous messages. The input for round n is roughly:
Round n input ≈ system prompt + accumulated length of history messages + user input for this round
This means that the longer the conversation, the more input tokens each round consumes, and costs grow cumulatively. Tool calls, web retrieval, and reasoning modes also increase token consumption because retrieval results and intermediate reasoning are also counted.
The following function estimates monthly cost and can be adapted to your own business parameters:
def estimate_monthly_cost(calls_per_day: int,input_tokens: int,output_tokens: int,price_in_per_1k: float,price_out_per_1k: float,days: int = 30,) -> float:"""Estimate monthly cost from average token usage; prices are per 1K tokens."""per_call = (input_tokens / 1000) * price_in_per_1k \+ (output_tokens / 1000) * price_out_per_1kreturn per_call * calls_per_day * days
When using it, average token usage should be calculated from real logs rather than estimated from experience. The sample should cover peak hours and long-conversation scenarios.
4. Why Prices Fluctuate
AI service prices are affected by multiple factors, and developers should leave room for fluctuation in their budgets.
- Supply and demand: when compute and storage resources are tight, pressure on providers to adjust prices increases. If demand grows faster than supply expands, the probability of price increases rises.
- Cost recovery cycle: providers often set prices based on equipment investment and expected recovery periods. Equipment upgrades, changes in electricity prices, and changes in resource utilization all affect unit costs and therefore pricing.
- Pricing announcements and effective dates: providers usually publish a price adjustment notice, then formal pricing, and finally an effective date. There is a time gap between announcement and effective date. Business teams should rely on formal notices and prepare early.
- Reseller markup: when purchasing through an intermediary channel, the final price depends on the upstream price and the channel’s own profit margin.
Common ways to handle price fluctuations include: regularly checking official pricing; setting cost alerts for critical business services; and performing cost sensitivity analysis for core scenarios to measure the budget impact of a 20% or 50% price increase.
5. A Framework for Model and Service Selection
When choosing an AI service, unit price is only one dimension. A more practical approach is to consider the following factors together:
- Task difficulty: tasks such as classification, information extraction, and summarization usually do not require the largest models, while code generation and complex reasoning place higher demands on model capability.
- Output quality: measure answer accuracy with a fixed evaluation set rather than relying on subjective impressions.
- Cost per task: the right comparison is “total cost to complete one task,” not “price per thousand tokens.” A model may have a lower unit price but require multiple retries, resulting in a higher total cost.
- Latency and stability: online interactive scenarios are sensitive to response time, while batch processing scenarios focus more on throughput and cost.
- Data compliance: when personal information, trade secrets, or regulated data are involved, confirm data storage location, retention period, and scope of use.
- Portability: avoid deeply binding business logic to private behavior of a specific interface, and preserve the ability to switch models.
Recommended implementation steps:
- First use a smaller model to run the complete workflow and validate the business logic.
- Upgrade to a larger model only when the smaller model’s evaluation results are insufficient.
- Build a fixed evaluation set and calculate “quality score ÷ cost per task” for horizontal comparison.
- Set usage alerts and monthly budget limits for each environment to prevent abnormal calls from causing runaway costs.
6. Engineering Methods for Reducing Costs
Cost optimization usually starts with reducing token usage. The following is a hypothetical example to illustrate the optimization approach; the numbers do not represent data from any real service.
Suppose a customer support question-answering scenario carries the full conversation history on every request, with an average input of about 6,000 tokens. In an optimized approach, only the most recent four turns are kept verbatim, and earlier content is compressed into a summary, reducing average input to about 2,000 tokens. Based on input tokens, the input cost per request can be reduced by about two-thirds.
Common optimization methods include:
- Context trimming: keep only history relevant to the current question.
- History summarization: periodically compress long conversations into brief summaries instead of raw messages.
- Result caching: return verified answers directly for repeated questions to avoid repeated calls.
- Output constraints: specify answer length and format in the prompt to reduce unnecessary verbose output.
- Request batching: for tasks that can be processed in batches, combine requests to reduce call count and fixed overhead.
After optimization, quality metrics must be checked at the same time. If compressing history reduces answer accuracy, fall back to a more conservative strategy.
7. Common Pitfalls
- Looking only at unit price while ignoring the number of calls and retry costs required to complete a task.
- Ignoring output length and allowing the model to generate more content than necessary.
- Treating free quotas as a long-term cost model and discovering after launch that usage exceeds expectations.
- Sending sensitive data directly to a remote service without a data compliance assessment.
- Failing to set up monitoring and starting investigation only after an abnormal bill appears.
