Who Pays for LLMs: Inference Cost Economics, Access Restrictions, and Compliance

1. Starting From a Cat-and-Mouse Game: What Actually Matters Behind the Hype

A large LLM provider restricting access for users in certain regions recently triggered heated discussion across the Chinese-speaking internet. Some people compiled the various restriction rules and workarounds into a single “guide,” and others even built a role-playing mini-game in which players act as a reviewer and try to spot the visitors whose identity does not match. Accessing an online service had turned into a game of cat and mouse. At the same time, accounts for that service circulate privately through some channels, and rate-limit evasion and routing tricks became “common knowledge” in certain circles.

“Who is paying for it” was another topic that came up repeatedly. Such discussions are entertaining, but for developers and enterprise users, a few questions deserve more serious answers:

• What keeps an LLM company running, and who ultimately bears the cost of training and inference?
• Why do platforms restrict access at the level of region, request frequency, and identity? What is the technical reasoning behind it?
• How should companies and individuals use LLM capabilities in a low-cost, compliant way?

The rest of this article follows that order: where the cost comes from, where the money goes, why the restrictions exist, and how to stay compliant while cutting cost.

2. Who Is Paying the Bill: Four Categories of Payers

LLM services are a classic “asset-heavy, operations-heavy” business. Training a frontier model requires large-scale compute clusters, and once it is live, the inference service occupies compute resources for the long term. Broken down by funding source, this kind of service draws revenue from roughly four places.

1. Individual subscribers: paying monthly or annually in exchange for higher usage quotas, faster response times, and advanced features. The upside is predictable revenue and stronger stickiness; the downside is that per-user compute cost swings widely with usage intensity.
2. Usage-based API callers: mainly developers, early-stage teams, and SMBs, paying for actual input and output volume consumed. This revenue grows fast but is highly price-sensitive and easily diverted to cheaper alternatives.
3. Enterprise customers: purchasing team seats, dedicated compute, or industry solutions through contracts. Deal sizes are large and cycles are long, while requirements for stability, data isolation, service-level response, and support are correspondingly higher.
4. Capital investment: in the early commercialization stage, funding is a major source for R&D and compute expansion. The capital market is buying future expectations, which requires revenue to eventually catch up with compute spending; otherwise the valuation logic keeps coming under pressure.

It is worth pointing out that “a lot of traffic from a given region” and “how much revenue that region contributed” are not the same thing. Traffic measures demand and interest; paid conversion is what actually keeps the service running. If a service is heavily occupied by cross-region arbitrage users during a free or low-price period, the gap between real cost and real revenue only widens.

3. Where the Money Goes: What Makes Up a Compute Bill

3.1 Training vs. Inference: Two Different Kinds of Spending

• Training cost: a large one-off investment, including data cleaning and labeling, cluster setup, and fault recovery and repeated training runs in distributed training. It is funded by capital budgets and can be recovered gradually through product revenue over a defined period.
• Inference cost: incurred every second the service is online. It is a marginal cost that directly determines “how much it costs to serve one user” and sets the floor and ceiling of gross margin.

As the user base grows, inference spending usually overtakes training spending and becomes the largest line item on the bill. That is why the sustainability of a business model is really tested by unit inference cost.

3.2 A Framework for Estimating Unit Inference Cost

In general terms, unit inference cost can be decomposed as follows:

Unit inference cost = Total compute cost / Effective tokens processed
Where:
Total compute cost = Number of accelerators × Price per card-hour × Runtime hours
Effective tokens   = Σ(Throughput × Utilization × Runtime) × (1 − Cache hit rate)
Notes:
- A token is the basic unit a model uses to process text; a passage of Chinese
  or English is split into several tokens
- Input (prompt) tokens are used to understand context; output (completion)
  tokens are the content the model generates
- Throughput and utilization vary with input length, generation length, and
  concurrency, so they must be determined through load testing

A simplified example: assume a single accelerator card delivers a throughput of 3,000 tokens per second, average utilization of 40%, and a card-hour cost of $20. The effective output per card per hour is then about 3,000 × 3,600 × 0.4 ≈ 4.32 million tokens, giving a unit compute cost of roughly $20 / 4.32 ≈ $4.6 per million tokens.

With the same hardware, raising utilization to 80% brings the cost down to about $2.3 per million tokens; and if semantic caching covers 30% of requests, the blended cost drops by roughly another 30%. In other words, what determines cost is often not “how expensive the card is” but “whether the card is fully utilized.”

All figures above are illustrative assumptions used only to explain the calculation method. They do not represent the real performance of any specific product or hardware.

3.3 Why LLM Inference Is More Expensive Than Traditional Services

Understanding the cost difference requires understanding the two stages of generative inference:

Stage	Main operations	Computational characteristics
Prefill	Process all input tokens in parallel; compute attention and the key-value cache	Compute-intensive, high accelerator utilization
Decode	Generate output tokens one at a time	Memory-access-bound, naturally low accelerator utilization

Combined, the two produce three amplification effects:

1. Longer input costs more: prefill compute grows roughly quadratically with input length.
2. Longer output is slower: each new token in decode depends on existing context, so it usually cannot be parallelized as fully as prefill.
3. Utilization is naturally uneven: in decode, when only a few requests are in flight at a given moment, expensive compute idles while waiting on memory access.

This is why LLM service pricing generally charges input and output tokens separately: the two token types consume fundamentally different resource profiles.

3.4 Mainstream Engineering Techniques for Lowering Unit Cost

Technique	Basic principle	Main benefit
Continuous batching	Free a slot and admit a new request as soon as one finishes generating, instead of waiting for the whole batch	Higher utilization, lower average latency
Key-value cache reuse	Compute shared prefixes such as system prompts and long documents once and share them across requests	Sharply reduces repeated prefill
Quantization	Store weights and cache at lower precision such as INT8/INT4	Lower memory footprint and access overhead
Speculative decoding	A lightweight “draft model” proposes several tokens, then the large model verifies them in a batch	Faster decoding with essentially unchanged generation quality
Semantic caching	Reuse historical results for requests that are semantically identical or highly similar	Fewer duplicate calls; especially suitable for Q&A and retrieval scenarios
Model tiering / routing	Send simple tasks to small models and reserve large-model calls for complex ones	Lowers average cost while keeping quality in check

Taken together, these techniques show that a drop in unit cost comes down to one thing: getting more effective tokens per card per unit of time.

4. Why Platforms Tighten Access: Four Reasons Behind the Restrictions

Back to the cat-and-mouse joke. If you treat platform restrictions as arbitrary, it is easy to conclude that they exist to make users’ lives difficult. In fact, from an architectural standpoint, most of these measures map onto concrete constraints.

4.1 Compliance: The Dual Boundary of Regulation and Terms of Service

Different countries and regions have their own regulatory requirements for generative AI services, covering content safety, personal information handling, and cross-border data transfer. Providers must configure model policies, content filtering rules, and data retention schemes per region, or they risk being taken down or penalized.

At the same time, most platforms’ terms of service define what is permitted regarding usage regions, account resale, and automated access. The terms themselves are a source of technical constraint.

4.2 Business Logic: Preventing Price Arbitrage and Cost Runaway

Pricing across regions is usually tied to local purchasing power, competitive conditions, and operating costs. If accounts in a low-cost region could indefinitely pay for requests originating from a high-cost region, the result would be obvious adverse selection: heavy, high-intensity users concentrate in the low-price region and the platform’s average service cost keeps climbing.

On top of that, account sharing, resale, and scripted abuse push actual call volume away from projections, undermining capacity planning and cost accounting.

4.3 Security: Blocking Credential Theft and Proxy Abuse

Once API credentials leak, they can be used for bulk scraping, spam content generation, or even other attacks. The platform needs to identify anomalous origins along the request path, for example:

• Many requests in a short time coming from a large number of different networks
• Request characteristics that closely match known automation scripts
• Concurrent sessions on one credential spanning multiple regions and device types

4.4 Accountability: Making Clear “Who Is Responsible for the Output”

When model output turns out to contain illegal or infringing content, the provider faces regulatory accountability. Platforms therefore need to be able to establish who sent a request, when, from where, and in what way. This is the fundamental reason access logs, identity verification, and audit trails exist.

5. Common Access Control Mechanisms: How They Work Together

Turning those constraints into engineering, a public-facing model service typically combines several types of controls:

1. Authentication: credential verification, short-lived tokens, and multi-factor authentication to ensure “the request comes from an authorized party.”
2. Region and network policy: tiered allow or deny decisions based on the location a request originates from and network type (data center, proxy, mobile network).
3. Rate and quota control: algorithms such as token bucket and leaky bucket that cap request frequency and daily quotas per user or per credential.
4. Risk scoring: combining signals such as device fingerprint, behavior sequences, and call patterns to downgrade or add verification for suspicious sessions.
5. Auditing and traceability: recording request metadata and content summaries to satisfy compliance audits and troubleshooting.

A simplified gateway decision flow looks like this:

Request arrives
  │
  ├─ 1. Credential check ─────── fails → reject (401)
  │
  ├─ 2. Quota and rate check ─── exceeded → throttle (429, return Retry-After)
  │
  ├─ 3. Region/network policy ── matches restriction → reject (403)
  │
  ├─ 4. Risk scoring ─────────── high risk → step-up verification or degraded service
  │
  └─ 5. Allow, and write the audit log

Stacked together, these mechanisms form a complete chain that spans “can you access it,” “how much can you use,” and “at what quality.”

6. The Risk Checklist for Circumventing Access: “It Works” Does Not Mean “You Should Use It”

For developers, accessing a restricted service through unofficial channels may look like a short-term fix, but it introduces several layers of risk.

6.1 Credential and Account Security

• Account transfer means loss of credential control: for an account that has changed hands, the original owner may still hold the recovery email, bound phone number, and similar information, and can reclaim it or change the password at any time.
• Session and key leakage: shared configurations, proxy nodes, and third-party “relay endpoints” can all retain credentials along the path, and can even tamper with request content.
• Unclear accountability: if the account is used for prohibited purposes, the actual user will find it hard to prove their innocence.

6.2 Compliance and Data Security

• Cross-border data transfer risk: sending business requests that contain personal information or important data to services outside the jurisdiction may trigger cross-border data transfer obligations.
• Missing audit trail: request content, response content, and user identifiers fall outside the company’s own audit system, making post-incident investigation difficult.
• Uncontrollable supplier: terms of service, pricing, and availability can change at any time, and business continuity has no contractual protection.

6.3 Stability and Quality

• Throttling and bans: abnormal access patterns trigger risk controls, causing intermittent API failures.
• Uncontrollable path: multiple forwarding hops add latency, and observability is poor when failures occur, making root-cause analysis hard.
• Quality drift: different relay implementations may implicitly rewrite prompts and parameters, so output becomes hard to reproduce.

6.4 A Larger Attack Surface

• Prompt injection: an untrusted relay service can embed instructions in returned content, steering the client into performing dangerous operations.
• Man-in-the-middle tampering: without end-to-end verification, response content can be replaced or degraded.

7. Enterprise Practice: Using LLM Capabilities Cheaply and Compliantly

Rather than spending effort on routes around restrictions, it is better to invest in selection and engineering optimization. The checklist below can be taken straight into an architecture review.

7.1 Selection: Compliance and Data Boundaries First

• Define data boundaries: for workloads involving personal information, important data, or customer data, prefer services that support in-region data processing and offer private or dedicated deployment options.
• Check regulatory obligations and credentials: providers offering generative AI services to the public carry their own registration, filing, and safety-assessment obligations, which vary by jurisdiction. Choosing a provider that already meets them reduces your own compliance burden.
• Maintain a multi-provider strategy: keep at least one replaceable alternative for core capabilities so that a single provider’s pricing or policy changes cannot box you in.

7.2 Architecture: Contain Complexity Behind a Unified Gateway

It is advisable to build an internal model access layer that exposes a standard interface outward and connects to one or more backend model services. This layer should handle:

• Unified authentication: short-lived tokens between internal services; no long-lived credentials hard-coded in clients
• Tiered quotas: differentiated quotas and rate limits per team, project, and environment
• Cost attribution: recording token usage per caller to produce shareable cost reports
• Observability: recording request latency, token counts, error codes, and cache hit rates
• Canary and rollback: shifting model versions by traffic percentage and reverting quickly when problems appear

7.3 Cost: Turn “Saving” Into a Quantifiable Engineering Metric

Optimization area	Concrete practice	Measurable metric
Prompt engineering	Trim redundant context and remove duplicate examples	Average input tokens
Cache reuse	Shared-prefix caching and semantic caching	Cache hit rate
Model tiering	Route simple tasks to smaller models	Call share per model and cost per task
Batching and async	Combine non-real-time tasks and run them off-peak	Batch throughput, average utilization
Output control	Set explicit maximum generation length and structured-output constraints	Average output tokens

Make “inference cost per 1,000 business requests” a routine operations metric in weekly reports or monthly reviews, rather than something you dig into only after an anomalous bill arrives.

7.4 Observability: Make Cost Visible and Traceable

• Metrics layer: request volume, success rate, time to first token, end-to-end latency, token throughput, GPU utilization
• Log layer: redacted request summaries, model version, latency breakdown (prefill/decode)
• Alerting layer: month-over-month spikes in unit cost, sudden drops in cache hit rate, abnormal quotas for a given caller

On Baidu AI Cloud, capabilities like these are typically carried by general-purpose cloud products such as cloud monitoring and log services, so enterprises do not need to build a complete observability stack from scratch.

8. Action Checklist: Seven Recommendations for Developers

1. Do the math before choosing a vendor: estimate unit cost with card-hour cost / effective token output instead of looking only at API list prices.
2. Treat utilization as the number-one metric: gains from batching, caching, and queue scheduling often outweigh switching to a cheaper model.
3. Filter vendors by data boundaries: for scenarios involving sensitive data, prioritize in-region compliance and no data leaving the jurisdiction.
4. Build a unified gateway: get authentication, quotas, auditing, and cost attribution right once, instead of wiring up a separate integration for every business line.
5. Stay out of credential gray markets: the security losses from resold accounts and shared keys usually far exceed the money saved.
6. Add multi-provider redundancy: keep switching capability on core paths and isolate concrete implementations behind abstract interfaces.
7. Write compliance into the architecture: registration, log retention, and data masking belong in the design phase, not patched in right before launch.

9. Closing Thoughts

The answer to “who is keeping LLMs alive” ultimately comes down to two numbers: the compute cost per token, and the real number of users willing to pay for that capability. Hype can be manufactured through topics; cost can only be pushed down through engineering, one increment at a time.

For developers, instead of studying how to bypass restrictions, it pays to focus on prompt optimization, cache reuse, model tiering, and a unified gateway. This work directly improves both cost and experience, and it makes the system more resilient when a provider changes its policies. Once the hype fades, what remains are the teams that accounted for cost clearly and drew their boundaries well.