Where LLM Compute Flows: Cost Accounting for Non-Explicit Scenarios

1. Data Sources and the Credibility of Conclusions

According to an analysis of approximately 100 trillion tokens from LLM API call data, user behaviors can be categorized into three types: core productivity scenarios (code generation, document drafting, data analysis), general knowledge interaction scenarios (Q&A, learning, creative ideation), and non-explicit scenarios (content rewriting, role-playing, private text generation).

It should be noted that this analysis did not disclose the sample scope, intent tagging methodology, or statistical period. Therefore, the proportions mentioned should only serve as an industry reference and not be directly adopted as a baseline for your own business. If you are making decisions based on this data, you should first verify these ratios using your own call logs.

Additionally, the distribution of token call volume does not equate to revenue distribution. Prices per 1,000 tokens vary significantly across different models. A model that consumes fewer tokens might actually contribute more revenue.

2. Why Non-Explicit Scenarios Consume Compute

The analysis indicates that roughly half of compute consumption is concentrated on generative assistance, creative expression, and privacy-sensitive interactions. These types of requests exhibit several observable characteristics: sessions typically have more turns, contexts are longer, peak traffic occurs at night, and users clear their history more frequently.

To understand the relationship between these characteristics and costs, we must look at the inference mechanism. Large language models generate tokens one by one, and each newly generated token must perform attention calculations with the key-value vectors of all preceding tokens. Inference engines typically use KV Cache to store these computed key-value vectors, avoiding the need to recalculate everything from scratch at each step. Thus, incremental computation in multi-turn dialogues does not restart from zero.

However, as the context lengthens, costs still rise. On one hand, the GPU memory occupied by the KV Cache grows linearly with context length, which squeezes the number of concurrent requests that can be served simultaneously. On the other hand, generating each new token requires reading the entire KV Cache, often making memory bandwidth the first bottleneck. When processing long prompts, the computation required for full attention scales quadratically with length. The longer the dialogue, the higher the per-turn service cost.

The magnitude of these costs depends on the model architecture, whether Grouped-Query Attention (GQA) is used, batching strategies, hardware models, and whether the inference framework supports prefix caching. The cost of the same scenario can vary drastically under different deployment methods, so fixed multipliers cannot be applied across the board.

These scenarios typically do not generate direct revenue but still consume GPU resources. Developers with limited budgets who are unaware of their application’s token flow will find it difficult to pinpoint the cause when their bills spike.

3. Model Selection: Three Actionable Approaches

Closed-source models account for a higher proportion of hosted API calls, while open-source models are more active in self-hosted scenarios. This reflects a selection tendency: many users are willing to pay for more stable inference quality and lower hallucination rates. Whether this conclusion applies to your business depends on your task types.

You can categorize tasks by difficulty:

1. Accuracy-sensitive tasks, such as complex code generation and financial analysis: prioritize more capable closed-source models. A single call is more expensive, but when the cost of an error is higher, this trade-off may be worth it.
2. High-volume, low-complexity tasks, such as content classification and information extraction: consider private deployment of open-source models, combined with quantization and distillation to reduce per-inference costs. Whether this is feasible requires first comparing accuracy on a validation set.
3. Intermediate requests: add an intent classifier at the entry point to route simple questions to a small model and complex questions to a large model. Routing itself incurs overhead, and the classifier’s accuracy must be tested.

The return on investment for routing strategies depends on traffic distribution. In applications where simple requests dominate, the benefits are usually more pronounced. It is recommended to conduct A/B testing with real traffic, comparing both quality and cost metrics before deciding to fully deploy.

4. Caching and Retrieval: Two Technical Paths to Reduce Token Consumption

Many articles conflate semantic caching with KV Cache, but they exist at different layers and must be treated separately.

Semantic caching operates at the application layer. The system saves user questions and model answers. When a new request arrives, it first computes the vector of the question and matches it against existing records for similarity. Upon a hit, it directly returns the cached answer without calling the model. This is suitable for highly repetitive scenarios like common questions or fixed-format rewriting requests. The risks are also obvious: similar does not mean identical, and it may return incorrect answers. If answers involving personal information are reused across users, it can also trigger privacy issues. Therefore, cache keys should include user permissions and session scopes, with reasonable expiration times set.

KV Cache operates inside the inference engine. It caches the key-value vectors corresponding to the same prefix tokens. Matching requires exact, token-by-token consistency with no semantic matching. As long as prefixes like system prompts and fixed documents remain unchanged, and the inference engine supports prefix reuse, redundant computation can be reduced. It does not alter the model’s output, but the slightest change to the prefix will invalidate the cache, making the stability of prompt templates crucial.

Retrieval-Augmented Generation (RAG) is another path. The system first uses an embedding model to recall relevant chunks from a vector database, placing only the content relevant to the current question into the prompt instead of the entire document. This significantly reduces input tokens, provided the retrieval quality is sufficient. If recalled chunks are irrelevant, the model might answer incorrectly or ignore key information in a long prompt, so retrieval effectiveness needs to be evaluated independently.

5. Division of Labor Between Edge and Cloud

The local compute power of smartphones and laptops is improving. Quantized models with around 7B parameters can already run lightweight tasks like text summarization and email polishing on some newer devices. However, memory capacity, heat generation, and battery life will limit their performance, and response speeds vary by device.

Therefore, the key to edge-cloud collaboration is delineating task boundaries. Simple, privacy-sensitive requests should remain local, while complex reasoning is offloaded to the cloud. This division reduces the amount of data uploaded and cloud token expenditures, while also improving the privacy experience. The exact amount of savings needs to be measured based on your specific device models and task distribution.

6. The Cost of Privacy and Compliance

For inputs involving health, emotions, or personal expression, users often prefer to keep them local or use encrypted services that do not retain logs. This drives demand for two types of infrastructure.

The first is a private inference gateway. Requests pass through a local desensitization module first, stripping direct identifiers like names and phone numbers before being sent to the remote model. Desensitization rules need regular auditing; otherwise, indirect identifying information may slip through.

The second is a Trusted Execution Environment (TEE). Some enterprise clients require APIs to run in hardware-encrypted memory regions. Such solutions introduce additional performance overhead, the extent of which depends on the model size, batching strategy, and hardware capabilities.