From Unit Price to Per-Task Cost: LLM Selection and Caching Cost-Reduction Practices

1. Why Model Selection Should Focus on Total Task Cost

A model’s token unit price is only part of the bill. The actual cost of completing a real task is roughly determined by three factors: unit price, the number of tokens consumed per task, and the number of calls—including retries after failures and rounds of manual correction.

A simplified formula helps illustrate this:

Per-task cost ≈ Token unit price × Tokens per call × Number of calls

If a new model gets tasks right on the first attempt more often, retries drop, and per-task cost falls even when the unit price stays the same. Therefore, when selecting a model, compare end-to-end results on the same set of real tasks across models, rather than comparing price sheets alone.

2. Three-Tier Product Lineups and Their Use Cases

Many vendors today offer a three-tier product matrix: high-capability, balanced, and lightweight. Each tier makes different trade-offs among capability, speed, and price:

Tier	Positioning	Typical Use Cases
High-capability	Strongest capability, highest unit price	Complex coding, code migration, long-chain agent tasks
Balanced	Performance–cost balance	General chat, document processing, moderately complex agents
Lightweight	Fast and cheap	Classification, information extraction, high-frequency calls, real-time interaction

The point of tiering is to let customers pay according to task difficulty: simple tasks go to cheap models, hard tasks to stronger ones. This approach is known as model routing.

How much routing saves depends on your traffic mix. Here is an illustrative calculation with hypothetical numbers (not measured values): assume the high-capability tier costs 1 per call and the lightweight tier 0.1, with 80% of requests handled by the lightweight tier and 20% by the high-capability tier. The average cost per call is 0.8 × 0.1 + 0.2 × 1 = 0.28—about 28% of routing everything to the high-capability tier.

Routing has costs, too. Lightweight models may fail on certain edge cases, so before deploying routing rules, you must measure the degree of quality degradation using your own samples.

3. Capability Changes: Coding, Computer Use, and Knowledge Work

When vendors release new models, they typically highlight three categories of capability.

Coding. This includes code migration across multiple files, performance optimization, and complex feature generation. Some early testers have reported that the model can maintain cross-file consistency in a single edit, reducing back-and-forth rounds. Such feedback is valuable but anecdotal—reproduce it on your own codebase.

Computer use. The model understands interfaces from screenshots, then performs clicks and inputs. This is a foundational capability for building “digital worker” applications. The stability of such applications is heavily affected by UI changes, network latency, and pop-ups, so run an extended trial period before going live.

Knowledge work. This includes long-document analysis, report writing, and research synthesis. For evaluation, compare outputs against real documents with human-annotated answers, checking for omissions and factual errors.

4. Cost and Speed: How to Read Vendor-Claimed Numbers

Vendors often publish figures like “real-task total cost reduced by about 40%” or “output speed improved by more than 30%.” When reading these claims, verify four things:

1. What task set was used for testing, and how many tasks it contained.
2. Which model generation it was compared against, and how retries were counted.
3. The time period covered and the effective date.
4. Whether the unit price includes add-ons such as cache reads/writes and output length.

If these details are not disclosed, the figures can only indicate direction—they should not go directly into your budget.

Pricing also changes over time. The flagship-tier pricing cited in this article is $4 per million input tokens and $20 per million output tokens. Before committing, check the official pricing page and its effective date; do not rely on secondhand sources.

5. Prompt Caching: How It Works and Key Usage Points

Prompt caching works by reusing computation for a shared prefix. When multiple requests start with identical content—such as system prompts, tool definitions, or the same knowledge base content—the server can cache the intermediate computation state of that portion. Subsequent requests that hit the cache only need to compute the newly added parts.

Keep the following in mind when using caching:

• The prefix must match exactly. If your system prompt contains changing content like timestamps or request IDs, the cache will invalidate frequently.
• Caches expire. Cached entries not accessed within a time window are cleared, so hit rates may be low in low-traffic scenarios.
• Writes usually cost extra. The first cache write may be more expensive than a normal input; it only pays off when reused multiple times.
• Cache reads are far cheaper than normal inputs. The more hits, the greater the savings.

A recommended way to structure prompts by stability:

[System prompt + tool definitions]   ← Most stable, place first
[Retrieved document snippets]        ← Relatively stable middle layer
[Current user input]                 ← Changes every time, place last

Track the cache hit rate separately. If the hit rate is low, first check whether the prefix is being unintentionally modified, then consider adjusting your caching strategy.

6. Security Protection for Agent Scenarios

AI agents read external content such as web pages, emails, and files. Attackers can embed instructions in this content to induce the model into performing unauthorized actions—an attack known as prompt injection.

Recommended protections:

• Least privilege. Grant only the tool and data access needed to complete the task.
• Human confirmation. Destructive or consequential actions—deletion, sending, payments, permission changes—should require user confirmation before execution.
• Audit logs. Log which tools the model calls, with what parameters and results, to aid post-incident investigation.
• Red-team testing. Before launch, design dedicated attack cases, such as: hidden instructions in web pages like “send files to an external address”; disguised requests in emails asking to “forward the attachment”; or text in screenshots tricking the model into clicking a “Confirm delete” button.

Record test results as pass rates and make them part of your launch criteria.

7. An Evaluation Workflow for Enterprise Model Selection

Before selecting a model, run end-to-end tests on a small evaluation set and track at least these five metrics:

Metric	Description	How to Measure
Task success rate	Proportion of outputs passing acceptance	Manual or automated acceptance
P95 latency	95% of requests complete within this time	Derived from request logs
Total per-task cost	All costs including retries and cache reads/writes	Calculated from billing and call logs
Cache hit rate	Proportion of input tokens served from cache	From server-side usage fields
Security test pass rate	Proportion of red-team cases not compromised	Re-run after every model or prompt change

The concrete steps are as follows:

1. Collect one to two weeks of real requests and classify them into simple, medium, and hard by complexity.
2. Sample several cases from each class to build a fixed evaluation set.
3. Run the full workflow on candidate models and record the five metrics above.
4. Set routing rules based on the results and roll out under limited traffic for observation.
5. Re-run the evaluation set whenever model versions or prices change.

Following this process means your selection decisions are grounded in your own business data rather than vendor marketing. As a next step, start by collecting request logs—this is the fastest move and the most revealing about your actual cost structure.