Cutting LLM Costs: Cached Input Pricing and Reasoning Effort Levels

As large language models are increasingly used for complex programming, deep retrieval, and multi-step agent tasks, calling costs have become a real constraint for many teams. If the same task repeatedly carries long context and runs at the highest reasoning intensity at every step, costs rise quickly. This article addresses the problem from two angles: first, using cached input pricing to reduce the cost of repeated context; second, choosing the appropriate reasoning effort level based on task complexity. All prices, tiers, and parameters mentioned here follow the providers’ current documentation, and no unverified percentages are cited.

1. Cached Input Pricing: Making Repeated Context Cheaper

Many providers’ new pricing plans keep base input and output unit prices unchanged, but offer a significantly lower read price for cached input. Discount levels vary by platform and version, so check the current price list directly before making a decision.

Agent workloads benefit most from this mechanism. Within a single task, the model goes through multiple rounds of planning, tool calls, and result reflection, during which the system prompt, conversation history, and code snippets are sent repeatedly. If the prefix stays consistent, the platform can reuse previously processed inputs and avoid redundant computation.

Whether the cache is hit depends on the following conditions:

• The prefix must match exactly: System prompts and fixed reference material must be byte-for-byte identical, with no embedded timestamps or random IDs.
• Minimum length requirement: Most platforms only cache prefixes that exceed a minimum length.
• Reuse within the TTL: Caches have a time-to-live (TTL); after expiration, they need to be written again.
• Append-only updates: Modifying historical messages invalidates all subsequent prefixes.

The following pseudocode shows how to keep the prefix stable to maximize cache hits:

# Pseudocode: keep the prefix stable to improve cache hit rate
SYSTEM_PROMPT = "You are a code review assistant. Strictly follow these guidelines: ..."  # Keep fixed
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
for step in agent_workflow:
    # Only append new tool results; never modify existing history
    messages.append({"role": "tool", "content": get_tool_result(step)})
    # Whether a cache parameter must be explicitly enabled depends on your SDK's documentation
    response = llm_client.chat(messages=messages)
    messages.append({"role": "assistant", "content": response.text})

Before going to production, confirm three things: whether your SDK version supports caching, what the minimum cache length and TTL are, and whether the actual cache hit rate is visible in your billing or usage APIs.

2. Reasoning Effort Levels: Default to Medium, Scale Up or Down by Complexity

The takeaway first: default to Medium, drop to Low for simple queries, raise to High for complex reasoning, and reserve Max for only a small number of very difficult tasks.

Some platforms offer multiple reasoning effort levels from Low to Max, while others provide only an on/off toggle. The number and naming of levels differ across platforms; this article discusses four common levels, and specifics should be verified against the provider’s documentation. Whether reasoning can be fully disabled also depends on the documentation—don’t assume either way.

The table below summarizes suitable scenarios and relative characteristics for each level. Latency and token consumption are qualitative comparisons; actual numbers should be measured in your own workload.

Level	Suitable for	Not suitable for	Latency	Token usage
Low / Minimal	Instruction following, simple factual Q&A, status checks	Multi-step reasoning or code review	Lowest	Fewest
Medium	Everyday development assistance, routine code generation, moderately complex text processing	Rare bugs, long-chain proofs	Medium	Moderate
High	Complex logical reasoning, deep code review, architecture evaluation	High-frequency, low-value requests	Higher	More
Max	Rare low-level bugs, multi-step mathematical proofs, system-level refactoring	Most interactive scenarios	Highest	Most

Higher effort levels mean longer internal reasoning before output. Accuracy generally improves, but latency and cost increase as well. For most applications, it makes more sense to start with Medium to validate the workflow, then raise the level for failed samples, rather than starting with Max from the beginning.

3. Routing Strategy: Mapping Task Signals to Effort Levels

Enterprises can add a lightweight intent-classification layer before each request to determine the effort level based on task type and complexity. Intent classification itself has a cost, so it should be done with a small model or simple rules.

The table below is a starting-point routing example; thresholds need to be tuned against your own data:

Task signal	Recommended level	Timeout handling	Failure handling
Classification, extraction, status checks	Low	Retry once after exceeding P95 latency	Escalate to Medium on failure
Routine development, single-file code generation	Medium	Fall back to Low on timeout and log it	Manual review or escalate to High on failure
Multi-file review, complex reasoning	High	Keep the result and notify the user on timeout	Escalate to Max on failure, with a budget cap
Rare bugs, system-level refactoring	Max	Set a per-task token limit	Stop and output intermediate conclusions once the budget is exceeded

When intent classification confidence falls below the threshold, uniformly fall back to Medium. This avoids underestimating task difficulty while preventing every request from consuming the highest level of resources.

4. Context Length and Knowledge Cutoff: Trust the Model Card

Some models advertise million-token context windows, but the practical usable length is constrained by price, latency, and output quality. The longer the context, the weaker the model’s recall of mid-context information may become. When choosing a model, run recall tests on your own long documents and rely on measured results.

Knowledge cutoff dates should also be verified against the model card. For time-sensitive Q&A, provide up-to-date material in the prompt rather than relying on the model’s built-in knowledge.