Choose Models by Task Tier: Cost and Caching from Usage Patterns

    Usage data shows developers pick fast, cheaper models for everyday work and stronger reasoning models for high-stakes tasks. This article explains a three-tier routing strategy, how prompt caching changes long-project cost, and how to measure and avoid common mistakes.

    Why usage, not peak capability, drives model choice

    In stats for AI coding tools, the most-called models are often not the strongest. They are the ones that respond quickly, cost little per request, and are good enough for most daily work. Developers optimize for efficiency and cost first, not absolute capability.

    Stronger reasoning models are not idle. For architecture, hard bugs, and cross-module refactors, people still switch up. The two classes coexist as a natural split.

    The implication is straightforward: model choice should be a set of rules that change with the task, not a one-time decision. Sending every request to one model either wastes money on simple work or under-delivers on hard work.

    Three tiers by complexity and risk

    Tier 1: Lightweight models. Completions, naming, format conversion, single-function comments, simple Q&A, boilerplate. Short context, easy to check by hand, low cost of error. Optimize for latency, unit price, and throughput.

    Tier 2: General models. Single-module features, small multi-file edits, routine APIs, tests. Need some project context and logical consistency, but not system-level decisions.

    Tier 3: High-end reasoning models. Architecture, performance bottlenecks, concurrency and consistency, refactor plans for legacy code. Long reasoning chains and high cost of mistakes. Extra spend is worth less rework.

    Define entry and exit for each tier. If lightweight output fails review on logic, escalate to a general or reasoning model instead of patching repeatedly.

    Why cache hit rate dominates cost on long projects

    Long projects send the same prefixes over and over: system instructions, repo layout, style guides, history, and frequently cited files. If the provider supports prompt caching, a prefix identical to the previous request can be reused instead of recomputed.

    Effective cost of a request can be written as:

    effective cost = cached input tokens × cached input price + uncached input tokens × input price + output tokens × output price

    Use the provider’s actual prices. When a long, stable prefix is reused, the cached share is high and total cost drops. In long sessions and large codebases that gap compounds.

    Ways to raise hit rate:

    1. Keep the start of the system prompt fixed. Do not put timestamps or random IDs there.
    2. Put volatile content (current task, latest error logs) at the end of the request.
    3. Load project context files in a stable order.
    4. Periodically compress settled conclusions in long chats into a stable summary so noise does not break the prefix.

    Routing in code

    A thin router at the call site is enough. Names and thresholds should match your team.

    1. from dataclasses import dataclass
    2. @dataclass
    3. class Task:
    4. description: str
    5. files_involved: int
    6. needs_architecture: bool
    7. error_cost: str # "low" | "medium" | "high"
    8. FAST_TIER = "fast-model"
    9. GENERAL_TIER = "general-model"
    10. REASONING_TIER = "reasoning-model"
    11. def choose_tier(task: Task) -> str:
    12. # Architecture or high error cost goes straight to reasoning
    13. if task.needs_architecture or task.error_cost == "high":
    14. return REASONING_TIER
    15. # Multi-file or medium risk uses the general tier
    16. if task.files_involved > 1 or task.error_cost == "medium":
    17. return GENERAL_TIER
    18. # Everything else uses the fast tier
    19. return FAST_TIER
    20. def estimate_cost(input_tokens: int, output_tokens: int,
    21. cached_tokens: int, prices: dict) -> float:
    22. uncached = max(input_tokens - cached_tokens, 0)
    23. return (cached_tokens * prices["cached_input"]
    24. + uncached * prices["input"]
    25. + output_tokens * prices["output"])

    Log tier, input/output tokens, and cache hits on every request. That data is the basis for later review.

    Metrics to check whether the split works

    • Cost per task: Average effective cost by task type. High-tier spend should cluster on work that needs it.
    • First-pass rate: Share of code that passes first review or tests. Persistently low rates on the lightweight tier mean thresholds are too loose.
    • Average time: Model latency plus human rework. Cheaper tokens with longer calendar time may not be a win.
    • Cache hit rate: By project and session. Chronically low hits usually mean the context layout needs to be rebuilt.

    Review about every two weeks. Change one threshold at a time so you can tell what moved the numbers.

    Common mistakes

    Always using the strongest model. Quality risk is low, but cost and latency rise a lot. Extra capability rarely pays off on simple tasks.

    Always using the cheapest model. Low unit cost is not low total cost. Rework and review time can wipe out token savings.

    Ignoring context structure. Teams watch unit price and miss the cost of repeated context. Bad cache layout can blow the budget even with the right model.

    No upgrade path. If a “light” task turns out to be architectural, move it up instead of looping the cheap model.

    How to roll it out

    1. Classify recent work by complexity and error cost; write an initial routing rule.
    2. Add the router and usage logs; run two to four weeks for a baseline.
    3. Stabilize prompt prefixes and push dynamic content to the end; watch hit rate.
    4. Tune thresholds from first-pass rate and cost per task; define upgrade conditions.
    5. Put the rules in the team’s engineering guide and check model choice in code review.

    Match model capability to the task. That keeps cost under control and reserves strong reasoning for problems that actually need it.