Cut API Costs with Tiered Routing: Lightweight Models for Office and Coding Tasks

    This article explains how to calculate LLM API costs from token usage, account for retries, prompt length, output length, and caching, and compare lightweight and standard models using real task samples. It then covers common office and coding scenarios and presents a tiered routing policy that sends simple requests to lightweight models, escalates failures or low-confidence results, and requires human confirmation for irreversible actions.

    The Short Answer

    Send low-complexity tasks to lightweight models first, and fall back to standard models for high-complexity tasks. Costs are calculated by token count. Unit prices should follow the vendor’s current published pricing; do not apply a fixed price ratio.

    1. How Costs Are Calculated

    The cost of a single call is calculated as follows:

    Cost per call = input tokens × input unit price + output tokens × output unit price

    Vendors generally price in units of per million tokens. Normalize the units first, then multiply. For example, assume a model’s input price is 1 currency unit per million tokens. If a call has 2,000 input tokens, the input cost is 2,000 ÷ 1,000,000 × 1 = 0.002 currency units. This unit price is only for illustrating the calculation; consult the vendor’s official website for actual prices.

    Several factors beyond the call itself can change the total cost:

    • Retries: Resending a request after a failure incurs another charge. If the failure rate is p, the average number of calls per task is approximately 1 ÷ (1 − p). At a 10% failure rate, the average number of calls is approximately 1.11, increasing cost by about 11%.
    • Prompt length: System prompts, examples, and formatting requirements all count toward input tokens. The longer the prompt, the higher the fixed overhead per call.
    • Output length: max_tokens can cap the output, but setting it too low can truncate results, so it should be used together with format validation.
    • Caching: Some vendors offer cached billing for repeated prefix content. Check the relevant documentation for availability and discount rates.

    When measuring costs, draw 50 to 200 task samples from real workloads and run each with both models. Many compatible APIs return a usage field in the response that records input and output token counts. Aggregate these values to obtain average cost per task, average latency, and pass rate. Unit price alone cannot show whether cost reduction actually improves efficiency; all three metrics should be reviewed together.

    Use the following table as a cost-check template. Fill in the blank cells with actual data:

    Item Lightweight model Standard model Data source
    Input unit price (per million tokens) To be filled To be filled Vendor website, with date checked
    Output unit price (per million tokens) To be filled To be filled Vendor website, with date checked
    Average tokens per task Measured Measured usage field in API responses
    Average cost per task Calculated by formula Calculated by formula Local statistics
    Pass rate Manual or rule-based judgment Manual or rule-based judgment Sample evaluation

    2. Typical Scenarios and Model Selection

    Most office and lightweight coding work has clear instructions and short inputs, making it suitable for lightweight models. The following table lists common scenarios, recommended models, and acceptance methods:

    Scenario Input characteristics Recommended model Acceptance metric Main risk
    Meeting minutes extraction Transcript within a few thousand characters Lightweight Recall of decisions and action items; completeness of owner fields Transcription errors can cause incorrect owner identification
    Table cleaning and classification Hundreds to thousands of rows of structured data, processed row by row Lightweight Agreement rate with human labels Costs accumulate when there are many rows; batch processing is needed
    PPT outline generation One topic plus several source passages Lightweight Share of key material points covered by the outline; time required for manual edits Outlines can be generic and require manual detail
    Static page skeleton Component-level or page-level requirements Lightweight Code renders directly and no console errors appear Styles and interactions need manual review
    Small script writing Single function with few dependencies Lightweight Runs successfully on test data Operations such as deleting files or sending email require manual confirmation
    Multi-file code refactoring Cross-module dependencies Standard Plan is reviewable and tests pass Incomplete context can easily miss dependencies
    Logical reasoning over long manuals Very long context Standard Key conclusions can be traced to source passages Reasoning cost is high with long context

    The labels “lightweight” and “standard” in the table only indicate the recommended direction. Actual model selection should be based on sample evaluation results from your own workload.

    3. Task Tiering and Routing

    Routing rules determine which requests go to lightweight models and which are escalated to standard models. The following is a starter set of rules; thresholds should be calibrated with your own data:

    1. If the total input is no more than 4,000 tokens and does not involve multi-file dependencies or cross-chapter reasoning, send it to the lightweight model.
    2. If the input exceeds 16,000 tokens or contains multiple related files, send it directly to the standard model.
    3. If the lightweight model’s output fails format validation (for example, JSON parsing fails or a required field is missing), retry once. If it still fails, escalate to the standard model.
    4. Escalate when result confidence is low. Confidence reported by the model itself is not reliable; where rules can make the judgment, prefer rules.
    5. For irreversible operations such as deletion, sending, or deployment, require human confirmation regardless of which model is used.

    Before rolling out routing rules, sample 50 tasks for each task category.