Cut API Costs with Tiered Routing: Lightweight Models for Office and Coding Tasks
This article explains how to calculate LLM API costs from token usage, account for retries, prompt length, output length, and caching, and compare lightweight and standard models using real task samples. It then covers common office and coding scenarios and presents a tiered routing policy that sends simple requests to lightweight models, escalates failures or low-confidence results, and requires human confirmation for irreversible actions.
The Short Answer
Send low-complexity tasks to lightweight models first, and fall back to standard models for high-complexity tasks. Costs are calculated by token count. Unit prices should follow the vendor’s current published pricing; do not apply a fixed price ratio.
1. How Costs Are Calculated
The cost of a single call is calculated as follows:
Cost per call = input tokens × input unit price + output tokens × output unit price
Vendors generally price in units of per million tokens. Normalize the units first, then multiply. For example, assume a model’s input price is 1 currency unit per million tokens. If a call has 2,000 input tokens, the input cost is 2,000 ÷ 1,000,000 × 1 = 0.002 currency units. This unit price is only for illustrating the calculation; consult the vendor’s official website for actual prices.
Several factors beyond the call itself can change the total cost:
- Retries: Resending a request after a failure incurs another charge. If the failure rate is p, the average number of calls per task is approximately 1 ÷ (1 − p). At a 10% failure rate, the average number of calls is approximately 1.11, increasing cost by about 11%.
- Prompt length: System prompts, examples, and formatting requirements all count toward input tokens. The longer the prompt, the higher the fixed overhead per call.
- Output length:
max_tokenscan cap the output, but setting it too low can truncate results, so it should be used together with format validation. - Caching: Some vendors offer cached billing for repeated prefix content. Check the relevant documentation for availability and discount rates.
When measuring costs, draw 50 to 200 task samples from real workloads and run each with both models. Many compatible APIs return a usage field in the response that records input and output token counts. Aggregate these values to obtain average cost per task, average latency, and pass rate. Unit price alone cannot show whether cost reduction actually improves efficiency; all three metrics should be reviewed together.
Use the following table as a cost-check template. Fill in the blank cells with actual data:
| Item | Lightweight model | Standard model | Data source |
|---|---|---|---|
| Input unit price (per million tokens) | To be filled | To be filled | Vendor website, with date checked |
| Output unit price (per million tokens) | To be filled | To be filled | Vendor website, with date checked |
| Average tokens per task | Measured | Measured | usage field in API responses |
| Average cost per task | Calculated by formula | Calculated by formula | Local statistics |
| Pass rate | Manual or rule-based judgment | Manual or rule-based judgment | Sample evaluation |
2. Typical Scenarios and Model Selection
Most office and lightweight coding work has clear instructions and short inputs, making it suitable for lightweight models. The following table lists common scenarios, recommended models, and acceptance methods:
| Scenario | Input characteristics | Recommended model | Acceptance metric | Main risk |
|---|---|---|---|---|
| Meeting minutes extraction | Transcript within a few thousand characters | Lightweight | Recall of decisions and action items; completeness of owner fields | Transcription errors can cause incorrect owner identification |
| Table cleaning and classification | Hundreds to thousands of rows of structured data, processed row by row | Lightweight | Agreement rate with human labels | Costs accumulate when there are many rows; batch processing is needed |
| PPT outline generation | One topic plus several source passages | Lightweight | Share of key material points covered by the outline; time required for manual edits | Outlines can be generic and require manual detail |
| Static page skeleton | Component-level or page-level requirements | Lightweight | Code renders directly and no console errors appear | Styles and interactions need manual review |
| Small script writing | Single function with few dependencies | Lightweight | Runs successfully on test data | Operations such as deleting files or sending email require manual confirmation |
| Multi-file code refactoring | Cross-module dependencies | Standard | Plan is reviewable and tests pass | Incomplete context can easily miss dependencies |
| Logical reasoning over long manuals | Very long context | Standard | Key conclusions can be traced to source passages | Reasoning cost is high with long context |
The labels “lightweight” and “standard” in the table only indicate the recommended direction. Actual model selection should be based on sample evaluation results from your own workload.
3. Task Tiering and Routing
Routing rules determine which requests go to lightweight models and which are escalated to standard models. The following is a starter set of rules; thresholds should be calibrated with your own data:
- If the total input is no more than 4,000 tokens and does not involve multi-file dependencies or cross-chapter reasoning, send it to the lightweight model.
- If the input exceeds 16,000 tokens or contains multiple related files, send it directly to the standard model.
- If the lightweight model’s output fails format validation (for example, JSON parsing fails or a required field is missing), retry once. If it still fails, escalate to the standard model.
- Escalate when result confidence is low. Confidence reported by the model itself is not reliable; where rules can make the judgment, prefer rules.
- For irreversible operations such as deletion, sending, or deployment, require human confirmation regardless of which model is used.
Before rolling out routing rules, sample 50 tasks for each task category.
