LLM Selection: Balancing Budget, Accuracy, and Scheduling
Selecting the right Large Language Model requires balancing per-call pricing, task success rates, and error correction costs. This article provides a quantitative framework for calculating the total cost per successful task, offers guidelines for evaluating models using metrics like Task Success Rate (TSR) and Effectiveness Per Token (ETP), and outlines strategies for routing tasks between smaller and larger models. Additionally, it discusses cost control rules and cloud platform capabilities to optimize LLM deployments.
Key Takeaways
- When comparing models, evaluate the total cost per successful task rather than the per-call price.
- Whether a larger model saves money depends on the error correction cost of a failure. When correction costs are high, larger models are typically more economical.
- Assign simple extraction and classification tasks to smaller models. Delegate multi-step planning, long-context processing, and complex tool calling to larger models, and establish escalation and fallback rules for both.
1. A Lower Per-Call Price is Not Always Cheaper
Smaller models have a lower price per call, but if they fail to understand requirements accurately, they generate three types of overhead:
First, retries. When a model misunderstands a task, an additional call is required, often with more detailed instructions. Each retry incurs token fees.
Second, manual debugging. If the output format is incorrect or contains errors, engineers must investigate the cause, modify prompts, or add validation rules.
Third, downstream overhead. Erroneous results propagate to subsequent steps, causing task orchestration interruptions, decreased cache hit rates, and repeated processing of identical requests.
These overheads are rarely reflected in the unit price sheet. If a team only compares per-call pricing, they underestimate the total cost of smaller models. Larger models have a higher unit price, but if they can decompose tasks correctly on the first try, the number of retries and manual debugging sessions decreases significantly. Conversely, using large models for simple tasks unnecessarily inflates the budget. Both scenarios must be validated with data rather than relying solely on experience.
2. A Formula for Calculating Cost Per Successful Task
Let us define three variables:
p: Per-call price, which can be expressed in relative units.TSR: Task Success Rate, the proportion of outputs that pass acceptance criteria after a single call.m: Error correction cost per failure, including additional token fees and the monetary equivalent of manual debugging time.
Assuming retries continue until success, the average cost to complete a single task is:
C = p / TSR + m * (1 - TSR) / TSR
Here, p / TSR is the expected call cost, and m * (1 - TSR) / TSR is the expected error correction cost.
Let’s use a set of example parameters to compare two models. Assume the smaller model’s per-call price is 1. These values are hypothetical, set to demonstrate the calculation method, and are not benchmark results. Teams should substitute their own data.
| Parameter | Smaller Model | Larger Model |
|---|---|---|
Per-call price p (relative units) |
1 | 3 |
Task Success Rate TSR |
0.67 | 0.95 |
Based on the smaller model’s TSR, completing 100 tasks requires an average of about 149 calls (100 / 0.67), meaning roughly 1.49 calls per successful task.
The cost per successful task for both options under different error correction costs is as follows:
Error correction cost m |
Smaller Model C |
Larger Model C |
Cheaper Option |
|---|---|---|---|
| 1 | 1.99 | 3.21 | Smaller Model |
| 2 | 2.48 | 3.26 | Smaller Model |
| 5 | 3.96 | 3.42 | Larger Model |
Setting the two costs equal and solving yields a break-even point at m ≈ 3.8. When the error correction cost exceeds approximately 3.8 units, the larger model is cheaper; below this value, the smaller model is more cost-effective.
For instance, consider processing 100,000 tasks per month with m = 5. The smaller model’s total cost is approximately 395,520 units, while the larger model’s is about 342,105 units, making the larger model roughly 13.5% cheaper. Note that this conclusion holds only under the example parameters. Different success rates or correction costs could reverse the outcome.
3. Defining and Measuring Metrics
Task Success Rate (TSR): The number of successful tasks divided by the total number of tasks. A task is considered successful if the output passes structural validation (e.g., correct JSON fields and types) and meets an acceptance checklist (which can be defined by rules or manual review).
Error Correction Cost Coefficient (m): The average additional cost per failure, calculated as extra token fees plus manual debugging time multiplied by the hourly wage. For example, if debugging takes 0.05 hours and the hourly wage is 120 units, the manual cost is 6 units.
Effectiveness Per Token (ETP): The number of successful tasks divided by the total cost, which mathematically equals 1 / C. A higher ETP means more effective tasks are completed per unit of budget.
The following guidelines should be observed during measurement:
- Sample size: At least 200 samples per task type, stratified by complexity.
- Data source: Real traffic from the past 30 days, avoiding manually written ideal cases.
- Fair comparison: Run the same batch of inputs across all candidate models using fixed prompts and temperature parameters.
- Retry limit: Uniformly set to 2 retries; attempts exceeding this are counted as failures.
The pseudocode for the evaluation process is as follows:
function evaluate_model(model, tasks, unit_price, manual_cost_per_failure, max_retries):total_cost = 0success = 0for task in tasks:for attempt in 1..max_retries:result = model.execute(task)total_cost += unit_priceif result.passes_acceptance():success += 1breaktotal_cost += manual_cost_per_failuretsr = success / len(tasks)etp = success / total_costreturn tsr, etp
In this process, unit_price and manual_cost_per_failure must use the same currency or relative unit, and max_retries must align with the production strategy; otherwise, the results are not comparable.
4. Task Routing Thresholds and Fallback Rules
The routing layer is responsible for assigning requests to the appropriate model. The table below presents a starting configuration; thresholds should be adjusted based on sample calibration results.
| Task Type | Model Tier | Confidence Threshold | Fallback Action |
|---|---|---|---|
| Extraction and classification with explicit fields | Smaller model | >= 0.85 | Return directly; escalate to medium model if below threshold |
| Multi-step planning, tool calling | Larger model | No threshold | Transfer to manual queue after 2 consecutive failures |
| Ambiguous long-tail requests | Smaller model (pre-analysis) | Escalate if intent confidence < 0.7 | Upgrade to larger model for re-parsing |
| Code generation, complex document processing | Larger model | No threshold | Output results only after passing test cases |
The confidence score output by the pre-analysis model can be provided by model self-evaluation or rule-based validation. Confidence from both sources must be verified on samples to confirm its correlation with actual accuracy.
For cost control, the following rules are recommended:
- A maximum of 2 retries per task; upon exceeding this, downgrade the request or transfer to a human.
- Trigger alerts when monthly quota usage reaches 70% and 90%.
- If a high-tier model times out or exceeds the single-task budget, switch to a lower-tier model and flag the result as low confidence.
5. When to Use Larger Models
Larger models are suitable for the following scenarios: long-context understanding, multi-step planning, complex tool calling, and tasks where the cost of a single failure is high. In these instances, a single correct parsing prevents multiple subsequent calls.
Larger models are not suitable for extraction with explicitly defined fields, simple classification, or fixed-template text rewriting. Smaller models typically handle these tasks stably, and using large models only adds unnecessary unit price.
A common practice is to let the larger model handle only task decomposition, outputting an execution plan, which is then executed by smaller models or deterministic tools. The prerequisite for this approach is that the decomposition step itself requires few calls and produces stable outputs. If the decomposition results frequently require correction, the strategy should be re-evaluated.
6. Cloud Platform Evaluation Checklist
When selecting a model service, infrastructure capabilities significantly impact the cost of implementing the workflow described above. This article does not endorse specific vendors, but the following capabilities can serve as an evaluation checklist:
- Model Gateway: A unified API interface that allows switching models without modifying business code.
- Canary Routing: Directing a percentage of traffic to a new model and comparing the TSR and cost of both groups.
- Budget Control: Support for quota monitoring, rate limiting on overage, and alerts.
- Call Logging and Distributed Tracing: The ability to aggregate success rates and costs by task type.
- Offline Batch Processing: Support for running non-real-time tasks during off-peak hours with clear pricing rules.
7. Conclusion
First, the primary basis for selection is the cost per successful task, C. Teams should measure TSR and m using 100 to 200 of their own samples, plug them into the formula, and then compare.
Second, budget scale dictates the sensitivity of the comparison. Taking 100,000 tasks per month as an example, if the difference in total cost between two options is smaller than the measurement error, prioritize the option with the lower unit price and easier maintenance.
Third, restrict larger models to scenarios like multi-step planning, long-context, and complex tool calling. Concurrently, implement retry limits, routing thresholds, and budget alerts, downgrading or transferring to humans upon failure according to established rules.
