LLM API Selection: Cost Engineering and Task-Tier Routing

1. The Selection Logic Has Changed

In the past, teams integrating LLMs usually started by asking which model had the strongest capabilities. Once a system goes live, the questions become practical: How much does a single request cost? Is the context window large enough? Is the model stable when an Agent executes multi-step tasks? Does its Chinese output meet business requirements? How much effort is required to integrate and switch models?

This means model selection is an engineering decision, not a comparison of leaderboard rankings. The most capable model is not necessarily suitable for every type of request. A model with the lowest unit price may produce malformed output or require multiple retries, making its overall cost higher. The goal is to find a balance among quality, cost, and latency that satisfies business constraints.

2. Six Evaluation Dimensions

When selecting a model, consider the following six dimensions together, and evaluate each one against your own business data.

1. Price. Look at input and output unit prices separately, because they are usually different. Also confirm whether cache-hit pricing, batch discounts, and price gaps between tiers exist. The same workload may cost several times more on one tier than another, so price must be calculated at the actual request volume.

2. Context window. The context window determines how many tokens a single request can contain. Long-document analysis, code repository understanding, and multi-turn conversation history all depend on this metric. A sufficiently large window does not guarantee that every part of the content will be used accurately, so testing is also needed to confirm how accurately the model locates information in long text.

3. Reasoning capability. Reasoning capability is reflected in tasks involving multi-step logic, mathematical calculation, and multiple constraints. The way to judge it is to construct a set of problems with standard answers and measure accuracy rather than relying on vendor marketing data.

4. Coding and Agent capability. This dimension focuses on three points: the runnable rate of generated code, the stability of tool-calling parameter formats, and the ability to keep the objective from drifting during long tasks. Agent scenarios have high requirements for output-format stability; a single format error can interrupt the entire workflow.

5. Chinese-language and domain quality. Whether Chinese expression is natural, whether technical terms are understood correctly, and whether industry knowledge is reliable all need to be tested with real business samples. General leaderboard scores can only serve as a reference.

6. Response speed and stability. First-token latency affects user perception, overall generation speed affects waiting time for long-text output, and success rate and rate-limiting frequency during peak periods affect system availability. These three items should be measured during periods similar to online traffic.

In addition to the dimensions above, two engineering metrics also need evaluation. The first is integration cost, including API compatibility and SDK maintenance difficulty. The second is replaceability, meaning whether a model can be switched without modifying business code. The better the replaceability, the more room there is to respond to later price adjustments or service changes.

3. From Single Calls to Overall Cost

3.1 Calculating the Cost of a Single Request

Most services bill input tokens and output tokens separately. The cost of a single request can be estimated as follows:

def estimate_cost(input_tokens: int, output_tokens: int,
                  in_price: float, out_price: float) -> float:
    """
    Estimate the cost of a single request.
    in_price and out_price are in CNY per million tokens; the values here are
    illustrative assumptions. Actual values are subject to the latest published rates.
    """
    return (input_tokens * in_price + output_tokens * out_price) / 1_000_000
# Example: 20,000 input tokens and 3,000 output tokens,
# assuming an input price of CNY 10 per million tokens and an output price of CNY 40 per million tokens
cost = estimate_cost(20_000, 3_000, 10, 40)
print(f"Estimated cost per request: about CNY {cost:.2f}")  # Output: Estimated cost per request: about CNY 0.32

The cost of a single request may look low, but business usage usually accumulates daily and monthly. If the average daily request volume is 100,000, a tiny difference in per-request cost can become a significant amount.

3.2 The Cumulative Effect in Agent Scenarios

When an Agent executes a task, each round requires sending historical messages, tool results, and relevant code snippets back into the model. As a result, input tokens continue to grow with the number of rounds.

Suppose a coding Agent uses about 20,000 input tokens per round on average and needs 10 rounds to complete a task. The cumulative input for that task is then about 200,000 tokens. Even at a lower unit price, the input cost of such tasks will be clearly higher than that of ordinary Q&A. If full logs and all file contents are sent to the model in every round, the cost will expand further.

Therefore, cost optimization for Agent systems usually starts from the following directions:

• Compress conversation history and keep only content relevant to the current step.
• Truncate or summarize tool results; for example, keep only error sections and a few lines of context from logs.
• For long repeated prefixes such as system instructions and fixed documents, use cache billing where the service supports it.
• Set a maximum number of rounds for each subtask to prevent the model from falling into ineffective loops.

3.3 Controlling Output Length

The unit price of output tokens is usually higher than that of input tokens, so controlling output length is equally important. Output can be limited through explicit format requirements, field whitelists, and maximum output length parameters. For tasks such as classification and scoring, the model should be required to return only labels or JSON structures and avoid generating explanatory text.

4. Task-Tier Scheduling

Mature systems generally do not use only one model. A more common approach is to tier by task difficulty:

• Light tier: handles high-concurrency, low-complexity tasks such as classification, information extraction, rewriting, and initial sensitive-word screening. This tier pursues low latency and low cost.
• Balanced tier: handles routine Q&A, document summarization, code completion, and general customer service replies. This tier balances quality and cost and carries most traffic.
• Advanced tier: handles complex reasoning, critical Agent planning, and difficult code defect localization. This tier is called less frequently but has high requirements for result quality.

4.1 Routing and Escalation Strategy

After tiering, a routing module is needed to decide which tier a request enters. Routing rules can be based on features such as task type, input length, and risk level. At the same time, an escalation mechanism should be set up: if the output from the light tier fails format validation or confidence checks, it is automatically escalated to the next tier for retry.

TIER_MODELS = {
    "light": "small-model",       # Light tier; the specific name is determined by configuration
    "standard": "balanced-model", # Balanced tier
    "advanced": "top-model",      # Advanced tier
}
LIGHT_TASKS = {"classify", "extract", "rewrite"}
def route(task) -> str:
    """Select the initial tier based on task features."""
    if task.kind in LIGHT_TASKS and task.risk_level == "low":
        return "light"
    if task.needs_multi_step_reasoning or task.risk_level == "high":
        return "advanced"
    return "standard"
def run_with_escalation(task, call_model, validate):
    """Call the model and escalate tier by tier when output validation fails."""
    order = ["light", "standard", "advanced"]
    start = order.index(route(task))
    for tier in order[start:]:
        result = call_model(TIER_MODELS[tier], task.prompt)
        if validate(result):
            return result
    raise RuntimeError("No tier passed output validation; manual handling required")

This code illustrates the structural approach. In a real system, routing rules need to be adjusted iteratively based on business data, and the number of escalations should also be capped to prevent cost overruns caused by overly strict validation rules.

4.2 Benefits and Costs of Tiering

The main benefit of tiering is cost control while ensuring the quality of critical tasks. The cost is increased system complexity: multiple model configurations, routing rules, and validation logic must be maintained, and quality must be monitored separately for each tier. Therefore, tiering is suitable for systems with relatively high traffic and many task types. For prototypes with low traffic, choosing one balanced model and abstracting the interface is often more appropriate.

5. Use Your Own Evaluation Set to Decide the Model

Public leaderboards reflect general capabilities and may not match a specific business. The most reliable basis for selection is an internal evaluation set.

5.1 Steps to Build an Evaluation Set

1. Collect samples. Extract several hundred typical requests from real business logs, covering common cases and edge cases.
2. Label expected outputs. Provide a standard answer or scoring rule for each sample. For open-ended tasks, define clear scoring criteria and have two or more reviewers score independently.
3. Fix the evaluation script. Ensure that inputs, parameters, prompts, and scoring methods are exactly the same across models; otherwise, the comparison is meaningless.
4. Set up a control group. Test candidate models and the current online model on the same set of samples.

5.2 Key Metrics

Metric	Meaning	Decision Use
Task accuracy	Whether the output meets the standard answer or scoring rule	Determine whether quality is acceptable
Format compliance rate	Whether structured output can be parsed correctly	Determine whether it can be used in automated workflows
Average cost per request	Cost calculated from actual token usage	Determine economic efficiency
P95 latency	Upper bound of response time for 95% of requests	Determine user experience and timeout risk
Failure rate	Proportion of timeouts, rate limiting, and errors	Determine stability

Only by combining these metrics can you reach the conclusion of “lowest cost while meeting quality requirements.” Looking at a single metric can easily lead to one-sided judgments. For example, a model may have the highest accuracy but exceed the business tolerance limit for latency; it may not be suitable for online scenarios, but it can still be used in offline batch processing.

5.3 The Need for Re-testing

Model versions are updated, and service providers may also adjust pricing and rate-limiting rules. Evaluation results have a shelf life, so re-testing is recommended every quarter or after each version change. At the same time, avoid test-set leakage: do not use evaluation samples for tuning over a long period, or the metrics will gradually become distorted.

6. Engineering Implementation Points

6.1 Unified Calling Layer

Business code should not directly depend on a specific provider’s SDK. It is recommended to add a unified interface layer between the business and the models, responsible for request-format conversion, parameter normalization, logging, and error handling. Model names, price parameters, and timeout settings should be managed in configuration. With this approach, switching models only requires changing configuration, not business logic.

6.2 Rate Limiting, Retries, and Circuit Breaking

When calling external model services, rate limiting and transient failures must be handled. Retries should use exponential backoff with random jitter to avoid a large number of requests being resent at the same time. Retry only timeouts, rate limiting, and server-side errors; parameter errors should not be retried. When the failure rate of a service continuously exceeds a threshold, circuit breaking should be triggered to switch traffic to a backup model or return a degraded result.

6.3 Keys and Sensitive Information

API keys should be stored in environment variables or a dedicated key management service, not written into code repositories or printed to logs. If request content sent to models contains user data, it should be desensitized in advance and handled in compliance with relevant data regulations.

6.4 Cost and Quality Monitoring

Record the input tokens, output tokens, elapsed time, and validation result of each call by task type, and aggregate them into daily or weekly reports. The monitoring dashboard should display both cost and quality metrics to avoid reducing quality just to optimize cost, or pursuing quality while ignoring cost growth.

6.5 Structured Output Validation

For output that needs to be parsed by programs, use explicit format constraints and perform strict validation on the program side. When validation fails, record the original output to support later analysis of prompt or model issues.

7. Implementation Recommendations

LLM API selection can proceed through the following steps:

1. Review core business scenarios and list three to five of the most important task types.
2. Build an internal evaluation set for each task type, labeling expected outputs and scoring criteria.
3. Select two or three candidate models, run evaluations on the same samples, and record quality, cost, and latency.
4. Based on the evaluation results, design a tiering plan and fallback strategy, and build a unified calling layer.
5. Launch with a small traffic canary, observe for a period, and then expand the scope.
6. Review cost and quality data every week and continuously adjust routing rules and prompts.