Cost-Aware Scheduling and Quota Monitoring in Multi-Model LLM Environments
This article explains how to control cost and reliability when a platform routes tasks across multiple LLMs. It covers task-specific quality floors, a minimum evaluation set, effective cost calculation with discount windows, gateway routing and fallback by error code, and quota, budget, and alert monitoring.
1. Model Selection: Setting the Quality Floor
After integrating multiple models into one platform, classify tasks first, then set a quality floor for each task type. Tasks such as summarizing meeting notes, rewriting copy, and adding code comments require less reasoning. Writing a complete technical proposal, diagnosing concurrency defects, or reading dozens of pages of contracts requires stronger reasoning and a longer context limit.
When selecting models, check the following dimensions one by one:
- Task complexity: Does the task require multi-step reasoning or planning? Complex tasks should prioritize models with stronger reasoning.
- Context length: Is
context_tokensplusmax_output_tokensbelow the model’s context limit? - Response latency: Does p95 latency on your own samples meet the business wait time?
- Output stability: Does the parse success rate for structured output meet requirements?
- Data sensitivity: Is the data level within the allowed range of the service hosting the model?
Values in the table should be produced by the team using its own samples for evaluation. Do not directly adopt descriptions from marketing materials.
Minimum Evaluation Set Template
Start with 30 samples per task type, covering normal, boundary, and abnormal inputs. Evaluation records should contain at least the following fields:
sample_id,task_type,model,score,eval_date,rubric_version
score ranges from 0 to 1 and is assigned according to the scoring rubric. A model qualifies for a task type when two conditions are met: the average score is no lower than quality_floor, and eval_date is no more than 90 days old. Results older than 90 days are considered expired and are not used in routing calculations.
Selection Procedure and Pitfalls
Follow this order:
- Define
quality_floorfor each task type. - Evaluate candidate models on your own samples, recording scores and dates.
- Calculate effective cost using the formulas in Section 2.
- Check latency and context limits.
- Confirm that data is allowed to be sent to the service hosting the model.
Common mistakes include comparing only consume multipliers while ignoring quality differences, ignoring the end time of a discount window, using expired evaluation results, and sending sensitive data to an unauthorized service.
2. Effective Cost and Discount Windows
2.1 Metering Basis
The platform deducts quota units. This article assumes that 1 token equals 1 base unit. If the platform meters per thousand tokens, convert according to platform rules before applying the following formulas:
- Base usage: input tokens plus output tokens, converted to the platform metering basis.
- Quota deduction = base usage × consume multiplier
- Effective multiplier = consume multiplier × (1 − discount rate)
- Effective cost = base usage × effective multiplier
The discount rate comes from platform-published rules. Internal team agreement discounts must also record their source and effective date for audit.
Assume a task has a base usage of 1000 units. Model A has a multiplier of 1.0 and no discount, so its effective cost is 1000. Model B has a multiplier of 0.4 and receives a 30% discount within a discount window. Its effective multiplier is 0.28, and its effective cost is 280.
Actual output tokens are unknown before the request is sent, so routing uses a conservative upper-bound estimate:
- Estimated base usage =
context_tokens+max_output_tokens, rounded up to an integer number of units.
As long as actual output does not exceed max_output_tokens, the estimate will not be lower than actual base usage. The estimate is used only for routing and prechecks. Deduction is based on units_used returned by the platform, and it should be reconciled with the platform bill periodically.
2.2 Discount Window Rules
Discount windows change the optimal choice. A model that is not cost-effective normally may be the most quota-efficient during a discount period. Model selection must therefore read the current time and calculate with rules in real time, not rely on a static table. Example rules:
discount_rules:- id: model-b-eveningmodel: model-btimezone: Asia/Shanghaiweekdays: [mon, tue, wed, thu, fri]start: "20:00" # inclusiveend: "23:00" # exclusivediscount: 0.30valid_from: "2026-01-01"valid_until: "2026-12-31"apply_on_holiday: false
The holiday calendar is maintained separately in a holidays list. Judgment rules are as follows:
- If the date is in the holiday calendar and
apply_on_holidayis false, the rule is inactive. - If
apply_on_holidayis true, the holiday date is judged byweekdays. - If the holiday calendar for a year is missing, all rules for that year are treated as having no discount, and an alert is sent.
Boundary convention: start is inclusive and end is exclusive. 20:00 is active, 22:59 is active, and 23:00 is inactive. end can be written as "24:00", meaning the end of the day.
A cross-midnight window, such as Monday to Thursday 22:00 to 02:00 the next day, cannot be written as one rule. Split it into two:
- First segment:
weekdaysis Monday to Thursday,startis"22:00",endis"24:00". - Second segment:
weekdaysis Tuesday to Friday,startis"00:00",endis"02:00".
The weekdays of the second segment are the next day of the first segment. Holiday judgment is performed on each segment’s own local date, with no additional linkage.
Example function. The passed utc_now must be timezone-aware:
from datetime import datefrom zoneinfo import ZoneInfoWEEKDAY_NAMES = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"]def rule_active(rule, utc_now, holidays: set[str]) -> bool:local = utc_now.astimezone(ZoneInfo(rule.timezone))d = local.date()if not (date.fromisoformat(rule.valid_from) <= d <= date.fromisoformat(rule.valid_until)):return Falseif WEEKDAY_NAMES[local.weekday()] not in rule.weekdays:return Falseif d.isoformat() in holidays and not rule.apply_on_holiday:return Falsehm = local.strftime("%H:%M") # zero-padded "HH:MM" can be compared as stringsreturn rule.start <= hm < rule.enddef active_discount(model_id, utc_now, rules, holidays) -> float:matched = [r.discount for r in rulesif r.model == model_id and rule_active(r, utc_now, holidays)]return max(matched, default=0.0)
Time and caching conventions:
- Time zone: rules specify a time zone. Timestamps are stored in UTC and converted to the rule time zone at judgment time.
- Rule cache: TTL is 300 seconds. If a fetch fails and the cache has not expired, continue using the cache. If a fetch fails and the cache has expired, treat it as no discount. This is the conservative approach.
- Judgment time: this article judges discounts at the time the request enters the gateway. Actual platform rules may differ, so verify against platform documentation before launch.
3. Gateway Configuration, Routing, and Fallback
3.1 Minimal Configuration
gateway:default_timeout_s: 30max_retries_same_model: 1 # only for TIMEOUT and UPSTREAM_5XXretry_backoff_ms: 500alert_webhook_env: ALERT_WEBHOOK_URLmodels:- id: model-aconsume_multiplier: 1.0context_limit: 128000latency_p95_ms: 2500sensitivity_allowed: [public, internal]api_key_env: MODEL_A_API_KEY # environment variable name only, not the secretquality:summarize: {value: 0.82, eval_date: 2026-10-15}code_review: {value: 0.91, eval_date: 2026-10-15}- id: model-bconsume_multiplier: 0.4context_limit: 64000latency_p95_ms: 1200sensitivity_allowed: [public]api_key_env: MODEL_B_API_KEYquality:summarize: {value: 0.80, eval_date: 2026-10-15}code_review: {value: 0.74, eval_date: 2026-10-15}budget:daily_soft_limit_units: 50000alert_thresholds: [0.8, 1.0]
Secrets are read only through environment variables or a secret management service. Secrets should not appear in configuration files, code, or logs.
3.2 Unified Request and Error Codes
The request structure includes task_type, messages, max_output_tokens, quality_floor, context_tokens, sensitivity, priority, and request_id. The response structure includes model, text, units_used, effective_cost, and latency_ms.
Adapters implement call(req, timeout_s) and raise an exception with an error code on failure. Error handling strategy:
| Error code | Meaning | Retry same model | Fall back to another model | Alert level |
|---|---|---|---|---|
| TIMEOUT | Request timeout | Yes | Yes | warning |
| UPSTREAM_5XX | Upstream service error | Yes | Yes | warning |
| RATE_LIMITED | Upstream rate limit | No | Yes, switch after backoff | warning |
| QUOTA_EXCEEDED | Platform hard quota exhausted | No | No, queue or reject | critical |
| SENSITIVITY_DENIED | Data level not allowed for the model | No | No | error |
| INVALID_REQUEST | Invalid request parameters | No | No | error |
Fallback occurs only among models that satisfy both the data sensitivity level and the quality floor. The gateway does not bypass platform hard quotas to complete a request, and it does not fall back sensitive data to external models.
3.3 Routing and Call Pseudocode
Routing has three steps: filter candidate models, sort by estimated cost, and decide retry or fallback by error code.
import mathimport timefrom datetime import datetime, timezoneRETRY_SAME = {"TIMEOUT", "UPSTREAM_5XX"}FALLBACK = {"TIMEOUT", "UPSTREAM_5XX", "RATE_LIMITED"}def eligible(req, models, today):result = []for m in models:if req.sensitivity not in m.sensitivity_allowed:continueif m.context_limit < req.context_tokens + req.max_output_tokens:continueq = m.quality.get(req.task_type)if q is None or q.value < req.quality_floor:continueif (today - q.eval_date).days > 90: # expired evaluations are not used for routingcontinueresult.append(m)return resultdef estimate_units(req, m, now, rules, holidays):base = req.context_tokens + req.max_output_tokensdiscount = active_discount(m.id, now, rules, holidays)return math.ceil(base * m.consume_multiplier * (1 - discount))def call_with_fallback(req, models, adapters, cfg, rules, holidays, usage_log):now = datetime.now(timezone.utc)candidates = eligible(req, models, now.date())if not candidates:raise RuntimeError("NO_ELIGIBLE_MODEL")candidates.sort(key=lambda m: (estimate_units(req, m, now, rules, holidays),m.latency_p95_ms))last_err = Nonefor m in candidates:for attempt in range(cfg.max_retries_same_model + 1):try:resp = adapters[m.id].call(req, timeout_s=cfg.default_timeout_s)# record only metering fields, not messages or textusage_log.append({"request_id": req.request_id, "model": m.id,"units_used": resp.units_used,"effective_cost": resp.effective_cost,"latency_ms": resp.latency_ms})return respexcept GatewayError as e:last_err = eif e.code not in FALLBACK:raiseif e.code in RETRY_SAME and attempt < cfg.max_retries_same_model:time.sleep(cfg.retry_backoff_ms / 1000)continueif e.code == "RATE_LIMITED":time.sleep(cfg.retry_backoff_ms / 1000)break # switch to next candidate modelraise last_err
Estimates during routing only determine order; they do not decide whether a request is allowed. Admission is decided by quota and budget check results, as described in Section 4.
4. Quota Monitoring and Alerts
Quota is a platform-side hard limit. Once exceeded, requests are rejected or queued. Budget is a soft limit set by the team for early warning. The budget soft limit is controlled by the gateway itself; the hard limit is based on platform settings. The statistics window is divided by UTC calendar day and aligned with the platform billing cycle. The platform bill is authoritative.
Monitor the following metrics:
units_usedandeffective_costaggregated by model and date.- Daily budget usage ratio: used units divided by
daily_soft_limit_units. - Incidence of each error code.
- Fallback rate: requests with fallback divided by total requests.
- Discount hit rate: requests matching discount rules divided by total requests.
- Number of expired evaluations and the age of rule cache usage.
Alert threshold examples are shown below:
| Metric |
