Cost-Aware Scheduling and Quota Monitoring in Multi-Model LLM Environments

    This article explains how to control cost and reliability when a platform routes tasks across multiple LLMs. It covers task-specific quality floors, a minimum evaluation set, effective cost calculation with discount windows, gateway routing and fallback by error code, and quota, budget, and alert monitoring.

    1. Model Selection: Setting the Quality Floor

    After integrating multiple models into one platform, classify tasks first, then set a quality floor for each task type. Tasks such as summarizing meeting notes, rewriting copy, and adding code comments require less reasoning. Writing a complete technical proposal, diagnosing concurrency defects, or reading dozens of pages of contracts requires stronger reasoning and a longer context limit.

    When selecting models, check the following dimensions one by one:

    • Task complexity: Does the task require multi-step reasoning or planning? Complex tasks should prioritize models with stronger reasoning.
    • Context length: Is context_tokens plus max_output_tokens below the model’s context limit?
    • Response latency: Does p95 latency on your own samples meet the business wait time?
    • Output stability: Does the parse success rate for structured output meet requirements?
    • Data sensitivity: Is the data level within the allowed range of the service hosting the model?

    Values in the table should be produced by the team using its own samples for evaluation. Do not directly adopt descriptions from marketing materials.

    Minimum Evaluation Set Template

    Start with 30 samples per task type, covering normal, boundary, and abnormal inputs. Evaluation records should contain at least the following fields:

    1. sample_id,task_type,model,score,eval_date,rubric_version

    score ranges from 0 to 1 and is assigned according to the scoring rubric. A model qualifies for a task type when two conditions are met: the average score is no lower than quality_floor, and eval_date is no more than 90 days old. Results older than 90 days are considered expired and are not used in routing calculations.

    Selection Procedure and Pitfalls

    Follow this order:

    1. Define quality_floor for each task type.
    2. Evaluate candidate models on your own samples, recording scores and dates.
    3. Calculate effective cost using the formulas in Section 2.
    4. Check latency and context limits.
    5. Confirm that data is allowed to be sent to the service hosting the model.

    Common mistakes include comparing only consume multipliers while ignoring quality differences, ignoring the end time of a discount window, using expired evaluation results, and sending sensitive data to an unauthorized service.

    2. Effective Cost and Discount Windows

    2.1 Metering Basis

    The platform deducts quota units. This article assumes that 1 token equals 1 base unit. If the platform meters per thousand tokens, convert according to platform rules before applying the following formulas:

    • Base usage: input tokens plus output tokens, converted to the platform metering basis.
    • Quota deduction = base usage × consume multiplier
    • Effective multiplier = consume multiplier × (1 − discount rate)
    • Effective cost = base usage × effective multiplier

    The discount rate comes from platform-published rules. Internal team agreement discounts must also record their source and effective date for audit.

    Assume a task has a base usage of 1000 units. Model A has a multiplier of 1.0 and no discount, so its effective cost is 1000. Model B has a multiplier of 0.4 and receives a 30% discount within a discount window. Its effective multiplier is 0.28, and its effective cost is 280.

    Actual output tokens are unknown before the request is sent, so routing uses a conservative upper-bound estimate:

    • Estimated base usage = context_tokens + max_output_tokens, rounded up to an integer number of units.

    As long as actual output does not exceed max_output_tokens, the estimate will not be lower than actual base usage. The estimate is used only for routing and prechecks. Deduction is based on units_used returned by the platform, and it should be reconciled with the platform bill periodically.

    2.2 Discount Window Rules

    Discount windows change the optimal choice. A model that is not cost-effective normally may be the most quota-efficient during a discount period. Model selection must therefore read the current time and calculate with rules in real time, not rely on a static table. Example rules:

    1. discount_rules:
    2. - id: model-b-evening
    3. model: model-b
    4. timezone: Asia/Shanghai
    5. weekdays: [mon, tue, wed, thu, fri]
    6. start: "20:00" # inclusive
    7. end: "23:00" # exclusive
    8. discount: 0.30
    9. valid_from: "2026-01-01"
    10. valid_until: "2026-12-31"
    11. apply_on_holiday: false

    The holiday calendar is maintained separately in a holidays list. Judgment rules are as follows:

    • If the date is in the holiday calendar and apply_on_holiday is false, the rule is inactive.
    • If apply_on_holiday is true, the holiday date is judged by weekdays.
    • If the holiday calendar for a year is missing, all rules for that year are treated as having no discount, and an alert is sent.

    Boundary convention: start is inclusive and end is exclusive. 20:00 is active, 22:59 is active, and 23:00 is inactive. end can be written as "24:00", meaning the end of the day.

    A cross-midnight window, such as Monday to Thursday 22:00 to 02:00 the next day, cannot be written as one rule. Split it into two:

    • First segment: weekdays is Monday to Thursday, start is "22:00", end is "24:00".
    • Second segment: weekdays is Tuesday to Friday, start is "00:00", end is "02:00".

    The weekdays of the second segment are the next day of the first segment. Holiday judgment is performed on each segment’s own local date, with no additional linkage.

    Example function. The passed utc_now must be timezone-aware:

    1. from datetime import date
    2. from zoneinfo import ZoneInfo
    3. WEEKDAY_NAMES = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"]
    4. def rule_active(rule, utc_now, holidays: set[str]) -> bool:
    5. local = utc_now.astimezone(ZoneInfo(rule.timezone))
    6. d = local.date()
    7. if not (date.fromisoformat(rule.valid_from) <= d <= date.fromisoformat(rule.valid_until)):
    8. return False
    9. if WEEKDAY_NAMES[local.weekday()] not in rule.weekdays:
    10. return False
    11. if d.isoformat() in holidays and not rule.apply_on_holiday:
    12. return False
    13. hm = local.strftime("%H:%M") # zero-padded "HH:MM" can be compared as strings
    14. return rule.start <= hm < rule.end
    15. def active_discount(model_id, utc_now, rules, holidays) -> float:
    16. matched = [r.discount for r in rules
    17. if r.model == model_id and rule_active(r, utc_now, holidays)]
    18. return max(matched, default=0.0)

    Time and caching conventions:

    • Time zone: rules specify a time zone. Timestamps are stored in UTC and converted to the rule time zone at judgment time.
    • Rule cache: TTL is 300 seconds. If a fetch fails and the cache has not expired, continue using the cache. If a fetch fails and the cache has expired, treat it as no discount. This is the conservative approach.
    • Judgment time: this article judges discounts at the time the request enters the gateway. Actual platform rules may differ, so verify against platform documentation before launch.

    3. Gateway Configuration, Routing, and Fallback

    3.1 Minimal Configuration

    1. gateway:
    2. default_timeout_s: 30
    3. max_retries_same_model: 1 # only for TIMEOUT and UPSTREAM_5XX
    4. retry_backoff_ms: 500
    5. alert_webhook_env: ALERT_WEBHOOK_URL
    6. models:
    7. - id: model-a
    8. consume_multiplier: 1.0
    9. context_limit: 128000
    10. latency_p95_ms: 2500
    11. sensitivity_allowed: [public, internal]
    12. api_key_env: MODEL_A_API_KEY # environment variable name only, not the secret
    13. quality:
    14. summarize: {value: 0.82, eval_date: 2026-10-15}
    15. code_review: {value: 0.91, eval_date: 2026-10-15}
    16. - id: model-b
    17. consume_multiplier: 0.4
    18. context_limit: 64000
    19. latency_p95_ms: 1200
    20. sensitivity_allowed: [public]
    21. api_key_env: MODEL_B_API_KEY
    22. quality:
    23. summarize: {value: 0.80, eval_date: 2026-10-15}
    24. code_review: {value: 0.74, eval_date: 2026-10-15}
    25. budget:
    26. daily_soft_limit_units: 50000
    27. alert_thresholds: [0.8, 1.0]

    Secrets are read only through environment variables or a secret management service. Secrets should not appear in configuration files, code, or logs.

    3.2 Unified Request and Error Codes

    The request structure includes task_type, messages, max_output_tokens, quality_floor, context_tokens, sensitivity, priority, and request_id. The response structure includes model, text, units_used, effective_cost, and latency_ms.

    Adapters implement call(req, timeout_s) and raise an exception with an error code on failure. Error handling strategy:

    Error code Meaning Retry same model Fall back to another model Alert level
    TIMEOUT Request timeout Yes Yes warning
    UPSTREAM_5XX Upstream service error Yes Yes warning
    RATE_LIMITED Upstream rate limit No Yes, switch after backoff warning
    QUOTA_EXCEEDED Platform hard quota exhausted No No, queue or reject critical
    SENSITIVITY_DENIED Data level not allowed for the model No No error
    INVALID_REQUEST Invalid request parameters No No error

    Fallback occurs only among models that satisfy both the data sensitivity level and the quality floor. The gateway does not bypass platform hard quotas to complete a request, and it does not fall back sensitive data to external models.

    3.3 Routing and Call Pseudocode

    Routing has three steps: filter candidate models, sort by estimated cost, and decide retry or fallback by error code.

    1. import math
    2. import time
    3. from datetime import datetime, timezone
    4. RETRY_SAME = {"TIMEOUT", "UPSTREAM_5XX"}
    5. FALLBACK = {"TIMEOUT", "UPSTREAM_5XX", "RATE_LIMITED"}
    6. def eligible(req, models, today):
    7. result = []
    8. for m in models:
    9. if req.sensitivity not in m.sensitivity_allowed:
    10. continue
    11. if m.context_limit < req.context_tokens + req.max_output_tokens:
    12. continue
    13. q = m.quality.get(req.task_type)
    14. if q is None or q.value < req.quality_floor:
    15. continue
    16. if (today - q.eval_date).days > 90: # expired evaluations are not used for routing
    17. continue
    18. result.append(m)
    19. return result
    20. def estimate_units(req, m, now, rules, holidays):
    21. base = req.context_tokens + req.max_output_tokens
    22. discount = active_discount(m.id, now, rules, holidays)
    23. return math.ceil(base * m.consume_multiplier * (1 - discount))
    24. def call_with_fallback(req, models, adapters, cfg, rules, holidays, usage_log):
    25. now = datetime.now(timezone.utc)
    26. candidates = eligible(req, models, now.date())
    27. if not candidates:
    28. raise RuntimeError("NO_ELIGIBLE_MODEL")
    29. candidates.sort(key=lambda m: (estimate_units(req, m, now, rules, holidays),
    30. m.latency_p95_ms))
    31. last_err = None
    32. for m in candidates:
    33. for attempt in range(cfg.max_retries_same_model + 1):
    34. try:
    35. resp = adapters[m.id].call(req, timeout_s=cfg.default_timeout_s)
    36. # record only metering fields, not messages or text
    37. usage_log.append({"request_id": req.request_id, "model": m.id,
    38. "units_used": resp.units_used,
    39. "effective_cost": resp.effective_cost,
    40. "latency_ms": resp.latency_ms})
    41. return resp
    42. except GatewayError as e:
    43. last_err = e
    44. if e.code not in FALLBACK:
    45. raise
    46. if e.code in RETRY_SAME and attempt < cfg.max_retries_same_model:
    47. time.sleep(cfg.retry_backoff_ms / 1000)
    48. continue
    49. if e.code == "RATE_LIMITED":
    50. time.sleep(cfg.retry_backoff_ms / 1000)
    51. break # switch to next candidate model
    52. raise last_err

    Estimates during routing only determine order; they do not decide whether a request is allowed. Admission is decided by quota and budget check results, as described in Section 4.

    4. Quota Monitoring and Alerts

    Quota is a platform-side hard limit. Once exceeded, requests are rejected or queued. Budget is a soft limit set by the team for early warning. The budget soft limit is controlled by the gateway itself; the hard limit is based on platform settings. The statistics window is divided by UTC calendar day and aligned with the platform billing cycle. The platform bill is authoritative.

    Monitor the following metrics:

    • units_used and effective_cost aggregated by model and date.
    • Daily budget usage ratio: used units divided by daily_soft_limit_units.
    • Incidence of each error code.
    • Fallback rate: requests with fallback divided by total requests.
    • Discount hit rate: requests matching discount rules divided by total requests.
    • Number of expired evaluations and the age of rule cache usage.

    Alert threshold examples are shown below:

    | Metric |