Multi-Model Scheduling and Quota Billing Design for AI Office Agents

    This article explores the architecture and implementation of multi-model scheduling and quota billing for AI office agents. It covers task classification, model routing strategies, cost calculation using base units and multipliers, time-based discount handling, and execution fallbacks for multi-step tasks. By balancing effectiveness, latency, and cost, developers can build a robust system that optimizes AI task routing and metering.

    1. Background: Why Office Scenarios Require Multiple Models

    AI office agents handle tasks with significant variations. Organizing meeting minutes, rewriting copy, checking spreadsheet data, and developing multi-step research plans each have distinct model requirements. Simple tasks prioritize response speed and unit cost, while complex tasks demand deeper reasoning and output stability. Routing all tasks to a single model typically leads to two issues:

    • Simple tasks are over-processed, leading to higher per-call costs and longer user wait times.
    • Complex tasks are underestimated; insufficient model capabilities result in outputs requiring repeated manual revisions, ultimately reducing efficiency.

    Therefore, a multi-model coexistence architecture has become standard for these products. The key is that the system must make selections based on task characteristics and allow users to understand the cost and trade-offs of each choice.

    From an engineering perspective, a single model invocation must consider three dimensions simultaneously:

    1. Effectiveness: Task completion quality, including factual accuracy, format adherence, and multi-step reasoning completeness.
    2. Latency: The time from submitting a request to receiving a result, with particular attention to tail metrics like P95.
    3. Cost: The fee calculated per invocation or by quota.

    These three dimensions often constrain each other. Improving effectiveness usually implies higher latency and cost, so the essence of a scheduling system is making trade-offs under constraints.

    2. Core Abstraction: From “Choosing a Model” to “Choosing a Strategy”

    Many teams initially interpret the problem as “letting users manually choose a model.” This approach shifts the judgment cost to users, who must understand the differences between each model—a knowledge gap for most people. A more reasonable approach is to split the selection process into two layers:

    • Task Classification: The system determines which complexity tier a task belongs to.
    • Model Mapping: The system selects a specific model within the available models for that tier based on a strategy.

    Users can still retain control by choosing from strategies like “prioritize effectiveness,” “prioritize speed,” or “balanced,” and the system combines this with task classification to deliver the result.

    2.1 Model Capability Profile

    Each model should have a structured capability profile containing at least the following fields:

    Field Meaning Usage
    tier Capability level (basic, standard, advanced) Matches task required level
    multiplier Quota consumption rate relative to the base model Calculates single task cost
    latency_p95_ms Historical P95 latency Determines if time constraints are met
    max_context Maximum context length Excludes models that cannot fit the input
    enabled Currently available Supports gray release and fault removal

    Profile data should be derived from stress testing or production statistics and refreshed regularly. Manually entered numbers easily become outdated or invalid after model version updates.

    2.2 Task Feature Extraction

    Task classification relies on quantifiable features. Common features include:

    • Step count: The number of sub-steps after task decomposition.
    • Multi-hop reasoning requirement: For example, comparing multiple materials and drawing a conclusion.
    • Input context scale: Estimated by tokens or characters.
    • Output format constraints: For instance, requiring valid structured data output.

    These features can be extracted by rules or a lightweight classifier. A rule-based approach is simple to implement and explainable, making it a suitable starting point.

    3. Architecture Design

    A maintainable scheduling layer typically includes the following components:

    1. User Request
    2. │
    3. ▼
    4. ┌─────────────┐
    5. │ Task Parser │ Extracts step count, context scale, reasoning needs
    6. └─────┬───────┘
    7. ▼
    8. ┌─────────────┐
    9. │ Router │ Selects candidate models based on capability tier, quota balance, and strategy
    10. └─────┬───────┘
    11. ▼
    12. ┌─────────────┐
    13. │ Executor │ Invokes the model, handles timeouts and downgrades
    14. └─────┬───────┘
    15. ▼
    16. ┌─────────────┐
    17. │ Quota Meter │ Deducts quota based on rate and time rules, writes to ledger
    18. └─────────────┘

    Each component has a single responsibility, making them easy to test and replace individually. Below is a code skeleton for the routing logic, using only the Python standard library.

    3.1 Model Registry and Task Classification

    1. from dataclasses import dataclass
    2. TIER_ORDER = ["basic", "standard", "advanced"]
    3. @dataclass(frozen=True)
    4. class ModelProfile:
    5. name: str
    6. tier: str
    7. multiplier_bp: int # multiplier in basis points, 10000 means 1.0x
    8. latency_p95_ms: int
    9. max_context: int
    10. enabled: bool = True
    11. REGISTRY = [
    12. ModelProfile("model-a", "basic", 5000, 800, 16000),
    13. ModelProfile("model-b", "standard", 10000, 1500, 64000),
    14. ModelProfile("model-c", "advanced", 20000, 3000, 128000),
    15. ]
    16. def classify(task: dict) -> str:
    17. # Returns the minimum required capability tier based on task features.
    18. score = 0
    19. if task["steps"] > 3:
    20. score += 1
    21. if task["needs_multi_hop"]:
    22. score += 1
    23. if task["context_tokens"] > 32000:
    24. score += 1
    25. return TIER_ORDER[min(score, len(TIER_ORDER) - 1)]

    Integer basis points are used for multipliers to avoid floating-point accumulation errors. Model names should come from configuration in a real system; the values here are for demonstration only.

    3.2 Routing Strategy

    The router first filters out models that do not meet the constraints, then sorts them by strategy:

    1. def route(task: dict, registry: list, budget_left: int, policy: str = "balanced"):
    2. required = classify(task)
    3. required_idx = TIER_ORDER.index(required)
    4. candidates = [
    5. m for m in registry
    6. if m.enabled
    7. and TIER_ORDER.index(m.tier) >= required_idx
    8. and m.max_context >= task["context_tokens"]
    9. ]
    10. if not candidates:
    11. return None
    12. # Remove unaffordable models when quota is insufficient
    13. base = task["base_units"]
    14. affordable = [
    15. m for m in candidates
    16. if base * m.multiplier_bp // 10000 <= budget_left
    17. ]
    18. if not affordable:
    19. return None
    20. if policy == "cost":
    21. key = lambda m: (m.multiplier_bp, m.latency_p95_ms)
    22. elif policy == "speed":
    23. key = lambda m: (m.latency_p95_ms, m.multiplier_bp)
    24. else: # balanced: select the cheapest model within the same tier
    25. key = lambda m: (TIER_ORDER.index(m.tier), m.multiplier_bp)
    26. return min(affordable, key=key)

    Note that the cost strategy pushes tasks to the lowest capability tier, which poses a greater risk to effectiveness. The balanced strategy prioritizes the tier in its sorting key, ensuring the minimum tier that meets requirements is selected, and then finds the cheapest model within that tier. This design is more robust than simply sorting by cost.

    4. Quota Multipliers and Unified Cost

    The per-call cost varies significantly between different models, making it difficult for users to estimate consumption if billed directly by invocation count. A common practice is to introduce two concepts: “base units” and “multipliers”:

    • Base units: Determined by the task itself, such as input/output scale, step count, etc.
    • Multiplier: Determined by the model, representing the consumption coefficient relative to the base model.

    The actual consumption is calculated as follows:

    Actual Consumption = Base Units × Model Multiplier × Time Discount Coefficient

    Here is a demonstration of the calculation. Assuming the base model consumes 100 base units for a specific task, a model with a 0.5x multiplier consumes 50 units, and a model with a 2.0x multiplier consumes 200 units. These values are for illustrating the calculation method and do not represent any real product specifications.

    4.1 Meter Implementation

    1. from dataclasses import dataclass, field
    2. @dataclass
    3. class Ledger:
    4. balance: int
    5. entries: list = field(default_factory=list)
    6. def charge(self, task_id: str, base_units: int,
    7. profile: ModelProfile, discount_bp: int = 10000) -> int:
    8. # Multiply first, then divide using integer arithmetic, and finally apply ceiling to avoid undercharging
    9. raw = base_units * profile.multiplier_bp * discount_bp
    10. cost = -(-raw // (10000 * 10000)) # Ceiling division
    11. if cost > self.balance:
    12. raise ValueError("Insufficient quota")
    13. self.balance -= cost
    14. self.entries.append({
    15. "task_id": task_id,
    16. "model": profile.name,
    17. "cost": cost,
    18. "multiplier_bp": profile.multiplier_bp,
    19. "discount_bp": discount_bp,
    20. })
    21. return cost

    The meter should follow two principles. First, the deduction and the task state write should be placed in the same transaction to prevent scenarios where billing succeeds but the task fails, or the task succeeds but is not billed. Second, the ledger records should retain snapshots of multipliers and discounts. Even if rules change later, historical bills can still be reproduced.

    4.2 Rounding

    The advantage of integer arithmetic is that the results are reproducible. For ceiling division, a common expression is -(-a // b), which is equivalent to applying a ceiling to a / b for positive numbers. If the business requires rounding to the nearest minimum billing unit, it should be clearly stated in the product rules, and boundary values should be covered in tests.

    5. Engineering Implementation of Time-Based Discounts

    Some products offer quota discounts for specific models during certain periods. From an implementation perspective, these rules have three areas prone to error:

    1. Time zones: The server usually stores time in UTC, while rules are defined in the business’s local time zone.
    2. Holidays: An independent calendar dataset must be maintained; it cannot be judged simply by day of the week.
    3. Rule effective time: Rule changes should have a clear effective timestamp to avoid inconsistent billing for the same request before and after the change.
    1. from datetime import datetime, timedelta, timezone
    2. BIZ_TZ = timezone(timedelta(hours=8))
    3. # Example time windows; actual config should be loaded from a rule center
    4. DISCOUNT_WINDOWS = [(9, 12), (14, 18)]
    5. def in_discount_window(ts_utc: datetime, holidays: set) -> bool:
    6. local = ts_utc.astimezone(BIZ_TZ)
    7. if local.date().isoformat() in holidays:
    8. pass # Holiday rules may differ from weekdays; this example uses the same window
    9. hour = local.hour
    10. return any(start <= hour < end for start, end in DISCOUNT_WINDOWS)

    When billing, the timestamp of the request arriving at the gateway should be used, not the time the task completes. Otherwise, long-running tasks may cross period boundaries, causing disputes. For cross-period tasks, discounts can be calculated per step and recorded individually in the ledger.

    6. Multi-Step Task Execution and Fallback

    Office tasks usually contain multiple steps. If the entire task is bound to a single model, a timeout in one step causes the whole task to fail. A better approach is to route per step and design a fallback chain.

    1. class ModelTimeout(Exception):
    2. pass
    3. def run_with_fallback(step: dict, chain: list, execute):
    4. # Try candidate models sequentially along the fallback chain until success or all fail.
    5. last_error = None
    6. for profile in chain:
    7. try:
    8. result = execute(profile, step)
    9. return profile, result
    10. except ModelTimeout as err:
    11. last_error = err
    12. continue
    13. raise RuntimeError("All candidate models failed") from last_error

    Fallback design should note the following:

    • Fallback can only downgrade to models that satisfy the step’s capability requirements; it cannot arbitrarily sink to an unqualified model due to failure.
    • Retries should have backoff intervals and a maximum limit to prevent amplifying downstream pressure.
    • Steps with side effects, such as writing files or sending messages, must have idempotency keys to avoid duplicate operations upon retry.
    • The state of each step should be persisted so that interrupted tasks can resume from the breakpoint rather than starting over.

    7. Quality and Cost Evaluation Methods

    Before a scheduling strategy goes live, data must verify its reasonableness. A feasible evaluation process includes the following steps.

    7.1 Build a Replay Set

    Sample from historical tasks, covering various scenarios: simple Q&A, long document processing, multi-step planning, and structured output. Each sample should save the input, task features, and expected results. The replay set should be updated regularly to avoid drifting from the real production distribution.

    7.2 Core Metrics

    • Task success rate: The proportion of outputs meeting format and business rules.
    • Human scoring: Sample output quality evaluation with consistent grading standards.
    • Average quota consumption: Grouped by tier and strategy.
    • P95 latency: Reflects the tail experience and is better at exposing issues than averages.
    • Fallback rate: The proportion of tasks downgraded to a lower tier. A high fallback rate indicates