Multi-Model Task Routing and Cost Control for Office AI Agents

    This article presents a general architecture for office AI agents that use multiple LLMs. It covers a decoupled model access layer, task classification and routing by complexity, quota estimation with billing multipliers, and budget gates. It also includes code examples for adapters, routing rules, and cost estimation, along with guidance on configuration and credential management.

    1. Background: Multi-Model Support Is Now Common for Office Agents

    Tasks in office environments vary widely. Meeting notes, copy polishing, spreadsheet checks, and multi-chapter analytical reports place different demands on model capability, response time, and cost. If every task is sent to a high-performance model, costs rise quickly. If every task is sent to a lightweight model, complex deliverables may not meet quality expectations. As a result, multi-model access and on-demand scheduling have become foundational capabilities for office agents.

    This article discusses a general technical approach for such systems and does not recommend or compare specific products.

    2. Overall Architecture: Three Decoupled Layers

    A practical design splits the system into three layers:

    1. Model access layer: Abstracts protocol differences across model services and provides a unified invocation interface to upper layers.
    2. Task scheduling layer: Decides which model to use based on task characteristics, user constraints, and resource conditions.
    3. Metering and billing layer: Records input/output scale, billing multiplier, and quota deduction results for each call, providing a basis for cost analysis and quota control.

    The benefit of this decoupling is that adding or retiring a model only requires changes to an adapter in the access layer. Scheduling policies and billing logic do not need to change.

    2.1 Unified Access Abstraction

    Model services differ in request format, streaming output, error codes, and timeout behavior. The access layer should define a unified interface and encapsulate these differences in adapters.

    1. from abc import ABC, abstractmethod
    2. from dataclasses import dataclass
    3. @dataclass
    4. class ModelRequest:
    5. task_id: str
    6. prompt: str
    7. max_output_tokens: int
    8. timeout_s: float = 60.0
    9. @dataclass
    10. class ModelResponse:
    11. text: str
    12. input_tokens: int
    13. output_tokens: int
    14. model_id: str
    15. class ModelAdapter(ABC):
    16. model_id: str
    17. billing_multiplier: float # Billing multiplier, provided by configuration
    18. @abstractmethod
    19. def invoke(self, req: ModelRequest) -> ModelResponse:
    20. ...

    Adapters are responsible only for protocol conversion. Common logic such as retries, timeouts, and fallback should be handled centrally in the scheduling layer to avoid duplicate implementation in every adapter.

    2.2 Configuration and Credential Management

    Model endpoints, access credentials, and billing multipliers should be stored in a configuration center or environment variables, not in the code repository. Credentials should follow the principle of least privilege and be rotated regularly. Logs should record request identifiers only, not full prompts or keys.

    3. Task Tiering: Assess Complexity Before Selecting a Model

    Routing depends on quantifiable task tiering. Common dimensions include:

    • Input scale: Text length, attachment page count, table rows and columns.
    • Reasoning depth: Whether multi-step derivation, cross-document comparison, or code generation is required.
    • Output requirements: Whether structured format, long sections, or strict factual consistency is needed.
    • Latency requirements: Whether the user is waiting for an interactive result or can accept asynchronous delivery.
    • Error tolerance: The cost of mistakes, such as the difference between financial data and ordinary wording changes.

    After scoring these dimensions, tasks can be mapped to three tiers:

    Tier Typical tasks Recommended strategy
    Lightweight Format conversion, summarization, simple rewriting Economy model, low latency
    Standard Meeting notes, email drafting, data interpretation Balanced model
    Complex Multi-document synthesis, long reports, complex reasoning Premium model, longer latency allowed

    3.1 Example Routing Rules

    1. def classify_task(input_chars: int, needs_multi_step: bool,
    2. needs_strict_format: bool) -> str:
    3. score = 0
    4. if input_chars > 20000:
    5. score += 2
    6. elif input_chars > 4000:
    7. score += 1
    8. if needs_multi_step:
    9. score += 2
    10. if needs_strict_format:
    11. score += 1
    12. if score >= 4:
    13. return "complex"
    14. if score >= 2:
    15. return "standard"
    16. return "light"
    17. TIER_TO_MODEL = {
    18. "light": "economy-model",
    19. "standard": "balanced-model",
    20. "complex": "premium-model",
    21. }

    Score thresholds should be adjusted based on human evaluation results from historical tasks rather than fixed by intuition. It is advisable to sample and review misclassified cases weekly and observe whether result quality declines after downgrading.

    4. Cost Modeling: Include Billing Multipliers in Scheduling

    Many platforms charge by quota rather than a per-call unit price. Quota consumption is usually equal to base usage multiplied by the model billing multiplier. The lower the multiplier, the less quota the same task consumes. Billing multipliers should be shown to users in the model selection interface so they can compare the cost of different options directly.

    The basic formula for estimated consumption is:

    Estimated quota = (input tokens + output tokens × output weight) × billing multiplier × base unit price

    The output weight reflects the relative cost of generated output and is usually higher than input. Exact values should follow the rules published by the service provider. The system should treat it as a configurable parameter rather than hard-coding it.

    1. def estimate_quota(input_tokens: int, output_tokens: int,
    2. multiplier: float, base_unit: float = 1.0,
    3. output_weight: float = 3.0) -> float:
    4. effective = input_tokens + output_tokens * output_weight
    5. return effective * multiplier * base_unit

    4.1 Budget Gate

    The scheduling layer should check the remaining quota before execution. If the estimated consumption exceeds the current quota, it should first ask the user for confirmation or automatically switch to a model with a lower multiplier. This step can prevent tasks from failing midway due to insufficient quota and wasting content that has already been generated.