Multi-Model Task Routing and Cost Control for Office AI Agents
This article presents a general architecture for office AI agents that use multiple LLMs. It covers a decoupled model access layer, task classification and routing by complexity, quota estimation with billing multipliers, and budget gates. It also includes code examples for adapters, routing rules, and cost estimation, along with guidance on configuration and credential management.
1. Background: Multi-Model Support Is Now Common for Office Agents
Tasks in office environments vary widely. Meeting notes, copy polishing, spreadsheet checks, and multi-chapter analytical reports place different demands on model capability, response time, and cost. If every task is sent to a high-performance model, costs rise quickly. If every task is sent to a lightweight model, complex deliverables may not meet quality expectations. As a result, multi-model access and on-demand scheduling have become foundational capabilities for office agents.
This article discusses a general technical approach for such systems and does not recommend or compare specific products.
2. Overall Architecture: Three Decoupled Layers
A practical design splits the system into three layers:
- Model access layer: Abstracts protocol differences across model services and provides a unified invocation interface to upper layers.
- Task scheduling layer: Decides which model to use based on task characteristics, user constraints, and resource conditions.
- Metering and billing layer: Records input/output scale, billing multiplier, and quota deduction results for each call, providing a basis for cost analysis and quota control.
The benefit of this decoupling is that adding or retiring a model only requires changes to an adapter in the access layer. Scheduling policies and billing logic do not need to change.
2.1 Unified Access Abstraction
Model services differ in request format, streaming output, error codes, and timeout behavior. The access layer should define a unified interface and encapsulate these differences in adapters.
from abc import ABC, abstractmethodfrom dataclasses import dataclass@dataclassclass ModelRequest:task_id: strprompt: strmax_output_tokens: inttimeout_s: float = 60.0@dataclassclass ModelResponse:text: strinput_tokens: intoutput_tokens: intmodel_id: strclass ModelAdapter(ABC):model_id: strbilling_multiplier: float # Billing multiplier, provided by configuration@abstractmethoddef invoke(self, req: ModelRequest) -> ModelResponse:...
Adapters are responsible only for protocol conversion. Common logic such as retries, timeouts, and fallback should be handled centrally in the scheduling layer to avoid duplicate implementation in every adapter.
2.2 Configuration and Credential Management
Model endpoints, access credentials, and billing multipliers should be stored in a configuration center or environment variables, not in the code repository. Credentials should follow the principle of least privilege and be rotated regularly. Logs should record request identifiers only, not full prompts or keys.
3. Task Tiering: Assess Complexity Before Selecting a Model
Routing depends on quantifiable task tiering. Common dimensions include:
- Input scale: Text length, attachment page count, table rows and columns.
- Reasoning depth: Whether multi-step derivation, cross-document comparison, or code generation is required.
- Output requirements: Whether structured format, long sections, or strict factual consistency is needed.
- Latency requirements: Whether the user is waiting for an interactive result or can accept asynchronous delivery.
- Error tolerance: The cost of mistakes, such as the difference between financial data and ordinary wording changes.
After scoring these dimensions, tasks can be mapped to three tiers:
| Tier | Typical tasks | Recommended strategy |
|---|---|---|
| Lightweight | Format conversion, summarization, simple rewriting | Economy model, low latency |
| Standard | Meeting notes, email drafting, data interpretation | Balanced model |
| Complex | Multi-document synthesis, long reports, complex reasoning | Premium model, longer latency allowed |
3.1 Example Routing Rules
def classify_task(input_chars: int, needs_multi_step: bool,needs_strict_format: bool) -> str:score = 0if input_chars > 20000:score += 2elif input_chars > 4000:score += 1if needs_multi_step:score += 2if needs_strict_format:score += 1if score >= 4:return "complex"if score >= 2:return "standard"return "light"TIER_TO_MODEL = {"light": "economy-model","standard": "balanced-model","complex": "premium-model",}
Score thresholds should be adjusted based on human evaluation results from historical tasks rather than fixed by intuition. It is advisable to sample and review misclassified cases weekly and observe whether result quality declines after downgrading.
4. Cost Modeling: Include Billing Multipliers in Scheduling
Many platforms charge by quota rather than a per-call unit price. Quota consumption is usually equal to base usage multiplied by the model billing multiplier. The lower the multiplier, the less quota the same task consumes. Billing multipliers should be shown to users in the model selection interface so they can compare the cost of different options directly.
The basic formula for estimated consumption is:
Estimated quota = (input tokens + output tokens × output weight) × billing multiplier × base unit price
The output weight reflects the relative cost of generated output and is usually higher than input. Exact values should follow the rules published by the service provider. The system should treat it as a configurable parameter rather than hard-coding it.
def estimate_quota(input_tokens: int, output_tokens: int,multiplier: float, base_unit: float = 1.0,output_weight: float = 3.0) -> float:effective = input_tokens + output_tokens * output_weightreturn effective * multiplier * base_unit
4.1 Budget Gate
The scheduling layer should check the remaining quota before execution. If the estimated consumption exceeds the current quota, it should first ask the user for confirmation or automatically switch to a model with a lower multiplier. This step can prevent tasks from failing midway due to insufficient quota and wasting content that has already been generated.
