Practices for Multi-Model Integration and Quota Management in Office AI Agents
This article explores the architecture, task routing, metering, and scheduling strategies for implementing multi-model access in office AI agents. It details task classification, cost calculation with multipliers, time-based scheduling, routing logic, fallback mechanisms, and observability to help developers balance performance, latency, and cost effectively.
1. Background
Office AI agents typically receive a goal, break it down into multiple steps, call models and tools to complete each step, and finally deliver the result. Unlike single-turn Q&A, the number of steps, context length, and output format vary significantly across these tasks. Some steps only require classification or field extraction, while others require long-form reasoning, code generation, or cross-document synthesis. If all steps are assigned to the same model, it is often difficult to balance effectiveness, latency, and cost.
Therefore, an increasing number of platforms are beginning to support multi-model integration. The system provides several models simultaneously, which can be selected by the user or the system based on the task. This design raises three direct questions: how to determine which model is suitable for which task, how to make the billing metrics of different models comparable, and how to schedule resources between peak and off-peak hours. The following sections will discuss these issues in detail.
2. Overall Architecture of Multi-Model Integration
A maintainable multi-model system is generally divided into four layers:
- Access Layer: Uniformly encapsulates authentication, request formats, timeout, and retry strategies for various models. The upper-layer business logic only interacts with the unified interface and is not directly aware of underlying differences.
- Routing Layer: Determines which model to call based on task characteristics, user choice, budget balance, and current load.
- Metering Layer: Records the input and output scale of each call and converts them into internal quota according to unified rules.
- Operations Rule Layer: Handles rules such as time-based strategies, active model tags, and free quotas.
The benefit of this layered approach is that when adding or removing models, only the access layer and routing configuration need to be modified, while the billing and operational rules remain stable. If billing logic is scattered across different business branches, settlement errors will increase rapidly as the number of models grows.
Below is a snippet of the model registry code. The values used are for demonstration only and do not represent any actual product parameters:
MODEL_REGISTRY = {"general-fast": {"tier": "light", "cost_multiplier": 0.5},"general-pro": {"tier": "heavy", "cost_multiplier": 1.0},"reasoning-max": {"tier": "expert", "cost_multiplier": 2.0},}
The registry is the data source that both routing and metering rely on. Centralizing model attribute management prevents inconsistent definitions of the same model across different modules.
3. Task Classification: Assess Difficulty First, Then Select Model
Tasks can be categorized into three tiers based on their characteristics:
- Light Tasks: Format conversion, short text summarization, field extraction, and simple rewriting. These tasks have low requirements for reasoning depth but are more sensitive to response speed.
- Regular Tasks: Structured writing, table formatting, and medium-length data analysis. They require a certain level of comprehension, but the cost of errors is manageable.
- Complex Tasks: Multi-document synthesis, long-chain reasoning, and code or solution design. The output quality directly impacts subsequent steps, requiring the highest model capabilities.
Classification criteria typically include input length, expected output length, the need for external tools, and the business’s tolerance for accuracy.
Classification should not rely entirely on the model’s self-assessment. A more robust approach is to prioritize rules: if the input exceeds a certain length threshold, contains tables or code, or the user explicitly requests in-depth analysis, it is directly routed to a higher tier. The remaining tasks can then be preliminarily judged by a lightweight classifier or a small model.
The cost of classification errors must be considered in advance. Misjudging a light task as a complex one only consumes a bit more quota. Assigning a complex task to a lightweight model, however, may yield unusable results, making the rework cost much higher. Therefore, the routing strategy should lean towards conservatism—it is better to spend a little more cost than to downgrade easily.
4. Metering and Multiplier Design
Because different models have different inference costs, platforms usually set a relative multiplier for each model. A lower multiplier means less quota is consumed to complete the same task. When implementing this, three points require attention:
First, the multiplier is a relative value and should be clearly displayed on the model selection interface. Users need to know the impact of choosing different models on their quota; hiding the multiplier will weaken user trust in the billing system.
Second, quotas should be recorded in a unified unit. A common practice is to separately count input and output tokens, multiply them by their respective unit prices, and then multiply by the model multiplier. The formula can be expressed as:
Consumed Quota = (Input Tokens × Input Unit Price + Output Tokens × Output Unit Price) × Model Multiplier
The following function implements this estimation logic:
def estimate_cost(model_id: str, input_tokens: int, output_tokens: int,unit_price_in: float, unit_price_out: float) -> float:multiplier = MODEL_REGISTRY[model_id]["cost_multiplier"]base = input_tokens * unit_price_in + output_tokens * unit_price_outreturn round(base * multiplier, 6)
Third, a single task often involves multiple model calls, such as planning first, then executing, and finally verifying. Metering should be recorded at the granularity of a single call and aggregated at the end of the task, making it convenient for users to check the details.
5. Time-Limited Discounts and Time-Based Scheduling
A common operational practice is to set up time-based strategies, temporarily lowering the multiplier for certain models during specific periods, or reserving free quotas for simple, low-cost tasks. From an engineering perspective, the following points need attention:
- Time Zone and Clock: Time-based judgments should use a unified time zone and rely on server time; client-submitted times should not be trusted.
- Instant Multiplier Recording: Each call should record the actual multiplier applied, rather than recalculating it when the bill is displayed. This avoids disputes across different time periods.
- Configurable Campaigns: The start time, end time, applicable models, and multipliers for campaigns should be managed through configuration. No code changes should be required after a campaign ends.
- Calendar Source: The distinction between working days and holidays should be provided by calendar configurations and must not be hardcoded.
Free quotas can be implemented in the metering layer using a threshold. Requests with costs below the threshold are directly approved and tagged as free, facilitating subsequent statistics on the scale and distribution of free usage.
6. Routing Strategy Implementation
The basic priority for routing is: explicit user choice first, followed by task tier mapping, and finally downgrade based on remaining budget. The sample code is as follows:
TIER_TO_MODEL = {"light": "general-fast","heavy": "general-pro","expert": "reasoning-max",}def select_model(task_tier: str, user_choice: str | None, budget_left: float) -> tuple[str, bool]:"""Returns (model ID, whether a downgrade occurred)."""if user_choice and user_choice in MODEL_REGISTRY:return user_choice, Falsemodel = TIER_TO_MODEL.get(task_tier, "general-pro")if budget_left < 1.0 and model != "general-fast":return "general-fast", Truereturn model, False
There are two notable design points in this logic:
First, downgrades must be explicitly communicated to the user. The downgrade flag returned by the function should be passed to the frontend to prompt the user about which model generated the current result. If the model is silently swapped, users may mistakenly believe the result came from their selected model, thereby affecting their judgment of the output quality.
Second, the routing function only makes decisions and does not handle billing. Billing is uniformly executed by the metering layer, ensuring that adjustments to routing rules do not affect the billing logic.
7. Quality Evaluation and Fallback Mechanism
Routing rules need to be adjusted based on data rather than fixed by experience. It is recommended to build an evaluation set, sample by task type, and compare the performance of different models in terms of accuracy, content completeness, and format compliance. The evaluation results are used to correct tier mappings. For example, if a certain type of regular task already achieves the required compliance rate on a lightweight model, it can be downgraded by one tier.
The fallback mechanism is used to handle call failures. Common triggers include timeouts, unparseable returned content, or model refusal to respond. During a fallback, the system switches to a backup model in a preset order and sets a retry limit to prevent repeated quota consumption. Every fallback should be logged in the metering system to accurately track costs.
8. Observability and Monitoring
The operational state of a multi-model system is more complex than that of a single-model system and requires targeted monitoring metrics. It is recommended to monitor at least the following categories:
- The average number of calls per task, to detect if the planning phase is out of control.
- The call proportion of each model, to evaluate whether the routing strategy meets expectations.
- Average latency and percentile latency, to judge user experience.
- Failure rate and fallback trigger count, to detect stability issues with a specific model.
- Quota consumption distribution, including breakdowns by time period, model, and task tier.
Logs should record task_id, model_id, task tier, applied multiplier, time period flag, and whether a downgrade occurred. These fields support both troubleshooting and bill reconciliation. Alert thresholds should be set based on historical baselines—for example, triggering a notification when a model’s failure rate exceeds the baseline for consecutive statistical cycles.
9. Security and Data Boundaries
Multi-model integration expands the scope of data flow, requiring corresponding enhancements to security design.
- Minimal Transmission: Only send the content necessary to complete the task to the model, removing irrelevant personal information and sensitive fields.
- Data Tier Labeling: Data processing terms may vary across different models.
