Architecture Design and Quota Scheduling for Multi-Model Integration in AI Office Agents

    This article explores the architecture design and quota scheduling practices for integrating multiple LLMs into AI office agents. It covers building a unified model gateway to decouple business logic from underlying models, abstracting pricing into configurable consumption multipliers, implementing time-based dynamic scheduling strategies, and applying intelligent routing with fallback mechanisms. Readers will gain insights into optimizing inference costs, ensuring service availability, and building resilient, scalable multi-model systems.

    1. Background: Why Office AI Agents Need Multi-Model Integration

    In recent years, AI agent-based office products have become essential tools for enterprise collaboration, offering capabilities such as document drafting, meeting minutes generation, data analysis, and task orchestration. These products rely on Large Language Models (LLMs) at their core. However, a single model often struggles to meet all business requirements simultaneously:

    1. Divergent Capabilities: Different models excel in various areas like long-text comprehension, code generation, reasoning depth, and response speed. In office scenarios, the requirements for “writing a weekly report” versus “running a data script” are significantly different.
    2. Cost Sensitivity: Office products typically have a large user base and high call frequency, with varying task values. Routing all requests through a single high-capability model can cause inference costs to spiral out of control.
    3. Availability and Elasticity: Multi-model integration inherently provides redundancy. When a specific model service experiences rate limiting or anomalies, requests can be downgraded or switched, enhancing overall system stability.

    Therefore, “integrating multiple models + unified scheduling + differentiated billing” has become a standard technical approach in the industry. Recent updates to multi-model integration capabilities in office agent products—such as publicly displaying quota consumption multipliers for different models on the model selection page and introducing time-based consumption discounts—actually reflect a comprehensive and reusable engineering practice worth analyzing.

    2. Architecture Design of the Multi-Model Unified Access Layer

    2.1 Core Layering

    To support the coexistence of multiple models, a “model gateway” or “unified access layer” is typically introduced between the application and the underlying models. The standard layering is as follows:

    1. ┌─────────────────────────────────────────┐
    2. │ Application Layer (Agent Task Orchestration) │
    3. ├─────────────────────────────────────────┤
    4. │ Routing Layer (Model Selection / Fallback) │
    5. ├─────────────────────────────────────────┤
    6. │ Adapter Layer (Protocol Conversion / Mapping) │
    7. ├─────────────────────────────────────────┤
    8. │ Model Layer (Inference Services) │
    9. └─────────────────────────────────────────┘
    • Adapter Layer: Different models have varying API protocols, parameter names, and response formats. The adapter layer normalizes these into a unified internal calling contract, such as standardizing streaming outputs from different vendors into the same event structure.
    • Routing Layer: Determines which model will handle the current request based on user selection, task type, quota status, and time-based strategies.
    • Application Layer: The Agent’s planning and tool-calling logic remains agnostic to specific model differences, ensuring decoupling between upper-layer business logic and underlying models.

    2.2 Unified Calling Contract Example

    Below is a simplified example of the adapter layer code, demonstrating how to normalize calls to different models using an abstract base class:

    1. from abc import ABC, abstractmethod
    2. class ModelAdapter(ABC):
    3. model_id: str # Unique model identifier
    4. quota_multiplier: float # Quota consumption multiplier
    5. @abstractmethod
    6. def chat(self, messages: list, **params) -> dict:
    7. """Returns unified format: {"content": str, "usage": {"tokens": int}}"""
    8. ...
    9. class AdapterA(ModelAdapter):
    10. model_id = "vendor_a_general"
    11. quota_multiplier = 1.0
    12. def chat(self, messages, **params):
    13. raw = call_vendor_a_api(messages, **params)
    14. return normalize(raw) # Convert to unified structure
    15. class AdapterB(ModelAdapter):
    16. model_id = "vendor_b_flash"
    17. quota_multiplier = 0.5
    18. def chat(self, messages, **params):
    19. raw = call_vendor_b_api(messages, **params)
    20. return normalize(raw)
    21. REGISTRY = {a.model_id: a for a in [AdapterA(), AdapterB()]}

    The benefit of this design is that adding a new model only requires implementing and registering an adapter, leaving the upper-layer business code unchanged, which aligns with the open-closed principle.

    3. Quota Consumption Multiplier: Abstracting Pricing into Configurable Coefficients

    A key product detail is displaying different quota consumption multipliers on the model selection page, where a lower multiplier means less consumption. From an engineering perspective, this is an elegant cost abstraction.

    3.1 Multiplier Model Design

    The general billing logic for quota-based products can be abstracted as:

    1. Actual Consumed Quota = Base Metric Unit × Model Multiplier × Time-Slot Discount Coefficient

    Where:

    • Base Metric Unit: Typically measured by token count or call frequency, ensuring a unified comparable baseline across different models.
    • Model Multiplier: Reflects the capability tier and inference cost of the model itself. Lightweight models have a lower multiplier, while flagship models have a higher one.
    • Time-Slot Discount Coefficient: Supports flexible promotional strategies configured by operations, defaulting to 1.0.

    3.2 Deduction Implementation Example

    1. def deduct_quota(user, model_id, usage_tokens, ts):
    2. adapter = REGISTRY[model_id]
    3. base = usage_tokens / 1000 # Base metric: 1k tokens
    4. multiplier = adapter.quota_multiplier
    5. discount = get_time_slot_discount(ts, model_id) # Time-slot discount, default 1.0
    6. cost = base * multiplier * discount
    7. user.quota -= cost
    8. record_ledger(user, model_id, base, multiplier, discount, cost)

    3.3 Why Multiplier Transparency is Valuable to Users

    Presenting the multiplier transparently to users is essentially a “predictable cost” product design:

    1. User Autonomy: Users can choose low-multiplier lightweight models for simple tasks and high-multiplier flagship models for complex ones, giving them the choice rather than relying on a black-box scheduler.
    2. Enhanced Quota Awareness: Users can clearly estimate how many tasks their remaining quota can complete, reducing the negative experience of “quota suddenly running out.”
    3. Promotional Leverage: The multiplier is a configurable parameter. Limited-time discounts only require lowering the multiplier for specific models without altering the main billing pipeline.

    4. Engineering Implementation of Time-Based Promotional Scheduling

    The concept of “enjoying quota discounts during specific promotional periods on workdays and holidays” corresponds to a typical dynamic strategy system based on time windows. Key implementation points include:

    4.1 Strategy Configuration Structure

    1. {
    2. "activity_id": "promo_2025_q4",
    3. "models": ["light_model_x", "flash_model_y"],
    4. "new_multiplier": 0.6,
    5. "time_slots": [
    6. {"days": "workday", "start": "20:00", "end": "23:00"},
    7. {"days": "holiday", "start": "10:00", "end": "22:00"}
    8. ],
    9. "effective_range": ["2025-11-15", "2025-12-31"]
    10. }

    4.2 Implementation Considerations

    1. Strong Consistency between Discount Calculation and Deduction: The deduction transaction must be based on the strategy snapshot effective at the exact moment of deduction to avoid billing disputes during strategy transitions. A common practice is to write the strategy version number into the billing ledger for auditing and reconciliation.
    2. Preheating and Caching: Call peaks often occur right before and after promotional periods start. The routing layer and strategy module should support hot configuration updates and reduce query pressure via local caching combined with version verification.
    3. Anti-Fraud and Rate Limiting: Limited-time discounts easily attract abnormal traffic. This requires user-level frequency control and circuit breaking for total campaign budgets to prevent scripts from exploiting the promotions.
    4. Holiday Calendar Management: Holiday determination relies on calendar data sources, which should be managed as independent configurations and maintained early each year to avoid inconsistencies between activity configurations and actual calendars.

    5. Model Selection and Fallback Practices in a Multi-Model System

    5.1 User Selection vs. System Selection

    Industry practices generally fall into two categories:

    • Explicit User Selection: Providing a model switch entry in the UI, suitable for advanced users sensitive to result quality or cost.
    • Intelligent System Routing: Automatically allocating models based on task characteristics (input length, task type, historical success rate), suitable for general users.

    Mature products often combine both: defaulting to automatic routing while allowing manual overrides.

    5.2 Fallback and Fault Tolerance

    Another core value of multi-model integration is fault tolerance. A simplified fallback strategy example:

    1. def chat_with_fallback(model_id, messages):
    2. # Prioritize similar capabilities
    3. for candidate in [model_id] + FALLBACK_CHAIN.get(model_id, []):
    4. try:
    5. return REGISTRY[candidate].chat(messages)
    6. except (RateLimitError, TimeoutError):
    7. continue
    8. raise AllModelsUnavailable()

    The fallback chain should follow the principle of “prioritizing similar capabilities” and transparently inform the user of the actual model used in the response to ensure a clear user experience.

    6. Conclusion

    Multi-model integration, multiplier-based quota systems, and time-based promotional scheduling constitute the three main technical pillars for office AI agent products in their commercialization phase:

    1. Unified Access Layer resolves protocol and capability differences among heterogeneous models, decoupling business logic from the models.
    2. Multiplier Model abstracts complex pricing into configurable coefficients, balancing user transparency with operational flexibility.
    3. Time-Based Scheduling Strategies turn cost discounts into programmable operational tools, requiring supporting measures for consistency, anti-fraud, and capacity planning.

    For teams building agent-based products, it is recommended to introduce adapter layer abstractions and metering middleware early on to avoid being locked into a single model later. Meanwhile, making multipliers, discounts, and other strategies configuration-driven leaves ample room for business growth within the technical architecture.