Multi-Model AI Agent Architecture and Cost Metering for Office Products
This article explores how to design multi-model AI Agent products for office scenarios and keep their costs under control. It breaks down a four-layer architecture (adapter, router, orchestrator, and metering/governance), explains three routing strategies that can be combined, and shows how a "base tokens × model multiplier × time-of-day multiplier" quota model makes costs explainable and predictable. It also covers configurable time-based discounts, idempotent asynchronous metering, full-link observability, and quota/circuit-breaking practices, with code examples throughout. It is aimed at developers who want to build multi-model Agents that can onboard new models quickly while keeping costs continuously controllable.
1. Why Office AI Agents Need a Multi-Model Architecture
In enterprise office scenarios, the tasks an AI Agent handles are highly diverse: document summarization favors long-context capability, spreadsheet and data organization favor structured reasoning, code generation favors instruction following, and everyday Q&A prioritizes response speed and cost. A single model can rarely balance quality, latency, and cost across all task types at once. That is why more and more office products introduce multi-model access into their architecture, letting users pick different models on demand within the same product.
Multi-model is not simply stitching together several APIs. It requires a complete layered architecture, a routing strategy, and a metering system. This article covers these three layers to help developers understand how to elegantly implement a “multi-model, cost-controlled” office Agent.
2. Layered Architecture for Multi-Model Access
A mature multi-model office Agent can usually be split into four layers:
- Adapter layer: Hides differences in interface protocols, parameter naming, and response structures across models, and exposes a unified invocation contract.
- Router layer: Decides which model a request goes to based on task type, user selection, and latency/cost constraints.
- Orchestrator layer: Handles multi-turn conversations, tool calls, context assembly, and result aggregation.
- Metering & Governance layer: Responsible for quota conversion, rate limiting, auditing, and monitoring.
The core goal of the adapter layer is to “program against interfaces,” decoupling upper-layer logic from specific models. Example code:
from abc import ABC, abstractmethodclass ModelAdapter(ABC):"""Unified model adapter contract that hides low-level differences"""@abstractmethoddef chat(self, messages: list[dict], **options) -> dict:...@abstractmethoddef estimate_tokens(self, text: str) -> int:...# Each model implements its own adapter; upper layers stay unaware of the detailsclass LiteAdapter(ModelAdapter):multiplier = 1.0 # base multiplierdef chat(self, messages, **options):...def estimate_tokens(self, text):...class FlashAdapter(ModelAdapter):multiplier = 0.8 # lower consumption multiplierdef chat(self, messages, **options):...def estimate_tokens(self, text):...
With a unified contract, the router only needs to hold a “model name → adapter” registry to switch dynamically. Adding a new model requires no changes to the upper-layer orchestration logic.
3. Intelligent Routing: From Manual Selection to On-Demand Optimization
Routing strategies generally fall into three categories that can be combined:
- Explicit routing: The user actively selects a model in the UI, and the product side only handles display and scheduling.
- Rule-based routing: Selects a model by task type or request length. For example, long document summarization goes to a long-context model, while short Q&A goes to a low-latency, low-cost model.
- Adaptive routing: Dynamically picks the best option based on real-time load, latency, and cost constraints, with automatic failover (fallback) on failure.
def select_model(task: str, user_choice: str | None = None,registry: dict | None = None) -> str:if user_choice in (registry or {}):return user_choice # explicit choice takes priorityif task == "long_summary":return "long_context_model" # long text prefers long-context capabilityif task == "code":return "instruction_model" # code tasks favor instruction followingreturn "fast_model" # default to a low-cost fast model
In real deployments, the router should also work with health checks: when a model is unavailable or hits a rate limit, it automatically switches to a backup model to keep the Agent running continuously.
4. Quota Consumption Multipliers: Making Cost Explainable and Predictable
One of the key designs in multi-model products is a unified quota metering standard. Because different models vary in compute cost and response capability, products typically convert actual quota consumption as “base tokens × model multiplier.” A lower multiplier means less consumption for the same task, and users can directly compare costs on the model selection page.
Example conversion logic:
def calc_quota(base_tokens: int,model_multiplier: float,time_multiplier: float = 1.0) -> int:"""base_tokens : base token count after input+output conversionmodel_multiplier : model multiplier; lower saves more quotatime_multiplier : time-of-day multiplier; off-peak < 1.0"""return int(base_tokens * model_multiplier * time_multiplier)
Splitting the “model multiplier” and the “time-of-day multiplier” into two independent factors keeps the logic clear and extensible: adding a model only requires registering its multiplier, and adjusting the discount strategy only requires changing the time-of-day multiplier. The two are decoupled.
5. Time-Based Discounts and Scheduling: Trading Scheduling for Cost
Office scenarios show clear time patterns, with load distributions differing significantly between working hours and holidays. Based on this, products can introduce time-based pricing and discount scheduling: lowering the multiplier of participating models during specific periods so that users consume less quota for the same task.
Key engineering points:
- Configurable periods: Turn discount periods, applicable models, and time-of-day multipliers into configurable rules rather than hardcoded values, making them easy for operations to adjust.
- Near real-time activation: Push rules through a config center and refresh the cache layer asynchronously, avoiding reading config on every call.
- Idempotent metering: Metering for the same request must be idempotent to avoid duplicate charges from retries. A unique request ID can serve as the idempotency key.
def resolve_time_multiplier(now, promo_rules):for rule in promo_rules:if rule.model in enabled and rule.is_active(now):return rule.time_multiplier # hit a discount periodreturn 1.0 # regular period
6. Engineering Practices for Metering and Governance
In a multi-model architecture, the metering and governance layer is often the most underestimated yet most critical part. Recommended best practices:
- Asynchronous metering: The main call chain only registers the charge; actual quota settlement is persisted asynchronously via a message queue, reducing latency and smoothing peaks and troughs.
- Real-time deduction + reconciliation fallback: Pre-deduct before the call, settle based on actual usage after the call, and correct deviations through periodic reconciliation tasks.
- Full-link observability: Instrument request logs, model multipliers, time-of-day multipliers, and consumed quota together, connecting them to logging services and monitoring alerts to form a traceable cost view.
- Quotas and circuit breaking: Set quota limits and rate limits for users/tenants, and alert or trip a circuit breaker when thresholds are hit to prevent abnormal consumption.
- Data and cache layering: Session context, metering records, and similar data can be persisted and accelerated with object storage and cache components, ensuring stability under high concurrency.
Example metering flow:
def handle_request(task, messages, user, registry, promo_rules):model = select_model(task, registry=registry)adapter = registry[model]tokens = adapter.estimate_tokens(str(messages))m = adapter.multipliert = resolve_time_multiplier(now(), promo_rules)if not quota.enough(user, tokens): # quota pre-checkraise QuotaExceeded()result = adapter.chat(messages) # call the modelcost = calc_quota(tokens, m, t)quota.deduct(user, cost, request_id=uid) # idempotent deductionemit_metrics(model=model, tokens=tokens, cost=cost)return result
7. Conclusion
Office AI Agents moving toward multi-model is essentially about dynamically balancing quality, latency, and cost. A robust architecture requires: a unified adapter layer to hide model differences, intelligent routing for on-demand optimization and failover, an explainable quota conversion of “base tokens × model multiplier × time-of-day multiplier,” and asynchronous metering, observability, and quota/circuit-breaking to keep production stable and costs controlled.
For developers, decoupling model differences, routing strategies, and metering rules is the key to letting a multi-model Agent both onboard new models quickly and stay continuously cost-controlled. Combined with the general cloud capabilities offered by Baidu AI Cloud (such as a container platform to host services, a message queue to support asynchronous metering, and a logging and monitoring/alerting system to ensure observability), teams can further reduce the cost of building in-house and focus more on business logic and user experience itself.
