Multi-Model Architecture: Routing, Metering, and Compliance Practices

    As AI-powered office agents evolve from chat boxes into task-executing workbenches, no single model delivers the best cost-performance across every task. This article breaks down a practical architecture for adopting multiple models, organized into four implementable layers: a unified inference gateway that hides provider differences, cost-aware routing that moves from static rules to scored scheduling, transparent metering built on consumption multipliers, and a quality-assurance loop. It also covers the compliance and security requirements of onboarding multiple models and provides a landing checklist so you can verify that routing, metering, quality, and compliance are all in place. Readers will learn how to make a multi-model deployment a controllable cost-performance advantage and source of supply resilience rather than a cost burden.

    Why Office Agents Are Moving Toward Multiple Models

    Over the past year, AI-agent office products have rapidly evolved from a “chat box” into a “workbench that can execute tasks”: reading documents, filling in spreadsheets, generating meeting notes, and calling enterprise APIs to complete approvals and retrieval. As task complexity grows, a practical problem emerges — no single model offers the best cost-performance across all tasks.

    Concretely, running multiple models together is usually driven by three factors:

    1. Complementary capabilities. Lightweight models respond fast and cost little on high-frequency short tasks such as summarization, rewriting, and format conversion; reasoning models perform better on multi-step planning, code generation, and complex spreadsheet analysis; ultra-long-context models suit document-scale understanding and cross-file retrieval.
    2. Cost tiering. Office scenarios generate enormous call volumes. If every request went to the strongest model, per-task cost would spiral out of control. Pushing simple tasks down to low-cost models is the most effective lever for cost governance.
    3. Supply resilience. Rate limiting, jitter, and version retirement from a single provider can all cause outages. Multiple models provide natural disaster-recovery redundancy.

    A recent industry trend — an office agent adding several third-party models and labeling a quota-consumption multiplier so users can choose for themselves — is essentially a product-level response to these three drivers. The following sections break this down into four implementable layers from an architectural standpoint.

    Layer 1: The Access Layer — Hiding Heterogeneous Differences Behind a Unified Inference Gateway

    The most common pitfall in multi-model integration is scattering if-else checks for each vendor’s request format throughout the business code. The correct approach is to converge all differences into a single adapter layer that exposes a unified contract to the upper layers.

    Abstracting a Unified Request/Response Model

    1. from dataclasses import dataclass
    2. from typing import Iterator, Literal
    3. @dataclass
    4. class ChatMessage:
    5. role: Literal["system", "user", "assistant", "tool"]
    6. content: str | list[dict] # supports text + image and other multimodal parts
    7. tool_call_id: str | None = None
    8. @dataclass
    9. class ChatRequest:
    10. model_id: str
    11. messages: list[ChatMessage]
    12. tools: list[dict] | None = None
    13. temperature: float = 0.2
    14. max_tokens: int = 4096
    15. response_format: Literal["text", "json_schema"] = "text"
    16. @dataclass
    17. class ChatChunk:
    18. delta_text: str
    19. finish_reason: str | None
    20. usage: dict | None # {"input_tokens": 0, "output_tokens": 0}

    The key design point is to replace vendor-proprietary parameters with a “capability declaration + optional parameters” model. Different vendors name their parameters for thinking budget, repetition penalty, and tool-call parallelism completely differently. Unifying them into optional fields and mapping them inside the adapter ensures the upper-layer business logic does not need rewriting when the provider changes.

    Capability Matrix and Negotiation

    Model capabilities are not equal. The gateway should maintain a versionable capability matrix and perform capability negotiation when a request enters: if a user selects structured output but the target model does not support it, the gateway either automatically degrades to “prompt constraints + post-processing validation” or returns an explainable rejection reason.

    1. models:
    2. - id: light-v1
    3. cost_multiplier: 1.0
    4. capabilities: [chat, tool_call, structured_output]
    5. max_context: 128000
    6. streaming: true
    7. - id: reason-v2
    8. cost_multiplier: 3.5
    9. capabilities: [chat, tool_call, structured_output, long_context]
    10. max_context: 1000000
    11. streaming: true
    12. - id: vision-v1
    13. cost_multiplier: 2.0
    14. capabilities: [chat, vision, structured_output]
    15. max_context: 32000

    Normalizing Streaming and Tool Calls

    The office-agent experience depends heavily on streaming output and tool calls. The adapter needs to handle three real-world differences:

    • Inconsistent incremental semantics: some implementations return pure incremental text while others return accumulated text; unify to incremental and add deduplication protection.
    • Tool-call chunking: function names and argument JSON may arrive across multiple chunks, so the adapter must maintain a state machine to concatenate them.
    • Layered error semantics: distinguish retryable errors (rate limiting, timeouts, 5xx), errors that require switching models (unsupported capability), and non-retryable errors (content-moderation blocks, invalid parameters).
    1. def stream_chat(req: ChatRequest) -> Iterator[ChatChunk]:
    2. adapter = REGISTRY.get(req.model_id)
    3. last_err = None
    4. for _ in range(2): # at most one retry
    5. try:
    6. yield from adapter.chat(req)
    7. return
    8. except RetryableError as e:
    9. last_err = e
    10. raise UpstreamUnavailable(req.model_id) from last_err

    Layer 2: The Routing Layer — From Static Configuration to Cost-Aware Scheduling

    With unified access in place, the next step is deciding which model each request should go to. A mature router is usually a hybrid of “rules as the baseline + model scoring as the fallback”.

    Three-Tier Routing Decision

    1. Pinned routing: the user explicitly selects a model, or the task contains compliance-sensitive steps (such as generating external contract terms), bypassing automatic scheduling.
    2. Rule-based routing: match rules by task type, input length, whether it contains images, and whether tool calls are needed. For example, when the input token count exceeds 80% of a lightweight model’s context window, go straight to a long-context model to avoid truncation and information loss.
    3. Scored routing: for requests not covered by rules, use a small classifier or heuristic to predict “the expected quality of that model on this task”, then rank by real-time latency and quota balance.
    1. def route(task: Task, ctx: Context) -> str:
    2. if task.pinned_model:
    3. return task.pinned_model
    4. rule = RULES.match(task) # length/modality/tool needs
    5. if rule:
    6. return rule.model_id
    7. candidates = [m for m in MODELS if m.satisfies(task.required_caps)]
    8. scored = sorted(
    9. candidates,
    10. key=lambda m: (
    11. -ctx.quality_score(m.id, task.kind), # quality first
    12. m.cost_multiplier, # then cost on ties
    13. -m.available_quota(ctx.tenant), # then quota headroom
    14. ),
    15. )
    16. return scored[0].id

    Fallback and Degradation

    A production environment must have a degradation chain: when the primary model is unavailable, switch in A → B → C order, and guarantee that the tool-call protocol and structured-output schema remain consistent after degradation, otherwise downstream parsing will fail in a cascade. Degradation events should be reported as independent telemetry points, because frequent degradation often means insufficient quota or upstream capacity problems.

    Caching and Batching

    Office scenarios have high repetition rates — many requests share the same template or prompt skeleton. Adding semantic-level prompt caching and an offline batch channel (for example, generating daily reports overnight) at the gateway can significantly lower the average cost multiplier without sacrificing quality.

    Layer 3: The Metering Layer — How the Consumption Multiplier Should Actually Be Calculated

    The “multiplier” is the most critical and the easiest to get wrong part of a multi-model product. Users have extremely low tolerance for opaque billing; once they feel “I can’t understand what my money is paying for”, trust erodes.

    A Unified Metering Basis

    Different models differ in how they tokenize, whether system prompts and tool definitions are billed, and whether cache-hit input is discounted. The gateway should converge everything to:

    1. base usage = normalized input tokens × input weight + normalized output tokens × output weight
    2. quota cost = base usage × model multiplier × business-scenario coefficient

    Here the “model multiplier” reflects the procurement-cost difference per token for that model, while the optional “business-scenario coefficient” reflects the resource overhead of different capabilities (such as visual understanding or long-document parsing). All coefficients should be versioned and written into a configuration center, with historical bills retaining a snapshot of the coefficients at the time, to avoid making later price changes hard to trace.

    1. def compute_charge(usage: Usage, model: ModelSpec) -> Charge:
    2. base = usage.input_tokens * model.input_weight + usage.output_tokens * model.output_weight
    3. cost = base * model.cost_multiplier * SCENE_COEF[usage.scene]
    4. return Charge(base_units=base, multiplier=model.cost_multiplier,
    5. quota_cost=round(cost, 4), model_id=model.id,
    6. spec_version=model.spec_version)

    Pre-Estimation + Mid-Flight Limits + Post-Hoc Billing

    • Pre-estimation: before a task starts, give an estimated consumption range (based on percentiles of similar historical tasks) so users “know how much it will cost before they click”.
    • Mid-flight limits: set budget caps for each tenant/user/task; when approaching the threshold, downgrade to a lightweight model or pause with a prompt.
    • Post-hoc billing: provide usage details broken down by model, by task, and by time; archive them to object storage for long-term retention, and use log services and a monitoring/alerting system to detect abnormal consumption.

    Baidu AI Cloud’s large-model service platform and general cloud product categories (container platforms hosting inference services, object storage for bill archiving, message queues for asynchronous decoupling, and monitoring/alerting for quota-level warnings) can be combined into a complete metering-governance pipeline; the engineering path to build it is mature and reusable.

    Transparent Product Expression

    Calculating accurately is only half the job; the other half is explaining the multiplier clearly. Suggested practices: show “relative multiplier + estimated consumption” directly in the model selector; show the actual consumption for this session and the cumulative balance on the results page; and provide an “auto mode” in which the system automatically picks the lowest-cost model within a quality threshold. This returns the choice to the user and makes the cost-responsibility boundary clear.

    Layer 4: Quality Assurance — Multiple Models Require a Unified Evaluation Loop

    After introducing multiple models, quality is no longer guaranteed by a single vendor, so evaluation must be internalized. Practical approaches include:

    • Building a business evaluation set: sample 500–2000 tasks from real de-identified logs, covering high-frequency office scenarios (summarization, table extraction, email drafting, multi-turn rewriting), with human-annotated golden answers.
    • Automated regression: every time a new model is onboarded or a vendor upgrades a version, run the full evaluation and output five metrics — accuracy, format-validity rate, tool-call success rate, average latency, and per-task cost.
    • Canary release: a new model first handles 1%–5% of traffic, comparing key online metrics (task completion rate, user retry rate, average quota consumption) before ramping up.
    • A/B and online feedback: feed explicit user feedback (regeneration, edit magnitude, copy/acceptance rate) back into the routing strategy as implicit quality signals.

    Compliance and Security: Filing, Content Labeling, and Embodied Models

    Multi-model integration is not just an engineering problem; it directly amplifies compliance complexity — every onboarded model must have legitimate qualifications to offer external services, and the product owner should retain the vendor’s filing/registration number and service agreement as audit evidence. Key points include:

    1. Generative AI service compliance: offering generative AI services to the public may require completing algorithm filing/registration and a security assessment, disclosing the applicable scope and limitations of the service, and having a clear legal basis for training-data sources and personal-information processing. (These obligations are jurisdiction-specific — they reflect China’s regulatory regime here — so equivalent requirements should be verified against the local law of each market you serve.)
    2. Content safety and labeling: filter both inputs and outputs, prominently label AI-generated content, and establish channels for intercepting, logging, tracing, and appealing violations.
    3. Data boundaries: clarify which data may be used for inference, whether it is retained, for how long, and whether it is used to improve models. When scheduling across models, prevent consent that a user explicitly disabled under model A from being carried by default into model B.
    4. Additional considerations for embodied models: embodied large models (VLA-class models that fuse visual perception, language understanding, and action output) have already entered the filing process in the industry. Their risk surface differs from traditional text models and requires additional assessment: the safety boundaries of physical-world actions, deviations during simulation-to-reality transfer, fail-safe behavior when sensor inputs are abnormal, and compliance authorization for the data-collection environment. If a product has only been validated in simulation, it should not imply that it is ready for real-world deployment.

    Landing Checklist

    • Has a unified inference gateway been extracted so the business layer has zero vendor coupling?
    • Is the capability matrix versioned, and does it support negotiated degradation?
    • Does the routing strategy cover all three tiers (pinned/rule/scored) and include a degradation chain?
    • Is the metering basis unified, and are the coefficients versioned with snapshots retained?
    • Are pre-estimation, mid-flight limits, post-hoc billing, and anomaly alerting all in place?
    • Is there a business evaluation set and a canary-release process?
    • Has every onboarded model completed qualification verification and filing-information archiving?
    • Are generated-content labeling, data-retention policy, and cross-model consent isolation all implemented?

    Conclusion: The competitiveness of multi-model integration lies not in “how many vendors you connect” but in whether routing is smart, metering is transparent, quality is measurable, and compliance forms a closed loop. Get these four things solid, and multiple models become a controllable cost-performance advantage and a source of supply resilience rather than a cost burden.