Multi-Model Integration Architecture and Cost Governance for AI Work Platforms

    Office AI workloads are becoming too heterogeneous for any single model to serve well, which turns model routing and cost governance into core platform infrastructure rather than a UI feature. This article walks through the engineering path: a unified capability framework that makes models comparable (capability tags, TTFT and P95 latency, normalized cost, reliability, data boundaries), three integration patterns (front-end switching, a unified LLM gateway, and orchestration-based routing), credit multipliers that reconcile inconsistent pricing schemes, and a cost governance layer built on budgets, quotas, configurable off-peak discounts, caching, and right-sizing. It closes with the enterprise governance items most teams overlook — data classification, credential management, auditability, access control, replaceability, and quality regression — plus a pre-launch checklist.

    1. Background: AI requirements in the office are fragmenting

    Early AI assistant products usually bound themselves to a single model. Office tasks were mostly Q&A, polishing, and summarization, and one strong general-purpose model covered the vast majority of them. Once large models moved into real production workflows, task structure became highly heterogeneous:

    Task type Core capability needs Latency sensitivity Cost sensitivity
    Code completion and debugging Instruction following, structured output, long context High (low-latency streaming required) Medium
    Long-form writing and rewriting Context length, style consistency Low (a few seconds of waiting is acceptable) Medium
    Spreadsheets and data analysis Reasoning depth, reliable tool calling Low (runs offline in batches) High
    Multi-turn conversation and knowledge Q&A Response speed, breadth of knowledge Very high High
    Contract clause review, complex reasoning Logical reasoning, low hallucination Low Low (quality first)

    This produces three practical pain points:

    • Capability mismatch. Handing a multi-step reasoning task to a lightweight, fast model causes rework, and the calls spent on that rework often cost more than using a strong model in the first place.
    • Tool fragmentation. Different capabilities live behind different product entry points, so users copy and paste context between them. Conversation state is lost, and cost is scattered across multiple invoices.
    • Invisible cost. Most products cannot produce a cost estimate before a call, so users “spend first, find out later” and have no way to enforce a budget.

    Seen this way, “letting users choose a model on the same work platform” is not a marketing feature. It is the combination of two pieces of infrastructure: model routing and cost governance. The rest of this article breaks down how to implement both from an engineering perspective.

    2. First, define “selectable”: a shared set of capability dimensions

    Letting users pick a model per task presupposes that the selection information itself is comparable. Common industry dimensions are:

    1. Task capability: tags such as code generation, long context, reasoning, multimodality, tool calling.
    2. Latency: time to first token (TTFT) and P95 total duration. Interactive scenarios care about TTFT; batch scenarios care about throughput.
    3. Cost: unit price per measure of usage, or normalized into a “credit consumption multiplier.”
    4. Reliability: availability, rate-limiting policy, success rate after retries.
    5. Data boundaries: whether specific sensitive data may leave your environment, and whether outputs may be retained by a third party.

    A practical approach is to maintain an internal model profile table, turning “is this model selectable” into “automatically determined from the profile” instead of asking users to read marketing copy:

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class ModelProfile:
    4. name: str
    5. tier: int # 1 = strong reasoning, 2 = balanced, 3 = lightweight/fast
    6. unit_credit: float # normalized cost (credits per call, illustrative)
    7. p95_latency_ms: int
    8. supports: tuple # ("code", "reason", "long_ctx", "vision")
    9. data_scope: str # ("public", "internal")
    10. PROFILES = [
    11. ModelProfile("reasoner_pro", 1, 1.0, 9000, ("code", "reason", "long_ctx"), "internal"),
    12. ModelProfile("balanced_mid", 2, 0.5, 3000, ("code", "doc", "long_ctx"), "internal"),
    13. ModelProfile("lightning_flash", 3, 0.15, 800, ("doc", "sheet", "chat"), "internal"),
    14. ]

    3. Three architecture patterns for multi-model integration

    Pattern A: Front-end model switching

    The product layer exposes multiple models as options and lets the user specify a model name manually.

    • Pros: lowest implementation cost, and the mental model is obvious to users.
    • Cons: switching models switches sessions and breaks context; cost stays opaque; there is no quality guarantee, so a model outage requires manual fallback.

    Pattern B: Unified model gateway (the mainstream approach)

    The platform deploys a model gateway (LLM gateway). Every upstream model is registered as an adapter, and a single request/response protocol is exposed to callers.

    • Pros: integrate once, route everywhere; credentials centrally managed; rate limiting, retries, auditing, and billing are intercepted at the gateway with zero changes to business code.
    • Cons: the gateway sits on the critical path, so you own capacity planning and failure isolation.

    The minimum capability set for a gateway:

    • Protocol normalization: unify message structures, streaming events, tool-calling formats, and error codes.
    • Credential hosting: store API keys in a secrets management service and inject them at the gateway, so they never reach clients or logs.
    • Failure isolation: set per-model timeouts, concurrency limits, and circuit-breaker thresholds so one failing upstream cannot take down the whole entry point.
    • End-to-end observability: carry a request ID from gateway to upstream and back, with logs and metrics centralized in your logging and monitoring/alerting systems.
    1. def route(task: str, ctx: dict) -> ModelProfile:
    2. """Default routing policy: filter by capability, then rank by budget and policy."""
    3. cand = [m for m in PROFILES if task in m.supports]
    4. if not cand:
    5. raise UnsupportedTask(task)
    6. # 1) Data boundaries first: sensitive tasks may only use internal models
    7. if ctx.get("sensitive"):
    8. cand = [m for m in cand if m.data_scope == "internal"]
    9. # 2) Within budget, prefer the higher-quality tier (staying within acceptable cost)
    10. affordable = [m for m in cand if m.unit_credit <= ctx["budget"]]
    11. pool = affordable or cand
    12. return max(pool, key=lambda m: (m.tier, -m.p95_latency_ms))

    Add an orchestration layer above the gateway that decomposes a user task into sub-steps and selects a model per step:

    1. def rewrite_with_cascade(prompt: str) -> str:
    2. """Cascading routing: try a cheap model first, escalate only if quality is insufficient.
    3. Average cost drops while a quality floor is maintained."""
    4. for name, threshold in (("lightning_flash", 0.85),
    5. ("balanced_mid", 0.92),
    6. ("reasoner_pro", 0.0)):
    7. out = call_model(name, prompt)
    8. if evaluate(out) >= threshold: # evaluate: rule checks / small-model scoring
    9. return out
    10. return out

    These patterns are not mutually exclusive. Most mature platforms combine B + C: the gateway handles integration and governance, and the orchestration layer decides “which model for which task.”

    4. Credit multipliers: turning inconsistent pricing into comparable prices

    The biggest engineering obstacle to coexistence of multiple models is inconsistent pricing units. Some bill input and output tokens separately, some bill per call, some bill per seat, and some add cache-hit discounts. Comparing raw unit prices is misleading, so the common industry practice is to introduce a normalized cost unit (credits) and annotate every model with a credit consumption multiplier.

    The multiplier can be abstracted as:

    1. credits consumed = actual usage × model multiplier × scenario factor
    2. where actual usage is derived from a unified conversion rule, e.g.:
    3. interactive single-turn conversation ≈ 1 call
    4. batch job ≈ input_tokens/1000 + output_tokens/1000 × output_weight
    5. scenario factor ≈ off-peak discount / cache-hit discount
    1. BASE_CREDIT = {"reasoner_pro": 1.0, "balanced_mid": 0.5, "lightning_flash": 0.15}
    2. def estimate_credit(model: str, in_tok: int, out_tok: int, hit_rate: float = 0.0) -> float:
    3. """Pre-call cost estimate (illustrative implementation)."""
    4. mul = BASE_CREDIT[model]
    5. raw = (in_tok / 1000 + 2.0 * out_tok / 1000) # output side is usually more expensive
    6. discounted = raw * (1 - 0.5 * hit_rate) # prefix cache hits get a discount
    7. return round(mul * discounted, 4)

    Three engineering details matter here, or the multipliers will “drift” out of meaning:

    • Align numerator and denominator. Multipliers must rest on the same conversion rule, otherwise models are not comparable.
    • Separate input from output. If a multiplier is defined purely per call, long-context tasks will have their cost severely underestimated.
    • Keep pricing-rule changes rollbackable. Pricing rules should be delivered as configuration with retained historical versions, which makes auditing and user-facing explanations much easier.

    5. Cost governance: budgets, quotas, and time-window discounts

    5.1 Budgets and quotas

    Layer the credit pool as “organization → project → individual,” and place quota checks in the gateway’s pre-call interception path. Over-limit requests are rejected or degraded up front rather than reconciled after billing:

    1. def guard(principal: str, need_credit: float) -> None:
    2. used = quota_store.used(principal)
    3. if used + need_credit > quota_store.limit(principal):
    4. raise QuotaExceeded(principal, used, need_credit)

    5.2 Time-window discounts: configurable, not hard-coded

    A limited-time discount (specific hours on weekdays and holidays) is essentially a time-dimension pricing policy. The right approach is to externalize the window table as configuration and have the gateway compute the factor from the request timestamp:

    1. DISCOUNT_WINDOWS = [
    2. {"days": [1, 2, 3, 4, 5], "start": "10:00", "end": "11:00", "factor": 0.8},
    3. {"days": [6, 7], "start": "14:00", "end": "16:00", "factor": 0.8},
    4. ]
    5. def factor_at(ts) -> float:
    6. for w in DISCOUNT_WINDOWS:
    7. if ts.weekday() + 1 in w["days"] and w["start"] <= ts.strftime("%H:%M") < w["end"]:
    8. return w["factor"]
    9. return 1.0

    Combined with estimate_credit() above, the platform can answer the question users care about most in the UI: “how many credits does this task cost right now, and which upcoming window is cheaper.” The orchestration layer can also schedule deferrable batch jobs into discounted windows.

    5.3 Caching and deduplication

    High-frequency requests (fixed system prompts, FAQs, common templates) can reuse results through prefix caching plus semantic caching — the highest return-on-investment item in cost governance.

    5.4 Model right-sizing

    Handle most requests with lightweight models and escalate only complex inputs to strong models. This typically lowers average cost substantially. The key is that escalation criteria must be measurable (failed rule checks, low confidence, repeated retries still failing) rather than based on subjective judgment.

    6. Governance items enterprises must not skip

    Governance area Key actions
    Data classification Sensitive data must not leave your environment; run masking and classification-label validation before any egress
    Credential management Secrets hosting plus periodic rotation plus gateway-side injection; never write keys to disk or logs
    Audit and compliance Retain request metadata and masked content, with traceability by person and by project
    Access control Assign model allowlists by role, project, and budget dimension
    Replaceability Isolate upstreams behind an abstraction layer to avoid single-vendor lock-in; keep switchover drills
    Quality regression Run version regression against a fixed evaluation set, automatically comparing before and after model upgrades

    Quality regression is the one most often overlooked. When a model is downgraded to a newer version, releasing it broadly without running the evaluation set can lower cost while regressing quality — and the resulting rework pushes total cost back up. For every model change, gate on two metrics: evaluation-set pass rate and cost per task.

    7. Pre-launch checklist

    1. Is a unified model profile and capability matrix in place?
    2. Has the gateway completed protocol normalization, streaming pass-through, and error-code mapping?
    3. Are the multiplier conversion rules configurable, versioned, and rollbackable?
    4. Do budget checks take effect before the call, with a degradation path when limits are exceeded?
    5. Can users correctly perceive and understand time-window discounts and cache discounts?
    6. Are circuit breakers and timeouts isolated per model?
    7. Is the egress policy for sensitive tasks enforced by hard validation rather than recommendation?
    8. Is there an evaluation-set regression and canary release process for model switching?

    8. Conclusion

    “Model freedom” looks like adding a dropdown for users; underneath, it tests four pieces of infrastructure: a unified abstraction layer (erasing protocol differences), capability routing (selecting models by task), cost observability (multipliers and estimates), and budget and compliance governance (quotas, auditing, egress control). Once these capabilities are settled into the gateway and orchestration layers, a growing number of models stops being an operational burden and becomes a resource pool that is orchestrable, measurable, and replaceable.

    One caveat: the actual list of callable models, the multiplier values, and the discount windows vary by platform — check the official documentation for the latest details. The multipliers, thresholds, and code in this article are illustrative and must be recalibrated against real billing rules and business scenarios before implementation.

    Organizations already running AI applications on Baidu AI Cloud can adopt a similar gateway-style architecture: complete model abstraction and data-egress control inside their own VPC, expose only a unified model service interface externally, keep credentials under enterprise-side secrets management, and use general-purpose capabilities such as object storage, logging, and monitoring/alerting for auditing and operational visibility — achieving self-directed control over both model selection and cost governance.