Multi-Model Routing and Cost Scheduling for AI Agents: Engineering Practices for Office Workloads

    This article explains why office-focused AI agents need multiple models running side by side, and how to build the architecture that makes that manageable. It covers a unified model abstraction layer (model registry), task profiling as the input to routing decisions, a three-stage routing strategy from simple rules to cost-sensitive selection and cascading fallbacks, quota accounting built on configurable cost multipliers, off-peak scheduling for deferrable jobs, and the observability and evaluation metrics needed to keep routing decisions measurable. It also addresses transparency, compliance, version drift, and supplier concentration risks, and closes with a six-step implementation roadmap. You will get concrete guidance, sample code, and a metric set you can apply directly to a production multi-model system.

    1. Why Office Agents Need Multiple Models

    Office agents differ fundamentally from chat-style question answering. They take a goal as input and produce a deliverable as output, and in between they often run document parsing, spreadsheet calculations, information retrieval, code execution, and formatting. This leads to three direct consequences.

    Tasks are highly heterogeneous. “Compress three pages of meeting notes into five bullet points” is a short-context summarization job. “Compare the clauses in a contract against a tender document and produce a risk table” is a long-context, multi-turn, tool-heavy task that demands strong reasoning. Running every task on the same model tier either makes simple tasks pay an unnecessary cost or makes complex tasks fail and retry because the model is not capable enough.

    User goals are layered. Behind the same set of features sit three typical priorities: quality first (accept spending more quota to succeed on the first attempt), latency first (interactive operations cannot keep users waiting), and cost first (batch background jobs must minimize unit cost). A product that only exposes “which model do you want” is really handing the hardest decision to the user.

    Capability supply keeps changing. Model availability, context length, tool-calling support, stability, and pricing all shift over time, and performance varies by task. If model names are hardcoded into business flows, every supply change means a code change and a new release.

    The conclusion is clear: you need a model-agnostic architecture that treats models as pluggable compute resources, with a scheduling layer making the tradeoff between quality and cost.

    2. A Unified Model Abstraction Layer: Model First, Scheduling Second

    Every routing strategy rests on comparable model descriptions. Introduce an internal model registry, describe each candidate model with a uniform spec structure, and let business code depend only on that abstraction rather than on any specific model.

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class ModelSpec:
    4. alias: str # internal alias, decoupled from the public display name
    5. tier: str # light / balanced / strong capability tier
    6. ctx_tokens: int # usable context length
    7. supports_tools: bool # function / tool calling support
    8. supports_vision: bool # image + text input support
    9. streaming: bool # streaming output support
    10. p50_ms: int # median response latency (refreshed by ongoing load tests)
    11. cost_multiplier: float # quota consumption multiplier, distributed by platform config
    12. available: bool = True # health flag, removed automatically when unhealthy

    This abstraction solves three problems. First, capabilities become comparable: routing decisions use structured fields instead of guesswork. Second, changes become configurable: onboarding a new model means registering one spec, and multipliers, tiers, and switches can all be hot-updated without touching business code. Third, presentation is decoupled: the model list and multipliers shown in the UI are a projection of the registry, so you avoid maintaining one model name in code and a different name in the interface.

    One point deserves emphasis: cost_multiplier should be a platform-wide, centrally configured coefficient that can be lowered at any time, not a constant baked into calling code. Implementing “the same task consumes less” as a multiplier reduction rather than a model swap keeps promotional strategies decoupled from the architecture.

    3. Task Profiling: The Input to Routing Decisions

    Routing quality depends on the quality of its inputs. Before execution, tag the task with a set of structured labels (a task profile). This is a low-cost, high-return step:

    • Estimated size: input length, number of attachments, number of documents to retrieve;
    • Complexity: whether multi-step planning is needed, whether tools are involved, whether strict structured output is required (for example, a table with fixed column widths);
    • Interaction constraints: whether the user is waiting, whether there is an explicit latency expectation;
    • Risk level: whether data leaves the organization, whether human review is required;
    • Business policy: quality first / cost first / latency first, configurable per session.

    Two profile fields are hard constraints: “does this need tool calling” and “does this require structured output.” They must be handled during candidate filtering, not left to the scoring stage. Otherwise you get the classic failure mode: a model without tool support gets selected, and you only find out halfway through the run that the task cannot execute.

    4. Routing Strategy: From Rules to Cascades

    Routing can start simple and evolve; there is no need to reach complexity in one step.

    Layer 1: Rule-based routing. Map difficulty tiers directly to capability tiers. Simple tasks go to lightweight models, complex tasks go to strong models. Cheap to implement, predictable behavior, and a good baseline.

    Layer 2: Cost-sensitive routing. Within the constraints and above the quality floor, pick the model with the lowest multiplier and acceptable latency.

    1. def select_model(task: TaskProfile, registry: dict[str, ModelSpec]) -> ModelSpec:
    2. # 1) hard-constraint filtering: context, tools, multimodality, availability
    3. candidates = [m for m in registry.values()
    4. if m.available
    5. and m.ctx_tokens >= task.est_tokens
    6. and (m.supports_tools or not task.needs_tools)
    7. and (m.supports_vision or not task.needs_vision)]
    8. if not candidates:
    9. raise NoViableModel(task.id)
    10. # 2) policy-based scoring
    11. if task.policy == "quality":
    12. key = lambda m: (-QUALITY_SCORE[m.tier], m.cost_multiplier, m.p50_ms)
    13. elif task.policy == "latency":
    14. key = lambda m: (m.p50_ms, m.cost_multiplier)
    15. else: # default: above the quality floor, take the lowest multiplier
    16. floor = QUALITY_FLOOR[task.difficulty]
    17. key = lambda m: (QUALITY_SCORE[m.tier] < floor, m.cost_multiplier, m.p50_ms)
    18. return min(candidates, key=key)

    Layer 3: Cascades and fallbacks. Try a low-cost model first; if the result passes validation, return it. If it fails or the quality is not good enough, escalate to a stronger model. This is the most effective way to lower cost and protect success rate at the same time.

    1. async def run_with_cascade(task, plan, judge):
    2. trace = []
    3. for spec in plan: # light -> balanced -> strong
    4. try:
    5. out = await invoke(spec, task)
    6. except TransientError as e: # rate limiting, timeouts, other retryable errors
    7. trace.append(("retry", spec.id, type(e).__name__))
    8. continue
    9. verdict = await judge(out, task) # format check, field completeness, assertion rules
    10. if verdict.ok:
    11. return out, spec, trace
    12. trace.append(("escalate", spec.id, verdict.reason))
    13. return await manual_fallback(task), None, trace # human fallback

    The key to cascading is the judge and the retry policy. Validation should rely on automatically decidable signals as much as possible — whether the JSON parses, whether required fields are present, whether numbers can be recomputed, whether citations actually exist in the source document — rather than fuzzy scoring from another large model. Retries must distinguish retryable errors from business errors, so that logic errors are not treated as transient failures that keep burning quota. Also set a total time budget for the whole cascade, so layered degradation does not wreck the interactive experience.

    5. Cost Accounting: Turning Multipliers into Explainable Budgets

    Users need to perceive cost for an agent to earn lasting trust. Keep the accounting model simple and explainable:

    1. def estimate_quota(spec: ModelSpec, in_tokens: int, out_tokens: int) -> float:
    2. base = in_tokens + OUT_WEIGHT * out_tokens # output weight is configurable, usually > 1
    3. return base * spec.cost_multiplier # a lower multiplier means less consumption per task

    Four supporting pieces are needed:

    1. Estimation and display: give a range estimate before execution and the actual consumption afterwards, so multiplier differences become tangible on real tasks;
    2. Quotas and circuit breakers: set three levels of budget — account, session, and task. When a threshold is reached, degrade to a lower-multiplier model or move the task into a queue instead of failing outright;
    3. Attribution analysis: track consumption and success rates by model, by task type, and by routing path to spot combinations with a high multiplier and a low success rate, and adjust tiers accordingly;
    4. A lightweight free allowance: giving free quota to small, lightweight tasks can meaningfully improve new-user conversion, but it needs clear allowance boundaries and abuse protection, or scripted calls will drain it.

    6. Off-Peak Scheduling: Moving Deferrable Tasks to Cheaper Time Windows

    Time-limited pricing is essentially a cost structure along the time dimension. For discounts to actually translate into a lower unit cost, the scheduler has to actively identify deferrable tasks.

    Deferrable tasks are background batch jobs, long-document analysis, bulk generation, and non-interactive reports. Non-deferrable tasks are the immediate Q&A and editing operations where a user is waiting.

    1. OFF_PEAK = {"workday": ["09:00-11:00", "14:00-16:00"],
    2. "holiday": ["09:00-12:00", "14:00-18:00"]}
    3. def dispatch(task, now):
    4. if task.deferrable and not in_window(now, OFF_PEAK):
    5. return enqueue(task, priority=task.priority, deadline=task.deadline)
    6. return run_now(task)

    Implementation notes: tasks must be idempotent so that repeated delivery has no side effects; the queue must support priorities and deadlines, otherwise a large batch of low-cost jobs will fill the cheap windows and starve interactive tasks; and for tasks that pass their deadline there must be an explicit degradation promise (for example, run in the standard window and notify the user) rather than a silent delay.

    7. Evaluation and Observability: Making Routing Decisions Measurable

    The biggest risk in a multi-model system is invisibility. Without observability, routing problems surface as “the answers are sometimes not great” and are nearly impossible to localize. The minimum viable metric set:

    Dimension Metric Purpose
    Quality task success rate, first-attempt success rate, manual rework rate detect whether tiers are set too low
    Cost average consumption per task, multiplier-weighted cost, escalation rate judge whether cascading is effective
    Latency p50 / p95 end-to-end latency, queue time detect impact on interaction
    Routing share of each path, validation pass rate, fallback rate detect strategy drift

    Evaluation should have three parts: an offline regression set (covering typical office tasks, run before every tier or multiplier change), shadow traffic (the new strategy computes but does not take effect, so you can compare the difference), and online A/B testing (evaluated on both success rate and unit cost, so that looking at cost alone does not collapse quality).

    Observability can be carried by general-purpose cloud services: a logging service for call traces, monitoring and alerting with thresholds on success rate and p95 latency, object storage for archiving evaluation samples, and a message queue for asynchronous tasks and off-peak scheduling. Such capabilities are offered as general categories on platforms such as Baidu AI Cloud, so the architecture does not need to bind to any particular implementation.

    8. Transparency and Governance

    Give users a choice, not a burden. Automatic routing is the default. Advanced users can see the model list with multiplier labels and set their own preference (quality first or cost first). What matters is that “what was chosen, why, and what it cost” are all visible.

    Risk and compliance need to be designed in up front. Office documents often contain sensitive information, so it must be clear whether data may be used for training, whether private or dedicated deployment is supported, and whether cross-region calls are constrained. These policies should participate in filtering as part of the model spec, not be patched in afterwards.

    Version drift and supplier concentration. The version behind the same model alias may be updated, which can cause sudden behavior changes, so evaluation sets must be fixed and regression testing must be routine. Also keep at least two suppliers with comparable capabilities available, so that a single model failure does not take the whole system down.

    9. Implementation Roadmap

    1. Abstraction first: build the model registry and a unified invocation adapter, removing the dependency of business code on specific models;
    2. Baseline routing: implement rule-based routing and multiplier-based accounting, and get cost data into the light;
    3. Cascades and validation: introduce automated judges and fallback chains to reduce average cost;
    4. Off-peak scheduling: connect a priority queue with idempotent tasks to deliver time-window-based cost savings;
    5. Data-driven tuning: continuously adjust tiers, multipliers, and thresholds using offline regression + shadow traffic + online A/B;
    6. Governance loop: complete compliance policies, quota circuit breakers, version regression testing, and multi-supplier redundancy.

    In one sentence: the value of connecting multiple models is not how many you connect, but whether you have built a decision system that is abstractable, measurable, schedulable, and recoverable — one that puts the right task on the right capability tier and makes every unit of quota explainable.