Multi-Model Aggregation: Model Selection and Routing for Office AI Agents

    This article explains how office AI agents can use multiple models to balance quality, cost, latency, and availability. It covers routing criteria, a layered architecture, fallback handling, and a Python example for filtering and selecting candidate models.

    Background: Moving from a Single Model to Multiple Models

    Early office AI agents were typically built around a single model. A product team selected one model, tuned its prompts, and connected it to features such as documents, spreadsheets, calendars, and email. This approach is simple to develop, but it creates three major problems.

    First, different tasks require different capabilities. Summarizing meeting notes and validating complex financial data should not consume the same level of resources. Second, models differ in response speed and pricing. Using one model for every request forces simple tasks to bear unnecessary latency and cost. Third, if a single provider experiences an outage or changes its policies, the entire feature set may be affected.

    Recently, more providers of office AI agents have introduced support for multiple models. Users can select a model based on the task, while the system can also assign models automatically in the background. This shift shows that model selection is moving from an internal product configuration to an explicit capability for users and developers.

    For teams building these features, the key question is no longer “Which model should we integrate?” but rather “How should we design a routing mechanism that can be adjusted over time?”

    1. What Multi-Model Aggregation Should Solve

    Multi-model aggregation means integrating multiple model services behind a single application entry point and using a unified scheduling layer to determine which model handles each request. Its main value lies in four areas:

    1. Capability matching: Simple classification, summarization, and format conversion can be assigned to lightweight models, while long-document analysis and multi-step reasoning can be handled by more capable models.
    2. Cost control: Pricing is usually related to the number of input and output tokens. The cost of a single request can be estimated as follows:

      Single-request cost = Input token count × Input price + Output token count × Output price

      Prices vary by model. Routing a large number of low-complexity requests to lower-priced models can significantly reduce overall spending.

    3. Response speed: Short tasks are often more sensitive to time to first token. Lightweight models are typically faster and are therefore suitable for interactive scenarios.
    4. Availability: When a provider times out or applies rate limits, the system can switch to a backup model and keep the feature available.

    Multi-model support does not mean choosing models randomly. Without a clear basis for routing, output quality becomes inconsistent, and users may not understand why the same operation produces different results. Routing therefore requires explicit decision criteria and an observable record of how each decision was made.

    2. Routing Criteria: Five Core Dimensions

    Routing decisions usually combine the following dimensions:

    • Task complexity: This can be determined by input length, the number of instruction steps, whether reasoning is required, and whether information must be combined across documents.
    • Latency requirements: Interactive scenarios generally require low time to first token, while background batch processing can tolerate longer execution times.
    • Cost budget: Usage or per-request cost limits can be defined for individual users, departments, or features.
    • Context length: Models support different input lengths. For long documents, models that cannot accommodate the required context should be excluded first.
    • Data compliance level: Requests involving sensitive data may be restricted to specific environments or providers.

    These dimensions can constrain one another. Compliance requirements may immediately exclude certain models. In that case, filtering should happen before optimizing for cost and latency.

    A recommended decision order is:

    1. Apply hard constraints, such as compliance, context length, and feature support.
    2. Rank the remaining candidates using soft constraints, such as latency, cost, and quality.

    3. Reference Architecture

    A practical multi-model routing system usually consists of six layers:

    1. Ingress layer: Provides unified authentication, rate limiting, and request validation so that the frontend does not depend on a specific model.
    2. Task analysis layer: Identifies the task type, estimates complexity, and detects sensitive information.
    3. Routing decision layer: Selects a list of candidate models based on rules or a scoring function.
    4. Model adapter layer: Converts the internal request format into the format required by each provider and handles differences in parameters.
    5. Execution and fallback layer: Calls the model, handles timeouts and errors, validates results, and switches to the next model according to priority when a request fails.
    6. Observability layer: Records the reasons for each routing decision, along with latency, token usage, and user feedback for future optimization.

    The adapter layer is often underestimated. Providers differ in message structures, system instruction support, tool-calling formats, streaming behavior, and error codes. If these differences are scattered throughout business logic, adding a new provider will require broad code changes.

    The following simplified example illustrates the basic approach to layered routing and fallback. The model names are placeholders and should be replaced with service identifiers registered in the team’s configuration center.

    1. from dataclasses import dataclass
    2. from typing import Callable, Optional
    3. @dataclass
    4. class Task:
    5. text: str
    6. sensitive: bool
    7. max_latency_ms: int
    8. @dataclass
    9. class ModelSpec:
    10. name: str
    11. tier: str # "light" or "heavy"
    12. max_context: int
    13. allow_sensitive: bool
    14. call: Callable[[str], str]
    15. def estimate_complexity(task: Task) -> str:
    16. # Example rule: treat long text or many instruction lines as complex
    17. if len(task.text) > 4000 or task.text.count("\n") > 30:
    18. return "heavy"
    19. return "light"
    20. def build_candidates(task: Task, registry: list[ModelSpec]) -> list[ModelSpec]:
    21. # Step 1: Apply hard-constraint filtering
    22. pool = [m for m in registry
    23. if len(task.text) <= m.max_context
    24. and (m.allow_sensitive or not task.sensitive)]
    25. # Step 2: Prefer the matching complexity tier; keep others as fallbacks
    26. want = estimate_complexity(task)
    27. return sorted(pool, key=lambda m: 0 if m.tier == want else 1)
    28. def run_with_fallback(task: Task, registry: list[ModelSpec]) -> Optional[str]:
    29. for model in build_candidates(task, registry):
    30. try:
    31. return model.call(task.text)
    32. except Exception:
    33. # Production systems should log the error and distinguish timeouts
    34. # from business or validation failures.
    35. continue
    36. return None # The caller handles degradation if all candidates fail

    The rules in this example are intentionally basic. A production system should combine them with task-type labels and other routing signals.