Model Selection, Cost, and Routing Practices for Multi-Model Office Agents
This article explores the practices of selecting, managing costs, and routing models for multi-model office agents. It covers task heterogeneity, cost multipliers, layered routing architecture, engineering considerations for scheduling, user interface transparency, evaluation systems, and security compliance. Readers will learn how to build a robust multi-model system that balances performance, speed, and cost effectively.
1. Background: Why Office Agents Need Multiple Models
Office agents operate differently from single-turn Q&A. A user provides a goal, and the system decomposes it into multiple steps, sequentially calling tools like retrieval, document parsing, and spreadsheet calculation to deliver the final result. A single task may involve planning steps requiring deep reasoning, as well as simple steps like format conversion or short text refinement.
If all steps use the same model, two issues arise. First, using high-performance models for simple steps leads to high per-call costs, rapidly depleting the overall budget. Second, using lightweight models for complex planning results in unstable output quality, often requiring multiple retries, which ultimately increases total latency. Therefore, the value of integrating multiple models lies not just in expanding options, but in establishing a mechanism for task-specific model allocation.
2. Task Heterogeneity: Trade-offs in Three Dimensions
Before selecting a model, it is necessary to clarify three mutually constraining metrics:
- Effectiveness: Whether the output is correct, complete, and passes business validation.
- Response Speed: Time to first token (TTFT) and overall completion time, determining interaction smoothness.
- Cost: Billing consumption per call, determining the sustainability of large-scale operations.
These three factors usually cannot be optimal simultaneously. Models with better effectiveness often have higher latency and higher unit prices; models with lower latency are suitable for interactive scenarios but are more prone to errors in long-chain reasoning. Model selection should assign weights based on task types rather than seeking a globally optimal model.
The table below outlines a common task classification approach:
| Task Type | Typical Example | Effectiveness Weight | Speed Weight | Cost Weight |
|---|---|---|---|---|
| Planning & Decomposition | Breaking down goals into execution steps | High | Medium | Low |
| Long Document Analysis | Meeting minutes organization, contract clause comparison | High | Low | Medium |
| Format Conversion | Spreadsheet field organization, Markdown conversion | Medium | High | High |
| Short Rewriting | Title refinement, email tone adjustment | Low | High | High |
This table serves as an example; specific weights should be calibrated based on business data.
3. Cost Multipliers: Translating Cost into Comparable Numbers
Pay-as-you-go platforms or internal systems often use a relative multiplier to standardize the consumption of different models. Each model corresponds to a multiplier, and actual consumption equals the base metric unit multiplied by the multiplier.
Consumption = Base metric unit × Model multiplier
The lower the multiplier, the less consumption required to complete the same task. This design serves three purposes. First, users can directly compare the relative costs of different models. Second, administrators can adjust a specific model’s multiplier without changing the base pricing rules. Third, the base metric unit can be derived from complex metrics like token count or call duration, making it easier for users to understand.
However, the multiplier only reflects relative cost and cannot replace effectiveness evaluation. A model with a very low multiplier might lead to a task needing to be redone three times, resulting in higher total consumption. Therefore, cost evaluation should be based on the total consumption required to complete a task, rather than the unit price of a single call.
4. Layered Routing Architecture
A mature multi-model system typically includes the following five components:
- Task Classifier: Determines the task level based on features like instruction length, number of tools involved, and whether multi-step reasoning is required.
- Routing Strategy: Selects a model from the candidates based on task level, user preferences, and remaining budget.
- Quality Gatekeeper: Performs structural validation, key field checks, or small model scoring on the output. If the result is invalid, it escalates to a stronger model for a retry.
- Budget Controller: Records cumulative consumption. When approaching the limit, it downgrades to a low-multiplier model or requests user confirmation.
- Logging and Metrics: Records the model, latency, consumption, and validation results for each call for subsequent optimization.
Below is an illustrative routing implementation. The model names and parameters are placeholders and do not correspond to any real service.
from dataclasses import dataclass@dataclassclass ModelProfile:name: strquality: int # 0-10, offline evaluation scorelatency_ms: int # Typical time to first tokenmultiplier: float # Consumption multiplierCANDIDATES = [ModelProfile("model_fast", quality=6, latency_ms=400, multiplier=0.5),ModelProfile("model_balanced", quality=8, latency_ms=900, multiplier=1.0),ModelProfile("model_strong", quality=9, latency_ms=2000, multiplier=2.0),]MIN_QUALITY = {1: 5, 2: 7, 3: 8} # Task level -> Minimum quality requirementdef select_model(task_level: int, max_multiplier: float, prefer_speed: bool) -> ModelProfile:min_quality = MIN_QUALITY.get(task_level, 7)eligible = [m for m in CANDIDATES if m.quality >= min_quality]if not eligible:eligible = [max(CANDIDATES, key=lambda m: m.quality)]# Remaining budget converted to the maximum acceptable multiplieraffordable = [m for m in eligible if m.multiplier <= max_multiplier]pool = affordable or [min(CANDIDATES, key=lambda m: m.multiplier)]if prefer_speed:return min(pool, key=lambda m: m.latency_ms)return max(pool, key=lambda m: m.quality)def run_with_gate(prompt, task_level, max_multiplier, prefer_speed, call_model, validate):model = select_model(task_level, max_multiplier, prefer_speed)result = call_model(model.name, prompt)if validate(result):return result, model.name# Validation failed, upgrade to the highest quality model for a single retrystrongest = max(CANDIDATES, key=lambda m: m.quality)return call_model(strongest.name, prompt), strongest.name
Three points require attention in this implementation. First, the quality threshold should come from offline evaluation rather than arbitrary manual settings. Second, retries should only happen once to avoid infinitely amplifying costs on the failure path. Third, call_model and validate are injected functions, making it easy to replace underlying interfaces and write unit tests.
5. Engineering Considerations for Periodic Discounts and Scheduling
Some platforms reduce the multiplier for specific models during certain time periods. The scheduling system can factor this into its decisions, but should follow a few principles:
- Distinguish task urgency: Interactive tasks should execute immediately, while non-urgent batch tasks (e.g., generating weekly reports at night) can be scheduled during discount periods.
- Use UTC uniformly for storage: Convert to the user’s local timezone only during display to avoid cross-timezone calculation errors.
- Externalize rules: Rules for holidays and discount periods should be placed in configuration files or a configuration center, not hardcoded in business logic.
- Handle long tasks crossing boundaries: Set checkpoints to allow execution to continue after a period switch, and re-estimate remaining consumption.
- Provide pre-execution estimates: Show estimated consumption to the user before execution. If a discount period is about to end, clearly inform the user of alternative options.
These measures aim to make cost optimization predictable and explainable, rather than having users incur unexpected charges unknowingly.
6. User Interface and Transparency
The experience of a multi-model system largely depends on whether users understand the cost of their choices. UI design recommendations include:
- On the model selection page, simultaneously display the relative multiplier, typical response speed, and applicable task types for each model.
- Recommend a default model with a one-sentence explanation (e.g., “Suitable for multi-step planning, moderate consumption”).
- Display actual consumption after task completion and allow users to pin a specific model as a favorite.
- Clearly define the scope of free quota (e.g., only covers single-step, short text simple tasks).
Transparent rules reduce user suspicion regarding billing and facilitate quick issue identification for customer service and operations teams.
7. Evaluation System: Measuring Routing Effectiveness with Cost Per Success
Before and after a routing strategy goes live, unified metrics are needed to evaluate its effectiveness. It is recommended to build three types of metrics:
- Offline Evaluation Sets: Build samples by task type, covering dozens to hundreds of real cases per category, with labeled expected results.
- Core Online Metrics: Task success rate, average consumption, P95 latency, retry rate, manual takeover rate.
- Composite Metric: Cost per success, calculated as total consumption divided by the number of successful tasks.
Cost per success is the metric that best reflects the value of routing. Cheap per-call costs do not guarantee overall savings; only by factoring in the costs of failed retries and manual interventions can you determine if a strategy truly reduces costs.
When launching a new strategy, a grayscale (canary) release is recommended, allocating a small portion of traffic to the new router for comparison with the original strategy. If the success rate drops, roll back even if costs decrease.
8. Security, Compliance, and Dependency Risks
Integrating multiple models expands the scope of data flow, requiring advance planning for the following:
- Data Classification: Categorize data into public, internal, and sensitive levels. Sensitive data can only be routed to security-audited models or privately deployed environments.
- Output Validation: Regardless of the model’s origin, outputs should undergo format and business rule validation; model results should not be trusted blindly.
- Log Desensitization: Personal information, credentials, and business secrets in call logs should be desensitized or encrypted before being written.
- Unified Abstraction Layer: Business code should only depend on internal interfaces, avoiding direct coupling with any specific model’s calling method to facilitate replacement and reduce dependency risks.
- Version Regression: Model version upgrades may change output behavior. Run regression test sets before upgrading to compare key metrics.
These measures keep the system controllable and auditable as more models are introduced.
9. Implementation Recommendations
In actual projects, you can proceed in the following order:
- Organize the task list,
