Multi-Model Collaboration in the Office: Engineering Practices for Task-Tiered Model Selection
This article explores engineering practices for selecting large language models based on task tiers in office scenarios. It covers four key dimensions for model selection, a three-tier classification system, routing layer design, long document processing, evaluation set creation, cost and latency governance, data security, and a phased implementation roadmap. By adopting this tiered approach, teams can improve task-model matching, control costs for high-frequency lightweight tasks, and maintain systematic model evaluation.
1. Background: Why Office Tasks Need Tiered Model Selection
AI tasks in daily office environments vary significantly. Organizing a dozen reports requires ingesting a large amount of material at once, polishing an official document focuses on whether the expression is natural, and modifying a single sentence requires very fast response times. Assigning all these tasks to the same model will either result in excessive costs or unstable performance for certain task types.
The tiered selection approach works as follows: first, categorize tasks by characteristics; second, assign an appropriate model tier to each category; finally, use a unified routing layer to dispatch the requests. This approach offers three main benefits. First, the matching between tasks and models is much higher. Second, the cost of high-frequency, small-scale tasks is controllable. Third, the team can use a consistent set of rules to explain why a specific model was used for a specific task.
2. Four Dimensions for Model Selection
When selecting models, it is recommended to fix four dimensions and evaluate them item by item, rather than relying on impressions.
2.1 Context Window
The context window determines how much input a model can receive at once. When the total material exceeds the window, segmentation becomes necessary. Segmentation introduces issues such as lost cross-segment references and contradictory conclusions. If a task requires a global grasp of the material, such as “comparing the assumption differences among three documents,” you should prioritize models with larger windows or switch to a retrieval-plus-summarization workflow.
Window size does not equal effective utilization rate. Information at certain positions within the window may receive less attention. This needs to be confirmed through evaluation and cannot be judged solely by parameters.
2.2 Language Understanding and Generation Quality
Office texts include official documents, meeting minutes, weekly reports, and marketing copy, which have strict requirements for wording, style, and format. When evaluating, you should not only check if the text is fluent but also examine the following:
- Are the terms accurate?
- Is the tone appropriate for the scenario?
- Are critical decision items omitted?
- Are there factual hallucinations?
(Note: For Chinese scenarios, particular attention should be paid to the naturalness and correctness of formal Chinese expressions.)
2.3 Latency and Unit Cost
Interactive scenarios are sensitive to time-to-first-token latency. If the wait time for real-time correction during input exceeds two seconds, it will disrupt the writing rhythm. Batch tasks, on the other hand, are more concerned with throughput and total cost. These two types of scenarios should be measured and monitored separately to avoid interfering with each other.
2.4 Stability and Controllability
Stability includes output format consistency, refusal rate, rate limiting policies, and service availability. For tasks that need to be integrated into downstream systems, the stability of structured output is often more important than the quality of a single response.
3. Typical Configuration for Three-Tier Classification
Models can be divided into high, medium, and low tiers:
- High tier: Long document comprehensive analysis, multi-source material summarization, and complex reasoning tasks.
- Medium tier: General production tasks such as native language writing, manuscript revision, and meeting notes organization.
- Low tier: High-frequency lightweight tasks such as rewriting sentences, generating outlines, and concept lookups.
Tiers should not be hard-coded to a specific brand mapping, but should be determined by evaluation results. Model versions iterate rapidly, and teams need to regularly re-evaluate using the same set of samples to confirm whether the tier assignments remain valid.
4. Routing Layer Design
The routing layer is responsible for mapping requests to tiers. The rules can be divided into three layers: first, hard match by task type; second, fallback based on input length; finally, use a default tier for uncategorized requests.
from dataclasses import dataclass@dataclassclass Request:task_type: str # e.g., "summarize_long", "polish_zh", "quick_fix"token_estimate: int # Estimated input tokensinteractive: bool # Whether it is an interactive request# Tier names are illustrative; actual mappings should be maintained in a configuration fileTIER_MAP = {"summarize_long": "high_context","multi_source_analysis": "high_context","polish_zh": "zh_writing","meeting_notes": "zh_writing","quick_fix": "lightweight","concept_lookup": "lightweight",}LONG_INPUT_THRESHOLD = 32_000 # Illustrative threshold, adjust based on actual window and costSHORT_INTERACTIVE_LIMIT = 2_000def route(req: Request) -> str:tier = TIER_MAP.get(req.task_type, "zh_writing") # Unknown tasks use a conservative default tierif req.token_estimate > LONG_INPUT_THRESHOLD:tier = "high_context"if (req.interactive and tier == "high_context"and req.token_estimate < SHORT_INTERACTIVE_LIMIT):tier = "lightweight"return tier
This code embodies three principles. The rules are explicit and readable; the default tier is conservative; thresholds are placed in configuration rather than scattered in the logic.
5. Long Document Processing Workflow
When materials exceed the single input window, two common methods are used.
The first is chunked summarization (map-reduce). First, cut the document into semantically complete paragraph blocks, extract key points block by block, and then perform a secondary summarization of the key points. Chunking should be done along chapter or paragraph boundaries to avoid splitting a single argument. Each chunk should retain its source identifier so the summarized results can be traced back to the original chapter.
The second is retrieval augmentation. First, build an index for the document, and when the user asks a question, only retrieve relevant fragments to send to the model. This method is suitable for Q&A requirements but has limited effectiveness for tasks requiring global structure.
Both methods should save intermediate results. Intermediate artifacts can be reused and facilitate manual spot-checking.
def summarize_long_document(chunks: list[str], call_model) -> str:"""call_model is the unified interface for calling the model, injected by the caller."""partial_notes = []for i, chunk in enumerate(chunks, start=1):prompt = f"Extract the core arguments and data of segment {i}, retaining the original chapter number:\n{chunk}"partial_notes.append(call_model(tier="high_context", prompt=prompt))merged = "\n".join(partial_notes)return call_model(tier="high_context",prompt=f"Based on the following segmented key points, output a structured summary of the full text:\n{merged}",)
6. Establishing a Custom Evaluation Set
Scores on public benchmarks cannot directly represent the effectiveness in office scenarios. It is recommended that teams extract around 100 samples from real tasks, covering various task types, and specify scoring criteria for each sample.
Scoring is divided into automatic metrics and manual scoring. Automatic metrics include format validation, key information coverage, word count range, and sensitive word checks. Manual scoring focuses on tone, accuracy, and usability, using a one-to-five point scale, scored independently by two reviewers, with samples with significant discrepancies discussed collectively.
Evaluation records should include the model version, prompt version, and date for reproducibility. After a model or prompt changes, re-run using the same set of samples to compare differences before and after.
7. Cost and Latency Governance
Cost governance mainly relies on three things: controlling input length, caching repeated results, and setting quotas by tier.
- Input length: Remove irrelevant templates, compress historical conversations, and summarize long attachments first.
- Caching: The combination of the same document and the same instruction can directly reuse the result. The cache key should include the model version and prompt version; otherwise, outdated results will be returned after version changes.
- Quotas: Set daily usage limits for each department or user. When exceeded, downgrade to a lower tier instead of throwing an error directly.
For monitoring, it is recommended to record the tier, input/output token count, duration, and whether a downgrade occurred for each call. This data can be written to a logging service, and anomalies can be observed through monitoring alerts. For example, if the average duration of a certain type of task suddenly doubles, it usually indicates that the prompt has become longer or upstream rate limiting has occurred.
8. Data Security and Compliance
Office data often contains customer information, financial data, and internal decisions. Before accessing any external model, three tasks should be completed: data classification, desensitization rules, and approval processes.
- Data classification: Label data as public, internal, or confidential. Only public and internal data can enter general tiers.
- Desensitization: Before the request enters the routing layer, replace sensitive fields such as phone numbers, ID numbers, and bank account numbers.
- Minimum privilege: Service accounts can only access the data sources required for the current task, and all call records are kept in audit logs.
9. Implementation Roadmap
It is recommended to advance in three phases.
Phase 1: Organize the task list. Count the frequency, average input length, and fault tolerance requirements of each type of task to form a task profile.
Phase 2: Establish an evaluation set. Select one or two candidate models for each of the three tiers, and run through the routing layer and logging pipeline.
Phase 3: Grayscale launch. First run it within a small team for two to four weeks, collect misclassification cases, and adjust rules and thresholds accordingly.
Common pitfalls include: selecting models based on a single metric, hard-coding tier mappings in the code, ignoring data compliance, and not re-evaluating after launch. Avoiding these issues will ensure that the tiered solution operates stably in the long term.
