Choosing Multi-Model Office AI: Evaluation Methods and Cost Governance
Office AI assistants are judged by a different standard than chat assistants. This article explains how to build a repeatable evaluation loop (controlled variables, a weighted scorecard, normalized quality/latency/cost metrics), how to model token multipliers and credits as a computable cost function, how to tier office tasks into four levels and route them across models with retries and fallback, and how to engineer long-form delivery through an outline-first pipeline with citation verification. It also covers quota guardrails, a versioned golden task set, regression and canary evaluation, and a practical implementation checklist for teams that need predictable quality, predictable spend, and a fallback path when external model services degrade.
1. Where the Problem Starts: From “Can Chat” to “Can Deliver”
Office work places requirements on an AI assistant that are fundamentally different from casual chat. A biography, a competitor comparison table, a summarized weekly report: none of these are judged on whether a single response reads smoothly. They are judged on four hard metrics — can the deliverable be used as-is, does the content carry verifiable sources, did it finish inside an acceptable time window, and was credit consumption predictable.
That difference produces two common misjudgments:
- Treating “occasionally got it right” as the capability ceiling. Long-form generation has a clear positional effect: the opening is well structured, the middle drifts into repetition and padding, and the ending drifts on facts. A single trial will not surface any of this. You need a fixed task set, measured repeatedly.
- Treating the subscription price as the total cost. Subscription price is only the entry threshold. Real cost is determined jointly by model multipliers, the input/output token mix, context reuse rate, and the number of retries and reworks. Two products with similar price tags can differ by an order of magnitude in credit consumption for the same workload.
A more practical risk is availability: an external model service can become briefly unusable because of policy changes, quota limits, or interface revisions. So the architectural goal for office AI is not “pick the single best model” — it is to build a multi-model system that is swappable, measurable, and reversible.
2. A Reproducible Evaluation Method: Controlled Variables Are the Point
For evaluation conclusions to be trustworthy, you have to fix three classes of variables and vary only one observation dimension.
| Variable | How to control it | Common contamination |
|---|---|---|
| Task | Same task brief, same word count and structure requirements | A biography in one run, a weekly report in another |
| Material | Same corpus, same retrieval scope, same citation rules | The full document pasted in one run, only a title given in another |
| Model | Same model, same parameters, same temperature | A flagship model on one side, a lightweight model on the other |
| Observation | Record end-to-end latency, credit consumption, output length, citation verifiability rate | Screenshots only, no metrics |
Under this control, use a scorecard instead of intuition. The hard part of scoring is unit alignment: quality is 0–100, latency is in seconds, credits are fractional, and everything has to be normalized before weighting.
from dataclasses import dataclass@dataclassclass TaskResult:quality: float # Quality score 0-100, scored through a dual human/rule-based passlatency_s: float # End-to-end latency (seconds)credits: float # Credits or points consumedcitation_pass: float # Citation verifiability rate 0-1schema_pass: bool # Whether the output matches the agreed structuredef score(r: TaskResult,w_q=0.5, w_t=0.15, w_c=0.2, w_r=0.15) -> float:# Normalize with the P95 latency and mean credits of the baseline task set so units stay comparablet_norm = min(1.0, 300.0 / max(r.latency_s, 1.0)) # within 5 minutes counts as full marksc_norm = min(1.0, 1.0 / max(r.credits, 1e-6)) # within 1 credit counts as full marksbase = (w_q * r.quality / 100+ w_t * t_norm+ w_c * c_norm+ w_r * r.citation_pass)# Schema-invalid output cannot enter downstream flows, so penalize it directlyreturn base if r.schema_pass else base * 0.6
The weights do not have to be identical everywhere, but they should be grouped by task type: for external-facing copy, push quality and citation weight up to 0.8; for internal drafts, relax quality and tighten cost. Run each task at least three times and take the median, so network jitter does not contaminate the conclusion.
3. Cost Abstraction: Turning “Multipliers” into Something Computable
Platforms commonly express cost differences as a multiplier plus credits, where a higher multiplier means a single call is converted into more consumption. In engineering terms, model it as a function — there is no need to agonize over unit naming:
PRICE = { # Per 1K tokens. Placeholder example values — use the provider's current published pricing"light": {"in": 0.001, "out": 0.002, "multiplier": 1.0},"pro": {"in": 0.020, "out": 0.060, "multiplier": 4.0},}def estimate(model: str, in_tok: int, out_tok: int,cached_in_tok: int = 0) -> dict:p = PRICE[model]billable = max(in_tok - cached_in_tok, 0)raw = billable * p["in"] / 1000 + out_tok * p["out"] / 1000return {"money": round(raw, 4),"credits": round(raw * p["multiplier"], 4),"cache_hit_rate": round(cached_in_tok / max(in_tok, 1), 3),}
Three frequently underestimated ways to cut cost:
- Prefix caching and context stabilization. Put the system prompt, glossary, writing guidelines, and long-lived organizational information at the very front of the prompt and keep it byte-for-byte stable; this materially raises prefix cache hit rates. Conversely, injecting a timestamp at the top of the prompt on every call invalidates the entire cached prefix.
- Input slimming. Extract, deduplicate, and chunk long documents before they reach the model instead of feeding in an entire book. Input-side waste is usually larger than output-side waste, because long documents tend to be re-read repeatedly.
- Asynchronous batching. Non-real-time work — batch summarization, tagging, condensation — belongs on an offline queue. This smooths peaks and is usually cheaper in credits than synchronous calls.
At the same time, add quota guardrails: set a daily budget ceiling per user or team, and degrade to a lightweight model with a notice when it is exceeded, rather than letting a task retry indefinitely.
4. Task Tiering and Model Routing
The real value of a multi-model setup is routing, not “picking one”. Split office tasks into four levels by complexity:
- L1 structured extraction and rewriting: field extraction, format conversion, terminology normalization — a lightweight, fast model is enough.
- L2 summarization and first drafts: meeting notes, source summaries, comparison tables — a mid-tier model plus citation constraints.
- L3 deep long-form: biographies, industry analysis, solution documents — a flagship reasoning model, and it must “plan first, then write in sections”.
- L4 long tool chains: multi-step retrieval, table computation, cross-document reconciliation — emphasize stability and retryability over single-point intelligence.
LEVELS = {1: {"model": "light", "retries": 1},2: {"model": "pro", "retries": 2},3: {"model": "pro", "retries": 2, "plan": True},4: {"model": "pro", "retries": 3, "plan": True},}def route(task) -> dict:cfg = LEVELS[task.level]if cfg.get("plan"):outline = complete(cfg["model"], OUTLINE_PROMPT, schema=OutlineSchema)return plan_then_write(cfg, outline) # generate section by section, then mergereturn complete(cfg["model"], task.prompt, schema=task.schema)def call_with_fallback(task, cfg):for i in range(cfg["retries"] + 1):try:return complete(cfg["model"], task.prompt, schema=task.schema)except (Timeout, RateLimited):if i == cfg["retries"]:# Degrade to a backup model and tag the call chain as degraded so it can be tracedreturn complete("light", task.prompt, schema=task.schema)backoff(i)
The more advanced pattern is cascading routing: answer with a lightweight model first and have it emit a self-assessed confidence score; escalate to the flagship model only when confidence falls below a threshold (for example, extracted fields are missing or citations are empty). In practice, a large share of L1 tasks pass on the first try with a lightweight model, which brings total cost well below “route everything to the flagship”.
The prerequisite is a unified backend abstraction, so business code depends on an interface rather than on a specific service:
class LLMBackend(Protocol):def complete(self, messages: list[dict],schema: type | None = None) -> Completion: ...
With that in place, switching providers, canary releases, load tests, and side-by-side comparisons are all configuration changes. If your evaluation and inference services run on a cloud platform such as Baidu AI Cloud, you can host evaluation jobs and the inference gateway on its container platform, keep source corpora and deliverables in object storage, write call traces to a log service, connect quota levels and failure rates to monitoring and alerting, and use a message queue to absorb the peaks of offline batch work.
5. Engineering Long-Form Delivery
Asking a model to produce 5,000 words in one call almost guarantees loose structure and internal inconsistency. The reliable approach is to split the work into a pipeline:
- Outline first. Produce a structured outline with section headings and key points, confirm it with a human or a rule check, and only then generate the body. The outline stage is very cheap, yet it eliminates most rework.
- Generate sections in parallel. Call the model independently per section, passing each one the shared glossary, timeline of people, and style samples to keep things consistent. Parallelism brings long-form latency down from “linearly accumulated” to roughly the time of a single section.
- Merge and deduplicate. At the rule layer, remove repeated paragraphs and repeated transition sentences, and normalize numbering, tense, and forms of address.
- Verify facts and citations. This is the step where long-form work most often breaks down.
def verify_citations(draft, index, threshold=0.9):bad = []for seg in draft.segments:for c in seg.citations:doc = index.get(c.doc_id)if doc is None or not snippet_match(c.quote, doc.text, threshold):bad.append(c)return bad
Citations must be traceable back to a specific source snippet, not self-reported by the model. Segments that fail verification go back for rewriting or get flagged as “to be confirmed”, so hallucinations never reach the deliverable.
- Normalize output. Constrain structured output with a JSON Schema, constrain naming with a terminology allowlist, and constrain layout with length and paragraph rules. The more explicit the constraints, the less work the downstream quality check has to do.
- Close the quality loop. An automated checker (structure, length, citations, banned terms) plus human spot checks (readability, facts, house style). The automated layer catches about 80% of defects; humans only review the key conclusions.
6. Turning Evaluation into a Sustainable Selection Process
- Build a golden task set. Cover 5–10 real tasks for each of L1–L4, versioned together with their source materials and expected answers. This set is the anchor for every later conclusion.
- Run regression evaluations. After a provider upgrades a model, a prompt changes, or a routing policy is revised, rerun the golden set and compare the quality, latency, and credit curves so that “fixing one thing” does not break a whole area.
- Canary and dual-run. Put a new model on 5%–10% of traffic, score it in parallel with the old path, and roll out fully only once it meets the bar.
- Suggested dashboard metrics: first-pass success rate, rework rate, credits per thousand words, P95 latency, degradation rate, citation verifiability rate, timeout rate. Looking only at the subscription price, or only at a single trial, is how teams get burned when planning capacity and budget.
7. Implementation Checklist
- Task tiering standard defined and mapped one-to-one to model tiers;
- Cost function and quota guardrails live, with automatic degradation on overrun rather than silent failure;
- All long-form work follows “outline first, then sections”, with single-call whole-document generation prohibited;
- Citations must be traceable, and segments that fail verification may not go straight into delivery;
- Golden task set versioned, and any change triggers a regression evaluation;
- Backends abstracted behind a unified interface that supports multi-service switching and canaries;
- Quality, latency, cost, and failure rate metrics wired into monitoring and alerting.
The conclusion is not complicated: the competitiveness of office AI does not come from how impressive a single generation looks, but from reproducible evaluation, predictable cost, and a reversible architecture. When an external service starts to fluctuate, a team that owns a golden set and a routing layer only has to flip one configuration flag, rather than re-estimating the whole workflow.
