One Platform, Multiple Models: Architecture and Cost Reduction Practices for Enterprise AI Productivity

    This article explains why enterprise productivity workloads need multiple large language models instead of a single flagship model, and how to support them through one unified access gateway. It covers a four-layer gateway design, vendor-agnostic model abstraction with data-driven routing, a cost model that turns pricing multiplier changes into measurable savings, and four engineering levers—context engineering, caching, batching, and output-length control—that reduce spend. It also maps common office scenarios to model tiers, outlines reliability, security, and compliance requirements, and lists recurring anti-patterns along with the metrics needed to evaluate quality, cost, and user experience.

    1. Why Enterprise Productivity Work Requires Multiple Models

    When large language models are introduced into office workflows, two extremes show up most often. One is standardizing on a single flagship model for everything, which lets cost be dominated by a small number of high-frequency, lightweight tasks. The other is letting every team integrate its own model, which scatters API keys across the organization, produces inconsistent behavior, and makes unified governance impossible.

    Real productivity traffic is highly skewed. Grouped roughly by task complexity, it falls into three tiers:

    • Lightweight tier: format conversion, text summarization, field extraction, sentiment and intent classification, keyword extraction, translation and polishing. Short inputs, short outputs, high error tolerance, and the highest call volume.
    • Standard tier: multi-turn Q&A, structured content generation (weekly reports, first drafts of proposals, table and form filling), and retrieval-augmented Q&A over a knowledge base.
    • Deep tier: cross-document comparison, multi-step reasoning and planning, long-form writing, and complex code or data pipeline generation. High consumption per call and very high accuracy requirements.

    One fact settles the question: cost is driven primarily by token consumption, and token consumption varies enormously across tiers. Model selection therefore should not be a one-size-fits-all decision made at the organization level; it should be a routing decision made at the request level. This is the real reason for “one platform, many models”—not the number of models on offer, but the ability to match the right model to each request.

    2. Target Architecture: A Four-Layer Unified Access Gateway

    The sound implementation collapses model capability into a single access layer (the model gateway). Applications above it speak one protocol; interchangeable models sit below it.

    1. Applications / productivity suite
    2. ↓ Unified protocol (chat completions / structured output / streaming)
    3. [Access Gateway] Auth → Routing → Context Assembly → Invocation → Post-processing → Billing Telemetry
    4. ↓ Adapters
    5. [Model Layer] Lightweight · Standard · Deep Reasoning · Embedding/Rerank · Speech Recognition
    6. ↓
    7. [Platform Layer] Object storage · Message queue · Container platform · Logging · Monitoring & alerting · Secrets and credential management

    Each layer has a clear responsibility:

    1. Protocol layer: expose a vendor-neutral interface style (the widely adopted HTTP convention of “chat completions + optional streaming + optional structured output”), and hide field-level differences between backends inside the adapters.
    2. Routing layer: decide which tier of model to use based on task labels, context length, remaining budget, and historical quality feedback, with degradation and failover built in.
    3. Capability layer: unify the abstraction of multimodal and tool capabilities—long context, document parsing, speech transcription, vector retrieval, and reranking—so that no single backend’s calling conventions leak into business code.
    4. Governance layer: quotas, rate limiting, cost attribution, audit trails, and content safety are all handled inside the gateway, so business services never reimplement them.

    3. Capability Abstraction and Routing: Illustrative Code

    The core idea is to turn “which model should I use” into a data-driven policy rather than scattered if-else branches in business code.

    1. # Illustrative code: vendor-agnostic model abstraction and routing policy
    2. from dataclasses import dataclass
    3. from enum import Enum
    4. import hashlib, json
    5. class Tier(str, Enum):
    6. LIGHT = "light" # summarization, rewriting, extraction, classification
    7. STANDARD = "standard" # multi-turn Q&A, structured generation
    8. DEEP = "deep" # multi-step reasoning, long-form writing, complex code
    9. @dataclass
    10. class ModelSpec:
    11. alias: str # internal platform alias, decoupled from the backend
    12. tier: Tier
    13. ctx_window: int # context window (tokens)
    14. input_rate: float # example price per 1K input tokens
    15. output_rate: float # example price per 1K output tokens
    16. supports_json: bool = True
    17. supports_stream: bool = True
    18. # Registry: delivered by the platform config service, adjustable via canary rollout
    19. REGISTRY = [
    20. ModelSpec("lite-a", Tier.LIGHT, 32_000, 0.0008, 0.0020),
    21. ModelSpec("std-b", Tier.STANDARD, 128_000, 0.0040, 0.0120),
    22. ModelSpec("think-c", Tier.DEEP, 200_000, 0.0080, 0.0240),
    23. ] # Rates above are assumed values used only to demonstrate the calculation logic; they are not any vendor's real pricing
    24. def route(task_label: str, est_input_tokens: int, require_json: bool,
    25. budget_left_ratio: float = 1.0) -> ModelSpec:
    26. tier = {"extract": Tier.LIGHT,
    27. "rewrite": Tier.LIGHT,
    28. "qa": Tier.STANDARD,
    29. "reason": Tier.DEEP}[task_label]
    30. candidates = [m for m in REGISTRY if m.tier == tier]
    31. if require_json:
    32. candidates = [m for m in candidates if m.supports_json] or candidates
    33. # Move up a tier automatically when the context would not fit
    34. candidates = [m for m in candidates if m.ctx_window >= est_input_tokens * 2] or candidates
    35. # When the budget tightens, fall back to the lighter tier: the first cost-control gate
    36. if budget_left_ratio < 0.3 and tier == Tier.DEEP:
    37. candidates = [m for m in REGISTRY if m.tier == Tier.STANDARD]
    38. return sorted(candidates, key=lambda m: m.input_rate)[0]
    39. def cache_key(payload: dict) -> str:
    40. """Normalize semantically stable fields to derive a reusable prefix cache key"""
    41. norm = json.dumps(payload, sort_keys=True, ensure_ascii=False)
    42. return hashlib.sha256(norm.encode("utf-8")).hexdigest()

    Three design points are worth emphasizing:

    • Alias decoupling: business code only holds an alias. Model switching, retirement, and canary rollout all happen in the config service, so “changing a model” never means “changing code”.
    • Automatic tier upgrade: when the estimated context exceeds a model’s window, the request moves to a larger-window model, preventing quality incidents caused by silent truncation.
    • Budget gate: cost constraints only actually hold down spending when they are an input to routing rather than a retrospective report.

    4. Cost Model: Turning Pricing Multiplier Reductions into Measurable Savings

    Billing is fundamentally cost per call = input tokens × input rate + output tokens × output rate, where the rate itself decomposes into base price × billing multiplier. The multiplier is an adjustment factor applied to a specific model tier. When a platform lowers the multiplier on the lightweight tier, the saving scales linearly with call volume—which is why the cost-reduction elasticity of high-frequency light tasks is far greater than that of low-frequency heavy tasks.

    Monthly cost can be written as:

    1. Monthly cost = Σ_tasks ( Q_t × ( I_t × p_in,t + O_t × p_out,t ) )
    2. where p = base unit price × multiplier, Q is call volume, I/O are average tokens per call

    A hypothetical scenario illustrates this (all prices and multipliers are illustrative values used only to show the calculation logic). A team makes 5 million calls per month, 82% of which are light tasks averaging 1,200 input tokens and 300 output tokens; standard and deep tasks average 1,500 input tokens and 500 output tokens.

    Option Calculation basis Monthly cost (example)
    Everything on the deep tier All 5 million calls billed at the high multiplier ~ CNY 8,400
    Tiered routing Light tasks go to the lightweight tier, the rest allocated per tier ~ CNY 2,800
    Tiered routing + a further 30% cut on the light-tier multiplier Light-task cost drops accordingly ~ CNY 2,600

    Two conclusions follow. First, the structural saving from routing (roughly two-thirds) is the primary source of gain. Second, a multiplier reduction is a “multiplicative bonus” layered on top of it: when light tasks account for more than 80% of call volume, its absolute saving is substantial, and it takes effect automatically with no architectural change—which is why running multiple models within a single platform matters most for heavy users.

    One caveat: any estimate must be based on your own real logs. The table above provides only a calculation framework; actual unit prices, multipliers, and discount terms should be taken from the service pages of your chosen platform and your commercial agreement with them.

    5. Four Engineering Levers for Lower Cost

    5.1 Context engineering: control the larger share of input cost

    Input tokens often account for more than half of the cost of a single call. Effective measures include:

    • Replace “stuff the whole document in” with vector retrieval that recalls only the fragments relevant to the question, then rerank them to improve the signal-to-noise ratio.
    • Apply rolling summarization to conversation history, keeping decision conclusions and discarding process redundancy.
    • Keep templates and system prompts stable at the prefix so prefix caching can hit.
    • Set an explicit budget: prompt budget = context window × 0.7, leaving the rest for output and tool results.

    5.2 Caching: prefix caching and semantic caching

    • Prefix caching: place invariant instructions and knowledge fragments at the very front of the prompt; on repeated calls the inference side reuses the computed result, which significantly reduces input-side cost and time to first byte.
    • Semantic caching: build a vector index for high-frequency, semantically equivalent questions (for example, “how is progress this week” versus “report on this week’s progress”) and return the historical answer directly when similarity exceeds a threshold. Always set a TTL and an invalidation policy, and disable caching for questions that involve personalized or real-time data—otherwise you introduce “stale answer” problems that are hard to diagnose.

    5.3 Batching and asynchrony

    Offline work such as meeting-note generation, bulk document archiving, and attachment parsing should be queued through a message queue and executed asynchronously, so it does not compete with interactive requests for real-time quota. It can also be served by a lower-priority tier, balancing cost against experience.

    5.4 Output length governance

    Output tokens are typically several times more expensive per token than input tokens, so the default inclination that “longer is better” needs a limit: specify an explicit max_tokens, require conclusions before elaboration, and expand in a second pass when necessary rather than in a single shot.

    6. Mapping to Real Productivity Scenarios

    Scenario Recommended tier Key engineering points
    Long-document summarization, attachment parsing Lightweight tier + long context Chunked map-reduce summarization with a uniform output schema
    Contract/form field extraction Lightweight tier with enforced structured output Retry once on validation failure, then fall back to the standard tier
    Knowledge-base Q&A Standard tier Retrieval + reranking + citation provenance; every answer must trace back to the source text
    Meeting notes → action items Speech transcription → lightweight extraction Speaker diarization, timestamp alignment, structured persistence of action items
    Email and copy polishing Lightweight tier Templated tone, a shared multilingual terminology glossary
    Cross-document comparison and due diligence Deep tier Explicit reasoning steps, with human review as the final gate
    Report interpretation and chart code Standard tier Delegate numeric computation to deterministic tools; let the model only explain

    One principle has been validated repeatedly: anything that rules and programs can determine deterministically should not be handed to the model. Leave classification, aggregation, arithmetic, and sorting to code; the model handles language understanding, information organization, and genuinely uncertain judgment.

    7. Reliability, Security, and Compliance Governance

    • Failover: configure a fallback chain for the primary tier (deep tier fails → standard tier with an explicit notice to the user), combined with exponential backoff and circuit breaking, so a single point of jitter cannot spread into a global outage.
    • Idempotency and replayability: assign a request ID to every call, and record the model alias, token usage, latency breakdown (especially time to first byte), cache hits, and cost estimates into the logging service; then configure cost and failure-rate thresholds for monitoring and alerting.
    • Data security: start with data classification (public / internal / confidential / containing sensitive personal information) and mask sensitive data before it leaves the network. Store keys centrally in a credential management service; never hardcode them or ship them to the frontend.
    • Prompt injection protection: treat retrieved documents, email bodies, and web content uniformly as untrusted input, explicitly isolate them from system instructions, and restrict the model from invoking write-capable tools without authorization.
    • Content safety: screen both inputs and outputs, and retain audit trails to meet enterprise compliance and traceability requirements.

    8. Evaluation and Continuous Iteration

    Optimization cannot rely on intuition. You need a small, stable evaluation set (a golden set): sample several hundred cases from real office tasks, annotate the expected outputs and acceptance criteria, and run a regression pass after every change to routing policy or prompts.

    Metrics worth tracking over the long term:

    • Quality: task first-pass success rate, structured-output parse success rate, human adoption rate, and rework rate.
    • Cost: average cost per task, cost per 1,000 calls, and the lightweight-tier interception rate (the share of traffic successfully routed down).
    • Experience: time to first byte at P50/P95, end-to-end latency, and failure and degradation rates.

    Beware the illusion of “watching unit price instead of total cost”. If the lightweight tier drives up the rework rate, the hidden cost of human review may far exceed the inference fees saved. The decision criterion should be “total cost of one acceptable delivery”, not the token unit price.

    9. Common Anti-Patterns

    1. Organization-wide single model: everyone shares the strongest model, and 80% of the budget is spent on summaries and rewrites.
    2. Changing a model means changing code: backend names are written into business logic, making canary rollout and rollback impossible.
    3. No budget guardrails: without quotas and alerts, one anomalous batch job empties the monthly budget.
    4. Over-caching: caching strongly personalized or time-sensitive results, which triggers hard-to-diagnose data inconsistencies.
    5. Ignoring structured-output validation: downstream parsing depends on fuzzy text, and a model upgrade causes widespread errors.
    6. Treating the model as a database: work that needs exact counting, sorting, or retrieval is more reliable and cheaper on a deterministic system.

    10. Conclusion

    The value of “one platform, many models” does not lie in the number of models. It lies in consolidating selection, quotas, cost, and observability into a single controllable plane: business teams write one codebase, the platform picks the tier automatically based on task complexity, lightweight tasks capture a larger share of the benefit whenever the multiplier is reduced, and deep tasks stay on the critical path.

    A three-step rollout is recommended. First, build the unified access layer and invocation telemetry to obtain a real picture of task distribution and cost. Next, introduce tiered routing plus context and caching optimizations to achieve structural cost reduction. Finally, complete the evaluation set, budget guardrails, and canary mechanisms, so that every adjustment to models or billing policy is backed by data. With these three steps done, AI becomes stable, predictable, governable infrastructure inside the office workflow—rather than an expense that is hard to estimate.