Building an Enterprise MaaS Platform: Architecture, Vertical Model Ecosystems, and Production Practices

    This guide walks through how to build an enterprise-grade MaaS (Model-as-a-Service) platform and cultivate a vertical model ecosystem. It breaks the platform into a "four layers plus one support system" architecture—infrastructure, model, model-serving, and agent/application layers, plus a cross-cutting support system—and explains how to put model routing, inference acceleration, and agent development into practice. It also covers data non-retention and privacy protection, the trade-offs between private and dedicated-instance deployment, cost estimation, evaluation, and the security checks to complete before integration. Written for developers and enterprise architects, it shows how to turn large language models into stable, controllable, and measurable business value.

    Conclusion First: What Enterprises Need Is a Sustainable Platform, Not a Single Model

    LLM adoption is shifting from “calling one model” to “operating a service system.” What enterprises really care about is rarely how large the model parameters are, but whether a model can be integrated reliably, called on demand, versioned in a controlled way, costed predictably, and connected securely to business systems.

    The core value of MaaS (Model-as-a-Service) is packaging complex concerns—training, inference, operations, security, and metering—into a platform, so enterprises can focus on business scenarios and their data assets. In a public–private partnership, an MaaS platform can also support regional industrial upgrading: the public side provides scenarios and resources, the private side provides technology and models, and companies across the ecosystem collaborate around compute, data, models, and applications.

    Using a regional public–private vertical-model ecosystem hub project as a reference, this article is aimed at developers and enterprise architects and answers four questions:

    1. How should an enterprise MaaS platform be layered?
    2. How do model routing, inference acceleration, and agent development work in practice?
    3. How do you choose between data non-retention, private deployment, and dedicated instances?
    4. Before integration, how do you handle selection, cost estimation, evaluation, and security readiness?

    Background and Platform Positioning: How a Public–Private Model Forms an Ecosystem Loop

    A regional MaaS platform is typically built through a partnership between a state-backed regional industrial investment operator and an LLM development company, positioned as the core carrier of a vertical-model ecosystem hub. The point of this model is not to attract investment for a single project, but to lower the barrier for small and mid-sized teams to use large models through a shared technical foundation.

    A public–private model offers three advantages:

    • Demand-side connection: The public side holds real scenarios such as city governance, public services, and traffic management, so the platform can support use cases like government Q&A, city operations, and hotline assistance.
    • Supply-side clustering: As a shared technical foundation, the platform attracts algorithm teams, data service providers, application developers, and system integrators to collaborate within one ecosystem.
    • Factor-side cost reduction: Coordinating compute facilities, data governance resources, and policy resources can significantly lower the early-stage investment required for companies moving in.

    As disclosed in a phase update after launch, dozens of upstream and downstream companies have moved in, covering compute infrastructure, data labeling and governance, model R&D and fine-tuning, and industry application development—forming an initial “compute–data–model–application” loop. The cluster effect shows up in three ways:

    1. Technology reuse: Underlying compute and model services are shared across teams, so applications do not rebuild infrastructure from scratch.
    2. Knowledge flow: Model teams and industry teams exchange ideas frequently within the same ecosystem, so vertical models iterate faster.
    3. Supply–demand matching: Industry application companies become consumers of model capabilities, while model teams receive feedback from compliant scenarios.

    The platform is not a single API forwarding gateway, but a unified service entry point for three roles:

    Role Core needs Platform capabilities
    Model provider Fast onboarding, stable service, version control Model hosting, inference optimization, elastic scaling, metering and billing
    Developer / tenant Low-cost access, fast app and agent building Unified API, agent development framework, evaluation and debugging tools
    End user Secure, trustworthy, reliable AI services Content safety, privacy protection, service-level guarantees

    From Policy Direction to Developer Action Items

    In recent years, many governments have introduced policies supporting large models and AI agents, and MaaS platforms are increasingly treated as the key infrastructure for putting those policies into practice. The policy logic can be summarized in three layers:

    • Factor supply: Through compute coordination, model openness, and data governance, lower the barrier for small and mid-sized companies to access AI capabilities.
    • Scenario pull: Use “AI-plus” initiatives to move key industries from pilot validation toward scaled replication.
    • Ecosystem governance: Use a public platform as the vehicle to unify content safety, data compliance, and service quality standards.

    For developers, policy direction ultimately translates into three action items:

    1. Predictable cost: Prefer platform-based, at-scale services, and set up monitoring for call volume and budget.
    2. Lower access barrier: Unified interfaces, unified authentication, and unified metering reduce redundant build-out.
    3. Clear compliance responsibility: Before integration, confirm data classification, retention scope, audit obligations, and responsibility boundaries between the platform and the enterprise.

    Overall Platform Architecture: Four Layers and One Support System

    A mature enterprise MaaS platform can be abstracted as “four layers and one support system”: an infrastructure layer, a model layer, a model serving layer, and an agent and application development layer, plus a support system that runs across all layers.

    1. ┌──────────────────────────────────────────────────────────────────┐
    2. │ Support System: Identity | Monitoring & Alerts | Logging & Audit | Content Safety | Billing │
    3. ├──────────────────────────────────────────────────────────────────┤
    4. │ Agent & App Dev Layer: Agent Orchestration | Tool Calling | Knowledge Base | Workflow │
    5. ├──────────────────────────────────────────────────────────────────┤
    6. │ Model Serving Layer: Unified API | Model Routing | Inference Acceleration | Fine-tuning Hosting | Versioning/Canary │
    7. ├──────────────────────────────────────────────────────────────────┤
    8. │ Model Layer: General Chat | Long Context | Multimodal | Code | Vertical Models │
    9. ├──────────────────────────────────────────────────────────────────┤
    10. │ Infrastructure Layer: Heterogeneous Compute | Container Platform | Object Storage | Message Queue │
    11. └──────────────────────────────────────────────────────────────────┘

    Infrastructure Layer

    The infrastructure layer provides stable compute, storage, and scheduling:

    • Heterogeneous compute: Training typically relies on GPUs, while inference also uses a range of accelerators. The platform needs to orchestrate resources through a container platform and enforce per-tenant quota isolation.
    • Storage and transport: Model weights, knowledge base documents, and datasets are usually stored in object storage; high-concurrency requests rely on message queues to smooth out spikes and troughs.
    • Elastic scaling: Scale up and down automatically based on request queue length and GPU utilization, avoiding peak-time queue timeouts and low-valley idle resources.

    Model Layer

    The model layer is the source of platform capability and typically includes:

    1. General foundation models: Covering everyday conversation, text understanding, and generation; some support long context and suit knowledge Q&A and document processing.
    2. Multimodal models: Understanding text, images, audio, and video, suitable for document parsing, image moderation, and multimodal interaction.
    3. Code and vertical models: Aimed at software development and specific business scenarios; domain knowledge can be injected through instruction fine-tuning and preference alignment.

    Model Serving Layer: Unified API and Model Routing

    The model serving layer turns “models” into “services.” Its core responsibilities include a unified API, inference optimization, fine-tuning hosting, version management, and model routing.

    When choosing an MaaS platform, model routing should not look only at model names; it should be driven by business constraints. Common routing dimensions include latency, cost, context length, multimodal support, compliance requirements, tool-calling capability, and availability.

    Priority Business scenario Routing recommendation
    Latency-sensitive Voice assistants, real-time Q&A Prefer high-speed API, low-latency models, streaming output
    Cost-sensitive Batch summarization, ticket classification Prefer low-cost models, cache reuse, batch processing
    Long-document Contract review, manual Q&A Prefer long-context models with retrieval augmentation
    Multimodal Design-to-code, image moderation Prefer multimodal or multimodal-code models
    Highly sensitive Government, healthcare, finance Prefer dedicated instances or private deployment

    Agent and Application Development Layer

    A model is only the “brain.” To deliver usable capability, you still need to build perception, memory, planning, and action on top of it. An agent development framework typically provides:

    • Agent orchestration: Define role settings, available tools, the knowledge base, and conversation strategy.
    • Tool calling: Connect to enterprise APIs through function or tool calling.
    • Knowledge base management: Handle document parsing, chunking, vectorization, and retrieval.
    • Workflows and multi-agent collaboration: Decompose complex tasks into serial, parallel, or routing flows.

    Support System

    The support system determines whether the platform can operate stably over the long term:

    • Unified identity: A three-level permission system for tenants, members, and API keys, with key rotation and revocation.
    • Monitoring and alerts: Set threshold alerts on call volume, error rate, latency, and compute utilization.
    • Logging and audit: Record the metadata needed to trace call chains, without retaining request content.
    • Content safety: Review both input and output. It is recommended that the platform expose metrics such as block rate, false-positive rate, appeal review, and threshold alerts, avoiding a one-size-fits-all security policy.
    • Billing and operations: Meter by call count or token usage, and provide tenants with a usage dashboard.

    Model Capability Evolution: Multimodal Code and High-Speed Inference

    One value of an MaaS platform is that enterprises can keep gaining capability upgrades without operating models themselves. The model matrix typically evolves along three directions:

    • Modality expansion: From plain text toward multimodal understanding of text, images, audio, and video.
    • Capability specialization: Dedicated foundation models for code generation, table understanding, image recognition, and more.
    • Service tiering: Beyond standard service, offering low-latency high-speed tiers.

    Native Multimodal Code Foundation Model

    A native multimodal code foundation model emphasizes introducing multimodal data during training, rather than bolting a separate vision module onto a text-only model. The “multimodal + code” combination is often used for:

    1. UI reconstruction: Input a design mockup screenshot, output front-end page code.
    2. Chart interpretation: Generate scripts, SQL, or configuration from screenshots, flowcharts, or architecture diagrams.
    3. Visualization assistance: Generate chart code from a natural-language description and validate the rendered result.

    Note that code capability should be validated by an automated toolchain—compilation, unit tests, static scanning—as the source of truth. Multimodal recognition can misread inputs, so generated code must pass human review and security scanning before it is merged into a production branch.

    High-Speed API and Inference Acceleration

    A high-speed API usually shares the same model capability as the standard tier; the difference lies mainly in service quality. Common inference acceleration techniques include:

    • Continuous batching: Dynamically adding new requests while a batch is executing, improving GPU utilization.
    • KV cache management: Paging and on-demand allocation of the cache to reduce memory fragmentation.
    • Quantization and cache compression: Representing model weights or the KV cache at lower precision to reduce memory footprint.
    • Speculative decoding: Using a small model or a draft mechanism to predict several tokens first, then having the large model verify them in a batch.
    • Execution-phase disaggregation: Deploying the compute-intensive prefill phase and the memory-bound decode phase separately.

    It is worth stressing that throughput optimization can increase memory usage, scheduling complexity, or tail-latency overhead, so costs and latency should be evaluated based on load-test results. The high-speed tier is better suited to interactive scenarios; it is recommended to load-test with your own business traffic first and then trade off latency against cost.

    Metric Meaning Where it matters
    TTFT Time to first token—from request sent to first output returned Chat, search Q&A
    TPOT Average generation time per output token (Time Per Output Token) Long-form generation
    Throughput Requests or tokens processed per unit time Batch jobs
    Error rate and availability Service stability Production

    Agent scenarios are especially sensitive to latency. A single agent task often goes through multiple “model inference—tool call—result回填” loops, and single-turn latency gets amplified. In this case, the high-speed API, streaming output, and parallel tool calls can be combined.

    Agent Development: From Tool Calling to On-Premises Deployment

    Agent Architecture

    A typical enterprise agent contains five modules:

    1. Inference engine: Handles planning, decision-making, and multi-turn dialogue.
    2. Tool layer: Connects to external capabilities such as enterprise APIs, databases, and search engines.
    3. Memory layer: Short-term conversation context and long-term knowledge base.
    4. Planning layer: Decomposes complex tasks into executable sub-task sequences.
    5. Guardrail layer: Handles permission checks, content safety, and output validation.

    Tool-Calling Example

    1. import json
    2. import os
    3. import urllib.request
    4. BASE_URL = os.environ["MAAS_ENDPOINT"] # Unified platform endpoint, injected via environment variable
    5. API_KEY = os.environ["MAAS_API_KEY"] # Never commit the key to source control
    6. MODEL = os.environ["MAAS_MODEL"] # Model identifier is chosen from the platform's model list
    7. TOOLS = [{
    8. "type": "function",
    9. "function": {
    10. "name": "query_inventory",
    11. "description": "Query inventory quantity by warehouse and SKU",
    12. "parameters": {
    13. "type": "object",
    14. "properties": {
    15. "warehouse": {"type": "string"},
    16. "sku": {"type": "string"},
    17. },
    18. "required": ["warehouse", "sku"],
    19. },
    20. },
    21. }]
    22. def call_model(messages, tools=None, stream=False):
    23. payload = {"model": MODEL, "messages": messages, "stream": stream}
    24. if tools:
    25. payload["tools"] = tools
    26. req = urllib.request.Request(
    27. f"{BASE_URL}/chat/completions",
    28. data=json.dumps(payload).encode("utf-8"),
    29. headers={
    30. "Content-Type": "application/json",
    31. "Authorization": f"Bearer {API_KEY}",
    32. },
    33. method="POST",
    34. )
    35. with urllib.request.urlopen(req, timeout=30) as resp:
    36. return json.load(resp)
    37. messages = [
    38. {"role": "system", "content": "You are an inventory assistant. Answer only from tool results, and state clearly when you cannot confirm."},
    39. {"role": "user", "content": "How much stock remains for SKU A-1024 in the East China warehouse?"},
    40. ]
    41. for _ in range(5): # Cap the maximum rounds to prevent tool-calling loops
    42. result = call_model(messages, tools=TOOLS)["choices"][0]["message"]
    43. messages.append(result)
    44. if not result.get("tool_calls"):
    45. print(result["content"])
    46. break
    47. for tc in result["tool_calls"]:
    48. args = json.loads(tc["function"]["arguments"])
    49. tool_result = {"warehouse": args["warehouse"], "sku": args["sku"], "stock": 0}
    50. messages.append({
    51. "role": "tool",
    52. "tool_call_id": tc["id"],
    53. "content": json.dumps(tool_result, ensure_ascii=False),
    54. })

    This example reflects three engineering points:

    • The key is injected via environment variables, not hard-coded in the code.
    • The loop rounds are capped to prevent the agent from falling into an infinite loop.
    • Tool return values must be computed by the server as trusted results; the model only selects and organizes them.

    Before integrating an agent, it is recommended to complete five security checks:

    1. Tool whitelist: Only allow the model to call registered tools.
    2. Parameter validation: Tool parameters must be validated server-side to prevent injection or privilege escalation.
    3. Timeout and retry: External tool calls must have timeout, retry, and circuit-breaking policies.
    4. Audit trail: Log the call trace, tool names, status codes, and durations—avoid logging sensitive content.
    5. Permission isolation: Split API keys by business line, limiting tool-call scope and data-access permissions.

    On-Premises and Private Deployment

    “Local invocation” usually takes two forms:

    • Private deployment: The model is deployed in the customer’s data center or a dedicated cloud environment. Whether data leaves the domain depends on the dedicated cloud boundary, contractual constraints, and audit requirements; this suits industry customers with clear data-boundary requirements.
    • Dedicated instance: The platform carves out an independent inference instance for a tenant, with isolated resources and context.

    Agents can connect through a unified API in both forms, and business code does not need to change with the deployment model.

    Evaluation and Observability

    Agent quality needs continuous evaluation. A three-tier evaluation system is recommended:

    1. End-to-end task evaluation: Measured by task completion rate and human-intervention rate.
    2. Single-turn tool-call evaluation: Focus on tool-selection accuracy and parameter correctness.
    3. Offline benchmark evaluation: Compare model versions on a fixed dataset to inform canary decisions.

    The call traces needed for evaluation can be saved by “de-identified, sampled retention.” This preserves analyzability without violating the data non-retention principle.

    Data Non-Retention and Privacy Protection

    Potential Risks of Data Retention

    Inputs and outputs of LLM services may contain three kinds of sensitive information:

    • Customer personal information: Names, phone numbers, ID numbers, and so on.
    • Enterprise trade secrets: Quotes, contract terms, source code, and so on.
    • Proprietary model data: Original internal knowledge base content, evaluation sets, and so on.

    If this content is retained long-term, it can create three risks: a breach of logs or storage leading to data leakage; retained data being used for training, causing information backflow; and unclear responsibility boundaries during cross-party flows that are hard to trace.

    Platform-Side Non-Retention Design

    The platform plans to ship a data content non-retention feature; its exact capabilities are subject to official documentation and pre-integration confirmation. This direction aligns with the industry-standard “zero data retention” practice, and its core principles can be summarized as “three don’ts”:

    1. Don’t persist to disk: Requests and responses are processed only in memory, never written to logging systems or object storage.
    2. Don’t retain: Context exists only within the current request’s lifecycle and is destroyed when the request ends; the chat API is designed to be stateless, and the platform does not store conversation content.
    3. Don’t train: Customer inputs and outputs do not enter model training data by default, unless the customer explicitly authorizes otherwise.

    At the same time, the platform still retains a small amount of operationally necessary metadata—for example, timestamps, model identifiers, token usage, status codes, and de-identified client identifiers. This content carries no business semantics; it is used for billing, rate limiting, and fault localization, and should have a defined retention period. The enterprise and the platform should also specify data classification, retention scope, and liability for breaches in the contract, and confirm that the platform holds the necessary compliance credentials.

    1. # Example privacy policy controlled by the caller
    2. privacy_policy:
    3. data_retention: none # Relies on the platform's "no content retention" capability; confirm it covers the models you use before integration
    4. log_prompt_and_completion: false
    5. pii_masking:
    6. enabled: true # Mask phone numbers, ID numbers, etc. on the client side before sending requests
    7. metadata_retention_days: 7 # Example value; adjust per contract, regulation, or platform rules
    8. training_opt_out: true # Opt out of training usage by default

    Enterprise-Side Safeguards

    Platform-side capability is only the baseline; enterprises also need defense in depth:

    • Client-side preprocessing: De-identify or replace personal information and secret-related data before sending requests.
    • Data classification: Split channels by sensitivity; route highly sensitive data through dedicated instances or private deployment.
    • Minimal input: Give the model only the information necessary to complete the task; avoid feeding entire tables or datasets into the context.
    • Permission isolation: Split API keys by business line and set quotas, limiting the blast radius if credentials leak.

    Common Misconceptions About Data Non-Retention

    A common misconception is that “non-retention means turning off logging.” In fact, fault troubleshooting, quality analysis, and billing reconciliation all depend on call information. The platform needs to strike a balance between “observability” and “no trace”:

    1. Trace correlation: Correlate requests with a trace ID, but do not log content.
    2. Fault localization: Only in clear fault scenarios, temporarily enable content sampling with tenant authorization, with an automatic expiry.
    3. Quality analysis: Tenants accumulate and de-identify data in their own environment; the platform does not store it on their behalf.
    4. Billing transparency: Provide usage details based on metadata, so enterprises can reconcile on their own.

    Privacy Protection by Deployment Model

    Deployment Data leaves the enterprise boundary Context retention Suitable scenarios Main trade-off
    Shared API + no retention Leaves the enterprise boundary Not retained by the platform General Q&A, content generation Low cost, but depends on platform commitments and compliance credentials
    Dedicated instance + no retention Enters the platform’s dedicated resource pool Tenant-isolated, not retained Shared across business lines Balanced isolation and cost
    Private / on-premises deployment Stays in-domain or controlled egress Managed by the enterprise Highly sensitive industries, intranet systems Stronger compliance, higher resource investment

    Integration Practice: From Selection to Launch

    A Five-Step Rollout

    1. Scenario assessment: Rank scenarios by “business value × implementation difficulty,” prioritizing tasks with clear rules and a human fallback.
    2. Model selection: Compare candidates horizontally against your own evaluation set, with metrics including task completion rate, latency, context length, multimodal support, and safety capability.
    3. API adaptation: Use a unified API as the integration layer, wrapping timeout, retry, fallback, and idempotency logic.
    4. Agent orchestration: Define tools, the knowledge base, and guardrails; get the main flow working first, then enhance it step by step.
    5. Canary and operations: Ramp traffic by percentage, continuously monitoring cost, latency, error rate, and content safety metrics.

    What to Prepare Before Integration

    It is recommended to prepare the following before integration:

    • Scenario description: Target users, task goals, input format, output format, and the human fallback plan.
    • Evaluation set template: At least 20 typical samples, 5 edge-case samples, and 5 safety samples.
    • Metric thresholds: Time to first token, task completion rate, error rate, block rate, and cost per task.
    • Data classification list: Which fields can be de-identified, which data must not leave the boundary, and which scenarios require dedicated instances.
    • Permission matrix: The mapping among applications, tenants, members, API keys, and tool-call permissions.

    How to Estimate Cost

    It is recommended to estimate monthly cost with the following formula:

    1. Monthly cost = request volume × avg input tokens × input price
    2. + request volume × avg output tokens × output price
    3. + instance minimum fee
    4. + add-on fees (storage, retrieval, vectorization, etc.)
    5. + operational fees (content safety, audit, logging, etc.)

    Illustrative example: assume 1,000,000 requests per month, an average of 800 input tokens and 300 output tokens, an input price of 0.002 per 1K tokens, and an output price of 0.006 per 1K tokens. The model-call cost is:

    1. 1,000,000 × 0.8K tokens × 0.002 + 1,000,000 × 0.3K tokens × 0.006
    2. = 1,600 + 1,800 = 3,400

    If you use the high-speed API, dedicated instances, or an additional toolchain, add the instance minimum fee and add-on service fees on top.

    Four Details for Production-Grade Calls

    1. import random
    2. import time
    3. from urllib.error import URLError
    4. def call_fallback_model():
    5. return {
    6. "choices": [{
    7. "message": {
    8. "role": "assistant",
    9. "content": "The service is busy. Please retry later or contact a human agent."
    10. }
    11. }]
    12. }
    13. def call_with_retry(call_model, retries=3):
    14. for attempt in range(retries):
    15. try:
    16. return call_model()
    17. except (URLError, TimeoutError):
    18. time.sleep(min(2 ** attempt, 8) + random.random())
    19. return call_fallback_model()
    • Streaming output: Lowers time to first token and improves the interactive experience.
    • Idempotency control: Attach a unique ID to each request to prevent duplicate billing or duplicate execution caused by timeout retries.
    • Fallback strategy: When the primary model is unavailable, automatically switch to a backup model, a rules engine, or a human fallback.
    • Cost and quota: Set budget thresholds on token consumption and alert in advance.

    Common Failures and Fixes

    Symptom Common cause Suggested fix
    Request timeout Model overload, long-form generation, network jitter Set timeouts, retries, fallbacks, and streaming output
    Cost spike Overly long context, excessive output, no budget threshold Cap max tokens, compress context, set alerts
    Tool-call failure Wrong parameters, insufficient permissions, API errors Server-side parameter validation, whitelist, retry and audit
    Content safety block Sensitive words, prompt injection, business false positives Tiered blocking, appeal review, refine prompts and context
    Unstable model answers Unclear prompts, insufficient knowledge, model version differences Regression with an evaluation set, canary release, version rollback

    Typical Use Cases

    • Smartphones and smart devices: Use multimodal understanding to process images and documents, use agent orchestration to complete cross-app tasks, and use the high-speed service to reduce voice-interaction latency.
    • Smart transportation: In traffic signal optimization, incident detection, and travel-service Q&A, combine vision models with an industry knowledge base to provide decision support and information services for managers and the public.
    • Government services and knowledge Q&A: Build policy Q&A and procedure-guide assistants on retrieval-augmented generation, and use non-retention capabilities to reduce the risk of exposing citizens’ personal information.
    • Industrial manufacturing: Device manual Q&A, ticket classification, and visual quality-inspection assistance.
    • Software development: In code generation, unit-test generation, and UI reconstruction, combine multimodal coding models with an automated toolchain to improve R&D efficiency.

    Challenges and Outlook

    Building an MaaS platform still faces four challenges:

    • Fast model iteration: Reduce upgrade risk through evaluation systems and canary mechanisms.
    • Complex ecosystem collaboration: Unified interface specifications and data standards reduce redundant build-out among resident companies.
    • High security and compliance requirements: Non-retention, private deployment, and content safety must be carefully balanced between cost and compliance.
    • Operational sustainability: After subsidies taper off, the platform must build self-sustaining capability from real scenario demand.

    Looking ahead, MaaS platforms will show three trends:

    1. Multimodal and agent convergence: Platforms move from “providing models” to “providing directly runnable intelligent applications.”
    2. Hybrid deployment: Public calls, dedicated instances, and private deployment are combined in tiers by data sensitivity.
    3. Privacy protection by default: Capabilities such as data non-retention shift from “optional” to “default,” becoming a baseline requirement for platform competitiveness.

    For enterprises and developers, whoever builds capability earlier in model selection, agent engineering, and security governance will convert large models into measurable business value faster.