Building an Enterprise MaaS Platform: Architecture, Vertical Model Ecosystems, and Production Practices
This guide walks through how to build an enterprise-grade MaaS (Model-as-a-Service) platform and cultivate a vertical model ecosystem. It breaks the platform into a "four layers plus one support system" architecture—infrastructure, model, model-serving, and agent/application layers, plus a cross-cutting support system—and explains how to put model routing, inference acceleration, and agent development into practice. It also covers data non-retention and privacy protection, the trade-offs between private and dedicated-instance deployment, cost estimation, evaluation, and the security checks to complete before integration. Written for developers and enterprise architects, it shows how to turn large language models into stable, controllable, and measurable business value.
Conclusion First: What Enterprises Need Is a Sustainable Platform, Not a Single Model
LLM adoption is shifting from “calling one model” to “operating a service system.” What enterprises really care about is rarely how large the model parameters are, but whether a model can be integrated reliably, called on demand, versioned in a controlled way, costed predictably, and connected securely to business systems.
The core value of MaaS (Model-as-a-Service) is packaging complex concerns—training, inference, operations, security, and metering—into a platform, so enterprises can focus on business scenarios and their data assets. In a public–private partnership, an MaaS platform can also support regional industrial upgrading: the public side provides scenarios and resources, the private side provides technology and models, and companies across the ecosystem collaborate around compute, data, models, and applications.
Using a regional public–private vertical-model ecosystem hub project as a reference, this article is aimed at developers and enterprise architects and answers four questions:
- How should an enterprise MaaS platform be layered?
- How do model routing, inference acceleration, and agent development work in practice?
- How do you choose between data non-retention, private deployment, and dedicated instances?
- Before integration, how do you handle selection, cost estimation, evaluation, and security readiness?
Background and Platform Positioning: How a Public–Private Model Forms an Ecosystem Loop
A regional MaaS platform is typically built through a partnership between a state-backed regional industrial investment operator and an LLM development company, positioned as the core carrier of a vertical-model ecosystem hub. The point of this model is not to attract investment for a single project, but to lower the barrier for small and mid-sized teams to use large models through a shared technical foundation.
A public–private model offers three advantages:
- Demand-side connection: The public side holds real scenarios such as city governance, public services, and traffic management, so the platform can support use cases like government Q&A, city operations, and hotline assistance.
- Supply-side clustering: As a shared technical foundation, the platform attracts algorithm teams, data service providers, application developers, and system integrators to collaborate within one ecosystem.
- Factor-side cost reduction: Coordinating compute facilities, data governance resources, and policy resources can significantly lower the early-stage investment required for companies moving in.
As disclosed in a phase update after launch, dozens of upstream and downstream companies have moved in, covering compute infrastructure, data labeling and governance, model R&D and fine-tuning, and industry application development—forming an initial “compute–data–model–application” loop. The cluster effect shows up in three ways:
- Technology reuse: Underlying compute and model services are shared across teams, so applications do not rebuild infrastructure from scratch.
- Knowledge flow: Model teams and industry teams exchange ideas frequently within the same ecosystem, so vertical models iterate faster.
- Supply–demand matching: Industry application companies become consumers of model capabilities, while model teams receive feedback from compliant scenarios.
The platform is not a single API forwarding gateway, but a unified service entry point for three roles:
| Role | Core needs | Platform capabilities |
|---|---|---|
| Model provider | Fast onboarding, stable service, version control | Model hosting, inference optimization, elastic scaling, metering and billing |
| Developer / tenant | Low-cost access, fast app and agent building | Unified API, agent development framework, evaluation and debugging tools |
| End user | Secure, trustworthy, reliable AI services | Content safety, privacy protection, service-level guarantees |
From Policy Direction to Developer Action Items
In recent years, many governments have introduced policies supporting large models and AI agents, and MaaS platforms are increasingly treated as the key infrastructure for putting those policies into practice. The policy logic can be summarized in three layers:
- Factor supply: Through compute coordination, model openness, and data governance, lower the barrier for small and mid-sized companies to access AI capabilities.
- Scenario pull: Use “AI-plus” initiatives to move key industries from pilot validation toward scaled replication.
- Ecosystem governance: Use a public platform as the vehicle to unify content safety, data compliance, and service quality standards.
For developers, policy direction ultimately translates into three action items:
- Predictable cost: Prefer platform-based, at-scale services, and set up monitoring for call volume and budget.
- Lower access barrier: Unified interfaces, unified authentication, and unified metering reduce redundant build-out.
- Clear compliance responsibility: Before integration, confirm data classification, retention scope, audit obligations, and responsibility boundaries between the platform and the enterprise.
Overall Platform Architecture: Four Layers and One Support System
A mature enterprise MaaS platform can be abstracted as “four layers and one support system”: an infrastructure layer, a model layer, a model serving layer, and an agent and application development layer, plus a support system that runs across all layers.
┌──────────────────────────────────────────────────────────────────┐│ Support System: Identity | Monitoring & Alerts | Logging & Audit | Content Safety | Billing │├──────────────────────────────────────────────────────────────────┤│ Agent & App Dev Layer: Agent Orchestration | Tool Calling | Knowledge Base | Workflow │├──────────────────────────────────────────────────────────────────┤│ Model Serving Layer: Unified API | Model Routing | Inference Acceleration | Fine-tuning Hosting | Versioning/Canary │├──────────────────────────────────────────────────────────────────┤│ Model Layer: General Chat | Long Context | Multimodal | Code | Vertical Models │├──────────────────────────────────────────────────────────────────┤│ Infrastructure Layer: Heterogeneous Compute | Container Platform | Object Storage | Message Queue │└──────────────────────────────────────────────────────────────────┘
Infrastructure Layer
The infrastructure layer provides stable compute, storage, and scheduling:
- Heterogeneous compute: Training typically relies on GPUs, while inference also uses a range of accelerators. The platform needs to orchestrate resources through a container platform and enforce per-tenant quota isolation.
- Storage and transport: Model weights, knowledge base documents, and datasets are usually stored in object storage; high-concurrency requests rely on message queues to smooth out spikes and troughs.
- Elastic scaling: Scale up and down automatically based on request queue length and GPU utilization, avoiding peak-time queue timeouts and low-valley idle resources.
Model Layer
The model layer is the source of platform capability and typically includes:
- General foundation models: Covering everyday conversation, text understanding, and generation; some support long context and suit knowledge Q&A and document processing.
- Multimodal models: Understanding text, images, audio, and video, suitable for document parsing, image moderation, and multimodal interaction.
- Code and vertical models: Aimed at software development and specific business scenarios; domain knowledge can be injected through instruction fine-tuning and preference alignment.
Model Serving Layer: Unified API and Model Routing
The model serving layer turns “models” into “services.” Its core responsibilities include a unified API, inference optimization, fine-tuning hosting, version management, and model routing.
When choosing an MaaS platform, model routing should not look only at model names; it should be driven by business constraints. Common routing dimensions include latency, cost, context length, multimodal support, compliance requirements, tool-calling capability, and availability.
| Priority | Business scenario | Routing recommendation |
|---|---|---|
| Latency-sensitive | Voice assistants, real-time Q&A | Prefer high-speed API, low-latency models, streaming output |
| Cost-sensitive | Batch summarization, ticket classification | Prefer low-cost models, cache reuse, batch processing |
| Long-document | Contract review, manual Q&A | Prefer long-context models with retrieval augmentation |
| Multimodal | Design-to-code, image moderation | Prefer multimodal or multimodal-code models |
| Highly sensitive | Government, healthcare, finance | Prefer dedicated instances or private deployment |
Agent and Application Development Layer
A model is only the “brain.” To deliver usable capability, you still need to build perception, memory, planning, and action on top of it. An agent development framework typically provides:
- Agent orchestration: Define role settings, available tools, the knowledge base, and conversation strategy.
- Tool calling: Connect to enterprise APIs through function or tool calling.
- Knowledge base management: Handle document parsing, chunking, vectorization, and retrieval.
- Workflows and multi-agent collaboration: Decompose complex tasks into serial, parallel, or routing flows.
Support System
The support system determines whether the platform can operate stably over the long term:
- Unified identity: A three-level permission system for tenants, members, and API keys, with key rotation and revocation.
- Monitoring and alerts: Set threshold alerts on call volume, error rate, latency, and compute utilization.
- Logging and audit: Record the metadata needed to trace call chains, without retaining request content.
- Content safety: Review both input and output. It is recommended that the platform expose metrics such as block rate, false-positive rate, appeal review, and threshold alerts, avoiding a one-size-fits-all security policy.
- Billing and operations: Meter by call count or token usage, and provide tenants with a usage dashboard.
Model Capability Evolution: Multimodal Code and High-Speed Inference
One value of an MaaS platform is that enterprises can keep gaining capability upgrades without operating models themselves. The model matrix typically evolves along three directions:
- Modality expansion: From plain text toward multimodal understanding of text, images, audio, and video.
- Capability specialization: Dedicated foundation models for code generation, table understanding, image recognition, and more.
- Service tiering: Beyond standard service, offering low-latency high-speed tiers.
Native Multimodal Code Foundation Model
A native multimodal code foundation model emphasizes introducing multimodal data during training, rather than bolting a separate vision module onto a text-only model. The “multimodal + code” combination is often used for:
- UI reconstruction: Input a design mockup screenshot, output front-end page code.
- Chart interpretation: Generate scripts, SQL, or configuration from screenshots, flowcharts, or architecture diagrams.
- Visualization assistance: Generate chart code from a natural-language description and validate the rendered result.
Note that code capability should be validated by an automated toolchain—compilation, unit tests, static scanning—as the source of truth. Multimodal recognition can misread inputs, so generated code must pass human review and security scanning before it is merged into a production branch.
High-Speed API and Inference Acceleration
A high-speed API usually shares the same model capability as the standard tier; the difference lies mainly in service quality. Common inference acceleration techniques include:
- Continuous batching: Dynamically adding new requests while a batch is executing, improving GPU utilization.
- KV cache management: Paging and on-demand allocation of the cache to reduce memory fragmentation.
- Quantization and cache compression: Representing model weights or the KV cache at lower precision to reduce memory footprint.
- Speculative decoding: Using a small model or a draft mechanism to predict several tokens first, then having the large model verify them in a batch.
- Execution-phase disaggregation: Deploying the compute-intensive prefill phase and the memory-bound decode phase separately.
It is worth stressing that throughput optimization can increase memory usage, scheduling complexity, or tail-latency overhead, so costs and latency should be evaluated based on load-test results. The high-speed tier is better suited to interactive scenarios; it is recommended to load-test with your own business traffic first and then trade off latency against cost.
| Metric | Meaning | Where it matters |
|---|---|---|
| TTFT | Time to first token—from request sent to first output returned | Chat, search Q&A |
| TPOT | Average generation time per output token (Time Per Output Token) | Long-form generation |
| Throughput | Requests or tokens processed per unit time | Batch jobs |
| Error rate and availability | Service stability | Production |
Agent scenarios are especially sensitive to latency. A single agent task often goes through multiple “model inference—tool call—result回填” loops, and single-turn latency gets amplified. In this case, the high-speed API, streaming output, and parallel tool calls can be combined.
Agent Development: From Tool Calling to On-Premises Deployment
Agent Architecture
A typical enterprise agent contains five modules:
- Inference engine: Handles planning, decision-making, and multi-turn dialogue.
- Tool layer: Connects to external capabilities such as enterprise APIs, databases, and search engines.
- Memory layer: Short-term conversation context and long-term knowledge base.
- Planning layer: Decomposes complex tasks into executable sub-task sequences.
- Guardrail layer: Handles permission checks, content safety, and output validation.
Tool-Calling Example
import jsonimport osimport urllib.requestBASE_URL = os.environ["MAAS_ENDPOINT"] # Unified platform endpoint, injected via environment variableAPI_KEY = os.environ["MAAS_API_KEY"] # Never commit the key to source controlMODEL = os.environ["MAAS_MODEL"] # Model identifier is chosen from the platform's model listTOOLS = [{"type": "function","function": {"name": "query_inventory","description": "Query inventory quantity by warehouse and SKU","parameters": {"type": "object","properties": {"warehouse": {"type": "string"},"sku": {"type": "string"},},"required": ["warehouse", "sku"],},},}]def call_model(messages, tools=None, stream=False):payload = {"model": MODEL, "messages": messages, "stream": stream}if tools:payload["tools"] = toolsreq = urllib.request.Request(f"{BASE_URL}/chat/completions",data=json.dumps(payload).encode("utf-8"),headers={"Content-Type": "application/json","Authorization": f"Bearer {API_KEY}",},method="POST",)with urllib.request.urlopen(req, timeout=30) as resp:return json.load(resp)messages = [{"role": "system", "content": "You are an inventory assistant. Answer only from tool results, and state clearly when you cannot confirm."},{"role": "user", "content": "How much stock remains for SKU A-1024 in the East China warehouse?"},]for _ in range(5): # Cap the maximum rounds to prevent tool-calling loopsresult = call_model(messages, tools=TOOLS)["choices"][0]["message"]messages.append(result)if not result.get("tool_calls"):print(result["content"])breakfor tc in result["tool_calls"]:args = json.loads(tc["function"]["arguments"])tool_result = {"warehouse": args["warehouse"], "sku": args["sku"], "stock": 0}messages.append({"role": "tool","tool_call_id": tc["id"],"content": json.dumps(tool_result, ensure_ascii=False),})
This example reflects three engineering points:
- The key is injected via environment variables, not hard-coded in the code.
- The loop rounds are capped to prevent the agent from falling into an infinite loop.
- Tool return values must be computed by the server as trusted results; the model only selects and organizes them.
Before integrating an agent, it is recommended to complete five security checks:
- Tool whitelist: Only allow the model to call registered tools.
- Parameter validation: Tool parameters must be validated server-side to prevent injection or privilege escalation.
- Timeout and retry: External tool calls must have timeout, retry, and circuit-breaking policies.
- Audit trail: Log the call trace, tool names, status codes, and durations—avoid logging sensitive content.
- Permission isolation: Split API keys by business line, limiting tool-call scope and data-access permissions.
On-Premises and Private Deployment
“Local invocation” usually takes two forms:
- Private deployment: The model is deployed in the customer’s data center or a dedicated cloud environment. Whether data leaves the domain depends on the dedicated cloud boundary, contractual constraints, and audit requirements; this suits industry customers with clear data-boundary requirements.
- Dedicated instance: The platform carves out an independent inference instance for a tenant, with isolated resources and context.
Agents can connect through a unified API in both forms, and business code does not need to change with the deployment model.
Evaluation and Observability
Agent quality needs continuous evaluation. A three-tier evaluation system is recommended:
- End-to-end task evaluation: Measured by task completion rate and human-intervention rate.
- Single-turn tool-call evaluation: Focus on tool-selection accuracy and parameter correctness.
- Offline benchmark evaluation: Compare model versions on a fixed dataset to inform canary decisions.
The call traces needed for evaluation can be saved by “de-identified, sampled retention.” This preserves analyzability without violating the data non-retention principle.
Data Non-Retention and Privacy Protection
Potential Risks of Data Retention
Inputs and outputs of LLM services may contain three kinds of sensitive information:
- Customer personal information: Names, phone numbers, ID numbers, and so on.
- Enterprise trade secrets: Quotes, contract terms, source code, and so on.
- Proprietary model data: Original internal knowledge base content, evaluation sets, and so on.
If this content is retained long-term, it can create three risks: a breach of logs or storage leading to data leakage; retained data being used for training, causing information backflow; and unclear responsibility boundaries during cross-party flows that are hard to trace.
Platform-Side Non-Retention Design
The platform plans to ship a data content non-retention feature; its exact capabilities are subject to official documentation and pre-integration confirmation. This direction aligns with the industry-standard “zero data retention” practice, and its core principles can be summarized as “three don’ts”:
- Don’t persist to disk: Requests and responses are processed only in memory, never written to logging systems or object storage.
- Don’t retain: Context exists only within the current request’s lifecycle and is destroyed when the request ends; the chat API is designed to be stateless, and the platform does not store conversation content.
- Don’t train: Customer inputs and outputs do not enter model training data by default, unless the customer explicitly authorizes otherwise.
At the same time, the platform still retains a small amount of operationally necessary metadata—for example, timestamps, model identifiers, token usage, status codes, and de-identified client identifiers. This content carries no business semantics; it is used for billing, rate limiting, and fault localization, and should have a defined retention period. The enterprise and the platform should also specify data classification, retention scope, and liability for breaches in the contract, and confirm that the platform holds the necessary compliance credentials.
# Example privacy policy controlled by the callerprivacy_policy:data_retention: none # Relies on the platform's "no content retention" capability; confirm it covers the models you use before integrationlog_prompt_and_completion: falsepii_masking:enabled: true # Mask phone numbers, ID numbers, etc. on the client side before sending requestsmetadata_retention_days: 7 # Example value; adjust per contract, regulation, or platform rulestraining_opt_out: true # Opt out of training usage by default
Enterprise-Side Safeguards
Platform-side capability is only the baseline; enterprises also need defense in depth:
- Client-side preprocessing: De-identify or replace personal information and secret-related data before sending requests.
- Data classification: Split channels by sensitivity; route highly sensitive data through dedicated instances or private deployment.
- Minimal input: Give the model only the information necessary to complete the task; avoid feeding entire tables or datasets into the context.
- Permission isolation: Split API keys by business line and set quotas, limiting the blast radius if credentials leak.
Common Misconceptions About Data Non-Retention
A common misconception is that “non-retention means turning off logging.” In fact, fault troubleshooting, quality analysis, and billing reconciliation all depend on call information. The platform needs to strike a balance between “observability” and “no trace”:
- Trace correlation: Correlate requests with a trace ID, but do not log content.
- Fault localization: Only in clear fault scenarios, temporarily enable content sampling with tenant authorization, with an automatic expiry.
- Quality analysis: Tenants accumulate and de-identify data in their own environment; the platform does not store it on their behalf.
- Billing transparency: Provide usage details based on metadata, so enterprises can reconcile on their own.
Privacy Protection by Deployment Model
| Deployment | Data leaves the enterprise boundary | Context retention | Suitable scenarios | Main trade-off |
|---|---|---|---|---|
| Shared API + no retention | Leaves the enterprise boundary | Not retained by the platform | General Q&A, content generation | Low cost, but depends on platform commitments and compliance credentials |
| Dedicated instance + no retention | Enters the platform’s dedicated resource pool | Tenant-isolated, not retained | Shared across business lines | Balanced isolation and cost |
| Private / on-premises deployment | Stays in-domain or controlled egress | Managed by the enterprise | Highly sensitive industries, intranet systems | Stronger compliance, higher resource investment |
Integration Practice: From Selection to Launch
A Five-Step Rollout
- Scenario assessment: Rank scenarios by “business value × implementation difficulty,” prioritizing tasks with clear rules and a human fallback.
- Model selection: Compare candidates horizontally against your own evaluation set, with metrics including task completion rate, latency, context length, multimodal support, and safety capability.
- API adaptation: Use a unified API as the integration layer, wrapping timeout, retry, fallback, and idempotency logic.
- Agent orchestration: Define tools, the knowledge base, and guardrails; get the main flow working first, then enhance it step by step.
- Canary and operations: Ramp traffic by percentage, continuously monitoring cost, latency, error rate, and content safety metrics.
What to Prepare Before Integration
It is recommended to prepare the following before integration:
- Scenario description: Target users, task goals, input format, output format, and the human fallback plan.
- Evaluation set template: At least 20 typical samples, 5 edge-case samples, and 5 safety samples.
- Metric thresholds: Time to first token, task completion rate, error rate, block rate, and cost per task.
- Data classification list: Which fields can be de-identified, which data must not leave the boundary, and which scenarios require dedicated instances.
- Permission matrix: The mapping among applications, tenants, members, API keys, and tool-call permissions.
How to Estimate Cost
It is recommended to estimate monthly cost with the following formula:
Monthly cost = request volume × avg input tokens × input price+ request volume × avg output tokens × output price+ instance minimum fee+ add-on fees (storage, retrieval, vectorization, etc.)+ operational fees (content safety, audit, logging, etc.)
Illustrative example: assume 1,000,000 requests per month, an average of 800 input tokens and 300 output tokens, an input price of 0.002 per 1K tokens, and an output price of 0.006 per 1K tokens. The model-call cost is:
1,000,000 × 0.8K tokens × 0.002 + 1,000,000 × 0.3K tokens × 0.006= 1,600 + 1,800 = 3,400
If you use the high-speed API, dedicated instances, or an additional toolchain, add the instance minimum fee and add-on service fees on top.
Four Details for Production-Grade Calls
import randomimport timefrom urllib.error import URLErrordef call_fallback_model():return {"choices": [{"message": {"role": "assistant","content": "The service is busy. Please retry later or contact a human agent."}}]}def call_with_retry(call_model, retries=3):for attempt in range(retries):try:return call_model()except (URLError, TimeoutError):time.sleep(min(2 ** attempt, 8) + random.random())return call_fallback_model()
- Streaming output: Lowers time to first token and improves the interactive experience.
- Idempotency control: Attach a unique ID to each request to prevent duplicate billing or duplicate execution caused by timeout retries.
- Fallback strategy: When the primary model is unavailable, automatically switch to a backup model, a rules engine, or a human fallback.
- Cost and quota: Set budget thresholds on token consumption and alert in advance.
Common Failures and Fixes
| Symptom | Common cause | Suggested fix |
|---|---|---|
| Request timeout | Model overload, long-form generation, network jitter | Set timeouts, retries, fallbacks, and streaming output |
| Cost spike | Overly long context, excessive output, no budget threshold | Cap max tokens, compress context, set alerts |
| Tool-call failure | Wrong parameters, insufficient permissions, API errors | Server-side parameter validation, whitelist, retry and audit |
| Content safety block | Sensitive words, prompt injection, business false positives | Tiered blocking, appeal review, refine prompts and context |
| Unstable model answers | Unclear prompts, insufficient knowledge, model version differences | Regression with an evaluation set, canary release, version rollback |
Typical Use Cases
- Smartphones and smart devices: Use multimodal understanding to process images and documents, use agent orchestration to complete cross-app tasks, and use the high-speed service to reduce voice-interaction latency.
- Smart transportation: In traffic signal optimization, incident detection, and travel-service Q&A, combine vision models with an industry knowledge base to provide decision support and information services for managers and the public.
- Government services and knowledge Q&A: Build policy Q&A and procedure-guide assistants on retrieval-augmented generation, and use non-retention capabilities to reduce the risk of exposing citizens’ personal information.
- Industrial manufacturing: Device manual Q&A, ticket classification, and visual quality-inspection assistance.
- Software development: In code generation, unit-test generation, and UI reconstruction, combine multimodal coding models with an automated toolchain to improve R&D efficiency.
Challenges and Outlook
Building an MaaS platform still faces four challenges:
- Fast model iteration: Reduce upgrade risk through evaluation systems and canary mechanisms.
- Complex ecosystem collaboration: Unified interface specifications and data standards reduce redundant build-out among resident companies.
- High security and compliance requirements: Non-retention, private deployment, and content safety must be carefully balanced between cost and compliance.
- Operational sustainability: After subsidies taper off, the platform must build self-sustaining capability from real scenario demand.
Looking ahead, MaaS platforms will show three trends:
- Multimodal and agent convergence: Platforms move from “providing models” to “providing directly runnable intelligent applications.”
- Hybrid deployment: Public calls, dedicated instances, and private deployment are combined in tiers by data sensitivity.
- Privacy protection by default: Capabilities such as data non-retention shift from “optional” to “default,” becoming a baseline requirement for platform competitiveness.
For enterprises and developers, whoever builds capability earlier in model selection, agent engineering, and security governance will convert large models into measurable business value faster.
