MaaS (Model as a Service) Architecture: A Complete Technical Guide to LLM Serving

    This article explains how Model as a Service (MaaS) platforms turn large language model capabilities into stable, metered, and governable infrastructure. It walks through the five-layer reference architecture, the unified API gateway (authentication, quotas, rate limiting, and tenant isolation), inference engine optimization (continuous batching, parallelism strategies, prefill/decode disaggregation, caching, and compression), elastic scaling signals and cold-start mitigation, reliability and security practices, observability and cost governance, and a phased implementation roadmap. Engineers will learn how to select, integrate, and operate MaaS components while balancing latency, throughput, elasticity, and cost.

    Background: Why Large Models Need to Be “Serviceized”

    The value of large models is ultimately realized through applications. In the past, most enterprises adopted a “project-based” model when using large model capabilities: procuring compute, deploying the model, and then building custom adaptations for each scenario. This model had long cycles, high costs, and was difficult to reuse. As foundation model capabilities continue to strengthen, the industry has gradually formed a new delivery model: packaging model capabilities as standardized interfaces, with a platform hosting compute, inference engines, and operations centrally, while users pay by call volume or token usage and obtain intelligence on demand. This model is commonly called MaaS (Model as a Service).

    From an architectural perspective, MaaS is essentially a “distributed online serving system centered on inference.” It plays at least four roles simultaneously:

    1. Manager of compute resource pools: organizing scattered heterogeneous accelerators into a schedulable resource pool;
    2. Provider of model services: responsible for model loading, batch scheduling, streaming output, and version management;
    3. Bearer of the developer experience: lowering integration barriers through unified interfaces, SDKs, and documentation;
    4. Executor of platform operations: handling authentication, quotas, metering, billing, monitoring, and auditing.

    Understanding these four roles is the prerequisite for designing a good MaaS platform.

    Architecture Overview: The Five Layers of a MaaS Platform

    A mature MaaS platform can typically be divided into five layers, from bottom to top:

    Layer Core responsibility Key design considerations
    Infrastructure Heterogeneous compute, high-performance networking, storage Resource pooling, topology-aware scheduling, accelerated weight distribution
    Inference engine Model loading and inference execution Continuous batching, KV cache management, parallelism strategies
    Service orchestration Multi-model hosting, elastic scaling, traffic governance Scaling signals, canary releases, multi-AZ disaster recovery
    Gateway Unified interface, authentication and quotas Protocol compatibility, streaming transport, rate limiting and tenant isolation
    Developer services Metering and billing, observability, content safety Usage transparency, end-to-end tracing, audit trails

    Among these, the infrastructure and inference engine layers determine the platform’s performance ceiling and cost floor; the service orchestration and gateway layers determine the platform’s stability and scalability; and the developer services layer determines the platform’s usability and trustworthiness. A shortcoming in any layer will be amplified as call volume grows rapidly.

    The Unified Gateway: The First Threshold of a MaaS Platform

    The gateway is the platform’s only external entry point. Once a request enters the platform, it must complete protocol parsing, authentication, quota checks, routing, streaming forwarding, and usage accounting in sequence. If the gateway is poorly designed, all subsequent optimizations will be blocked here.

    Interface Design and Protocol Compatibility

    The interface is the “contract” between the platform and developers. A well-designed MaaS interface typically has the following characteristics:

    • Clear semantics: request parameters cover message context, sampling parameters (such as temperature and maximum generation length), and output format constraints;
    • Protocol compatibility: remaining compatible with mainstream online inference protocols, so developers only need to adjust a few configurations to migrate;
    • Streaming first: supporting streaming responses based on server-sent events, so users can see output as soon as the first token is returned, greatly reducing the sense of waiting;
    • Clear error semantics: using standard status codes to distinguish authentication failures, parameter errors, rate limiting, and model errors, making it easier for clients to implement differentiated retries.

    Authentication, Quotas, and Tenant Isolation

    Multi-tenancy is the norm for MaaS platforms. Different tenants vary greatly in call volume, SLA, and business importance. The platform needs to do three things at the entry point:

    1. Authentication: verify call credentials to identify the caller’s identity and application ownership;
    2. Quota checks: check remaining capacity along the “tenant × model × time window” dimension;
    3. Priority determination: reserve capacity or grant higher scheduling weight for higher-tier tenants.

    Rate Limiting Algorithms and Request Priority

    Common rate limiting algorithms each have their applicable scenarios:

    • Fixed window counter: simple to implement, but a doubled traffic spike may occur at the moment the window switches;
    • Sliding window: smoother statistics, at the cost of maintaining additional state;
    • Token bucket: allows a certain degree of burst traffic, making it more suitable for online services like large models where a single request takes a long time and traffic is spiky.

    On a shared inference resource pool, a single tenant’s burst traffic may crowd out other tenants’ capacity. Fair scheduling can be achieved by combining multi-level queues: first allocate time slices per tenant, then schedule by arrival order within each tenant. When high-priority requests arrive, they can be inserted at the front of the queue.

    1. # Illustration: gateway rate limiting + multi-level queue scheduling (pseudocode to explain the idea)
    2. def handle_request(req, tenant):
    3. # 1. Authentication: verify credentials and identify the tenant; return 401 on failure
    4. if not verify_credential(req):
    5. return error(401, "UNAUTHORIZED")
    6. # 2. Quota: token bucket decrements by "tenant + model"; return 429 when no token is available
    7. if not token_bucket(tenant.id, req.model).try_acquire():
    8. return error(429, "RATE_LIMITED")
    9. # 3. Priority: requests from higher-tier tenants (e.g., enterprise plans) enter the queue first
    10. p = tenant.tier if tenant.tier else NORMAL
    11. queue = scheduler.priority_queue(p)
    12. # When queue backlog exceeds the threshold, shed or queue-protect to avoid an avalanche
    13. if queue.depth() > queue.capacity() * 0.9:
    14. scheduler.shed_or_queue(req, tenant)
    15. queue.push(req)
    16. return stream_forward(req) # Forward to the client while receiving model output

    Observability Instrumentation

    While forwarding requests, the gateway should record the request ID, tenant, model, input and output token counts, time to first token, total latency, and response code. This data is both the basis for metering and billing and the foundation for later capacity planning and fault localization.

    The Inference Engine: Dual Optimization of Latency and Throughput

    Large model inference is the core of a MaaS platform. It differs both from traditional web services and from model training, mainly in three aspects:

    • Large per-request computation: even generating a few dozen characters requires billions of floating-point operations;
    • GPU memory is a hard constraint: both model weights and caches must reside in GPU memory;
    • Obvious memory-access bottleneck: in the single-request decoding stage, every token generated requires reading the full model weights. The bottleneck is often memory bandwidth, not compute peak.

    Therefore, the goal of inference optimization is not simply to “compute faster,” but to achieve both lower latency and higher throughput under limited GPU memory.

    Core Metrics for Measuring Inference Performance

    Metric Meaning Main influencing factors
    TTFT (time to first token) Time from sending the request to receiving the first token Input length, queue time, scheduling strategy
    TPOT (time per output token) Average time to generate one token in the decode stage Batch size, memory bandwidth, context length
    Throughput (Output TPS) Total tokens generated by the system per unit time Batching strategy, concurrency scale, parallelism configuration
    GPU memory usage Sum of weight and cache occupancy Model size, concurrency, context length

    TTFT is mainly determined by the Prefill stage, which processes the entire input in parallel and is compute-intensive. TPOT is mainly determined by the Decode stage, which generates tokens one by one and is memory-access-intensive. The two stages have different characteristics but share the same accelerator and the same cache; their mutual interference is one of the main sources of latency jitter.

    Continuous Batching: The Key to Higher GPU Utilization

    Traditional static batching requires all requests in a batch to start and finish at the same time. Because each request’s output length differs, requests that finish early can only wait idly for the whole batch to complete, resulting in low resource utilization.

    Continuous Batching refines the scheduling granularity to each decoding step: once a request finishes generating (or reaches its length limit), its slot is immediately released and filled by a newly arrived request. The inference engine continuously maintains a dynamic batch of “in-flight requests” and, together with paged cache management that allocates GPU memory blocks on demand, can significantly increase token output per unit of compute.

    Parallelism Strategies

    • Data parallelism: deploy multiple replicas of the same model, with the gateway load balancing across them; suitable for throughput-first scenarios;
    • Tensor parallelism: split the computation of a single layer across multiple cards to complete it cooperatively; suitable for large models that don’t fit on a single card;
    • Pipeline parallelism: split by layer across different cards, executed in a pipeline fashion.

    Parallelism introduces communication overhead, so tensor parallelism is usually limited to single-machine, multi-card scenarios with high-bandwidth interconnect. When scaling across nodes, pipeline parallelism is more commonly adopted.

    Prefill and Decode Disaggregation

    A common approach is to split Prefill and Decode into two separate types of instances—PD disaggregation: Prefill instances handle the compute-intensive prefill, and Decode instances handle the memory-access-intensive decode, with the cache transferred between them over a high-speed network. The two types of instances can each select different accelerator specifications, set parallelism strategies independently, and scale elastically on their own, thereby mitigating mutual interference.

    Caching and Compression

    • Prefix caching: a large number of requests in the system share the same system prompt, knowledge context, or multi-turn conversation history. After caching the intermediate results of the common prefix, the repeated Prefill computation can be skipped directly;
    • Quantization: storing weights and caches in lower-bit numerical formats to reduce GPU memory usage and improve memory-access efficiency. The precision loss needs to be validated with offline evaluation sets;
    • Speculative decoding: a lightweight draft model first generates several candidate tokens continuously, then the large model verifies them in parallel all at once. When the large model accepts only one token at a time, compute is underutilized; parallel verification fills this gap precisely, while the final result distribution remains consistent with the original model;
    • Output compression: compression can be enabled during streaming to reduce public-network bandwidth consumption and client-side parsing overhead.

    Service Orchestration and Elastic Scaling: Handling Exponential Growth

    When a business enters a period of rapid growth, call volume can double within weeks or even days. Static, fixed capacity causes queuing timeouts at peaks and resource idleness at troughs. Elastic scaling is therefore an essential capability for MaaS platforms.

    Choosing the Right Scaling Signal

    CPU utilization is a common scaling signal for traditional web services, but it isn’t always reliable in inference scenarios. If the bottleneck is memory bandwidth or cache capacity, CPU utilization may not be high while GPU utilization is already saturated. Signals closer to reality include:

    • In-flight requests or in-flight tokens;
    • Inference queue length and queue wait time;
    • The P95/P99 percentile values of TTFT;
    • Changes in cache hit rate.

    Combining Predictive and Reactive Scaling

    • Predictive scaling: based on historical call curves and activities already scheduled on the business side, warm up capacity in advance;
    • Reactive scaling: triggered by real-time metrics to respond to burst traffic;
    • Stability window: scale-up can be fast, but scale-down should be slow and set with an observation window to avoid frequent capacity fluctuations.

    Cold Start Optimization

    A model with tens of billions of parameters may have tens of GB just in weights; loading them into GPU memory and completing warm-up can take minutes. This is extremely unfriendly to second-level traffic spikes. Common mitigation approaches include:

    1. Warm instance pool: keep a certain number of idle instances that can take traffic at any time;
    2. Local cache for faster loading: cache weights on high-speed storage local to compute nodes, avoiding pulling from remote object storage on every startup;
    3. Tiered deployment: keep core models resident and load long-tail models on demand. Set reasonable request timeouts for long-tail models and return a clear status code when queuing takes too long.

    Topology-Aware Scheduling and Heterogeneous Resource Pools

    • Topology awareness: parallel inference relies on high-bandwidth communication between instances, so the scheduler should prioritize orchestrating each process of the same model instance within the same high-speed interconnect domain;
    • Heterogeneous co-location: models of different sizes have different compute demands, so the platform can schedule by model-size tiering to improve overall resource utilization.
    1. # Illustration: elastic scaling policy for an inference service
    2. model_service:
    3. min_replicas: 4 # Minimum replicas to guarantee baseline capacity
    4. max_replicas: 60 # Maximum replicas, determined by quota and budget
    5. warm_pool_size: 4 # Warm instances to absorb burst traffic
    6. scale_up:
    7. target_queue_wait_ms: 100 # Scale up when queue wait exceeds the threshold
    8. step: 50% # Scale to at most 1.5x the current replicas at once
    9. scale_down:
    10. stable_window_minutes: 15 # Scale down only after 15 min consistently below threshold
    11. # Reserved capacity: derived from enterprise tenants' SLA commitments
    12. reserved_capacity_ratio: 0.2

    Reliability and Security: The Baseline for Enterprise Services

    Fault Isolation and Graceful Degradation

    The large model service chain is long with many dependencies, so a failure in any link can cause an overall outage. Feasible measures include:

    • Multi-AZ deployment: distribute traffic across availability zones by weight;
    • Instance health probing: automatically remove unhealthy instances;
    • Call timeout control: avoid long requests piling up and exhausting concurrency quota;
    • Fast failure: promptly reject requests that exceed queue capacity or are expected to wait too long, and return an identifiable status code so clients can decide early;
    • Degradation plans: when the primary model lacks capacity or inference is abnormal, the platform can switch to a backup model, reduce the default generation length, or shorten the context window per predefined policies.

    Security and Data Protection

    • Encryption in transit and at rest: use TLS encryption for the entire API call, and encrypt sensitive data such as model weights and keys at rest;
    • Least privilege: grant independent credentials per application or sub-account, supporting revocation and rotation;
    • Tenant data isolation: requests, logs, and billing data are isolated between tenants at the storage and query layers;
    • Content safety: review input and output in both directions, intercepting illegal and non-compliant content;
    • Transparent data usage policy: clarify whether user input data is used for model training, and provide user-controllable options;
    • Audit trails: call logs, configuration changes, and administrative operations should all be auditable.

    Tiered SLAs

    MaaS platforms usually need to provide differentiated commitments for different service tiers. A common approach is to set availability targets and response latency targets, such as “monthly availability of at least 99.9%.” Higher-tier tenants receive reserved capacity and priority scheduling, while lower-tier tenants share elastic capacity and accept queuing or rate limiting when resources are tight.

    Observability and Cost Governance

    Four-Layer Observability Metrics

    Dimension Key metrics Purpose
    Business Call volume, active developers, paid conversion rate, revenue Business decisions
    Service quality TTFT/TPOT percentiles, error rate, rate-limit rate, queue time Experience assurance
    Resource Accelerator utilization, GPU memory usage, cache hit rate, network bandwidth Capacity and performance optimization
    Cost Cost per token, cost per request, idle resource ratio Cost governance

    End-to-End Tracing

    Generate a globally unique trace ID for each request that runs through the gateway, queues, inference instances, and cache layer, precisely localizing high-latency requests to specific stages such as queuing, prefill, decode, or network transport.

    Cost Optimization Approaches

    The cost of a MaaS platform is mainly composed of compute costs. Common optimization approaches include:

    • Dynamic batching: improving compute utilization through continuous batching;
    • Cache reuse: reducing repeated Prefill computation through prefix caching;
    • Load-based tiered deployment: scaling down during off-peak hours and using spot resources to handle offline batch jobs;
    • Usage transparency: providing developers with token usage and cost estimation to guide them in optimizing prompt length and call frequency.

    Implementation Roadmap

    Building a MaaS platform doesn’t have to be done all at once; it can proceed in three phases:

    1. MVP phase: complete a unified gateway, single-model inference service, basic authentication and rate limiting, and token metering with a usage dashboard to quickly validate market demand;
    2. Scale-up phase: introduce multi-model routing and orchestration, continuous batching and prefix caching, elastic scaling based on multi-dimensional metrics, multi-AZ disaster recovery, as well as FinOps capabilities like cost forecasting, call quotas, and budget alerts;
    3. Refined operations phase: implement PD disaggregation and heterogeneous resource scheduling, mixed deployment of inference traffic, per-tenant tiered SLAs, and full-lifecycle model version management.

    When selecting infrastructure and platform capabilities, you can start by evaluating common capabilities offered by major cloud providers, such as GPU cloud servers, container platforms, object storage, message queues, and monitoring, alerting, and logging services. Platforms such as Baidu AI Cloud offer model services of various parameter sizes along with a supporting toolchain, and can host training and inference workloads through an AI heterogeneous computing platform. Ultimately, however, the architecture should still be determined based on your own business scale, latency requirements, and cost budget.

    Conclusion

    The essence of MaaS is to turn large model capabilities into stable, metered, and governable public infrastructure through engineering. Beyond model quality, it must solve multiple balances simultaneously: latency and throughput, elasticity and cost, security and experience. For enterprise developers, understanding MaaS’s layered architecture and core optimization approaches helps them make more reasonable selection and integration decisions and guides them in designing AI applications with greater resilience and more controllable costs.