LLM MaaS: Engineering Practices for Token-Based Billing and Gross Margin Improvement
A practical guide to the Model-as-a-Service (MaaS) business model for large language models. It explains why ARR matters more than raw revenue, why inference compute is the core cost driver, and how four engineering levers—inference acceleration, tiered model routing, semantic caching, and resource scheduling—lower per-token costs and improve gross margins as usage scales.
1. What the MaaS Model Really Is: Turning Model Capability into a Callable Service
Model as a Service (MaaS) is a business model that packages large-model inference capability into standardized interfaces and delivers it on demand to enterprise customers and developers through APIs. Unlike traditional project-based delivery, the defining trait of MaaS is “continuous invocation, usage-based billing”: instead of buying a one-time custom model build, customers embed model calls into their business systems, and each request corresponds to a certain amount of token consumption that is then billed accordingly.
This model produces two direct outcomes:
- Predictable, recurring revenue: as long as the customer is running, calls keep happening, so revenue takes on a recurring character.
- Revenue tightly coupled to resource consumption: every token generated maps to real compute cost, so revenue and cost sit on the same chain.
Precisely because revenue and cost are bound together, over the past year leading model providers have faced a critical question while their MaaS revenue climbs quickly—how to hold or even improve gross margin while scaling up. This is the core entry point for understanding the financial health of MaaS.
2. Why Revenue Health Is Measured by ARR, Not Total Revenue
ARR (Annual Recurring Revenue) is the metric obtained by annualizing periodic, repeatable revenue. Its biggest difference from “total revenue for the current year” is that ARR measures a business’s sustained revenue capacity, not the book revenue at a single point in time.
In MaaS scenarios, the significance of ARR is amplified further:
- It reflects genuine invocation stickiness. If a customer only runs a one-time PoC (proof of concept) or a single project integration, that contributes one-off revenue; revenue only converts into sustainable ARR when the customer embeds model capability into daily production workflows and steadily consumes tokens every day.
- It strips out project-based noise. For large-model companies that were previously project-driven, revenue often fluctuates with project delivery, making it hard to gauge how reusable the underlying product is. After shifting to a MaaS model centered on API calls, ARR becomes a more objective measure of “whether the model is actually being used.”
- It is a dual signal of growth and quality. A large year-over-year jump in ARR (for example, growing by tens of times from a base in the hundreds of millions) indicates on one hand that the customer base and usage intensity are expanding, and on the other that the revenue mix is migrating from “selling projects” to “selling ongoing service.”
For developers and enterprise decision-makers, judging whether a model provider is healthy usually means looking at ARR and its trend rather than the revenue figure in any single earnings report.
3. The Root of the Margin Dilemma: Compute Cost Behind Tokens
A common question the market has long raised about MaaS is whether it easily falls into the trap of “revenue grows but profit does not.” To answer this, you first have to look clearly at the cost structure.
Under the MaaS model, the economics of a single API call can be simplified as:
gross margin per request = billed revenue - (inference compute cost + storage/network/ops cost + allocated fixed cost)
Among these, inference compute cost is the largest and least compressible variable. It depends on several factors:
- Total input/output tokens: output tokens typically consume more compute than input tokens (autoregressive, token-by-token generation).
- Model parameter count: the larger the parameter count, the more compute and GPU memory each forward pass requires.
- Concurrency and utilization: low GPU utilization and fragmented requests cause a lot of compute to be wasted on waiting and scheduling.
- Context length: long contexts significantly increase KV Cache memory usage and lower single-GPU throughput.
When revenue grows linearly while compute cost also grows nearly linearly, the margin space gets squeezed continuously. So whether MaaS can be profitable is fundamentally an engineering efficiency problem: can you serve the same token consumption at a lower compute cost?
4. Four Engineering Paths to Improve Gross Margin
1. Inference Acceleration: Raising Output per Unit of Compute
The core optimization direction at the inference stage is “generate the same content with fewer resources”:
- KV Cache optimization: more efficient memory management (such as paged KV Cache) reduces memory waste on long contexts and increases the concurrency a single GPU can carry.
- Quantization and low-precision inference: with controllable accuracy loss, using lower-bit weight and activation representations reduces memory usage and improves throughput.
- Continuous Batching: stitching the dynamic generation of different requests into the same compute batch, so short and long requests don’t wait on each other, significantly improving GPU utilization.
- Speculative Decoding: a small model first “guesses” several tokens, then the large model validates them in a batch, reducing the number of large-model forward passes.
These optimizations do not change model capability, but they directly determine “the number of tokens produced per GPU per day,” making them the first lever for improving gross margin.
2. Tiered Model Routing: Sending Simple Requests Down the Cheap Path
Not every request needs the strongest, largest model. A typical waste is “using a sledgehammer to crack a nut”—using a large model for simple intent recognition, format conversion, or summarization is a waste of resources. The idea behind tiered routing is to dynamically select an appropriately sized model based on request complexity:
# Example: model routing by task complexitydef route_request(query: str) -> str:complexity = estimate_complexity(query) # complexity estimation: length / task type / reasoning requiredif complexity == "simple":return "small_model" # low cost, high throughput, for intent recognition / format conversionelif complexity == "medium":return "medium_model" # balanced cost and qualityelse:return "large_model" # only for complex reasoning / long-form generationdef handle(query: str):model = route_request(query)return model.infer(query)
In real engineering, complexity estimation can be based on signals such as request length, task labels, and historical confidence, or even a lightweight classification model to score each request. Common industry practice suggests that a well-designed routing strategy can push a substantial share of high-cost calls down to small models, thereby lowering average inference cost without noticeably sacrificing experience.
3. Semantic Caching: Reusing Repeated Computation
In enterprise scenarios, a large number of requests are highly similar: the same knowledge Q&A asked repeatedly, summaries of similar documents, templated extraction tasks, and so on. The idea of a Semantic Cache is:
- Vectorize the request and store it in a vector retrieval store;
- When a new request arrives, first run a similarity match; if it hits a sufficiently similar historical result, return the cache directly and skip inference;
- If there is no hit, run real inference and write the result back to the cache.
# Example: semantic cache hit checkdef get_answer(query: str, threshold: float = 0.95):vec = embed(query)hit, cached = vector_store.search(vec, threshold) # similarity searchif hit:return cached # reuse directly, zero inference costresult = model.infer(query)vector_store.add(vec, result) # write back to cachereturn result
The value of semantic caching is turning “repeated computation” into “zero-cost retrieval,” which pays off especially well for high-frequency, low-timeliness scenarios. Note that the threshold must balance hit rate against accuracy: set it too high and the cache is effectively useless; set it too low and there is a risk of returning wrong answers.
4. Batching and Resource Scheduling: Squeezing the Most Out of GPU Utilization
The cost efficiency of MaaS depends heavily on resource utilization. On the engineering side, you can optimize from these angles:
- Offline batch (Async Batch): for tasks that don’t require low latency (such as batch content generation or offline data processing), offer a batch API at a lower price and centrally schedule it to fill idle compute.
- Elastic scaling: dynamically adjust the number of instances based on traffic peaks and troughs, combined with message queues to smooth load, avoiding long-term idle resources reserved for peak.
- Multi-tenant co-location: co-locate requests from different customers and different priorities in the same cluster to smooth overall load and raise average utilization.
- Tiered storage: store infrequently used KV Cache, logs, and model weights in object storage of different performance/cost tiers to reduce storage overhead.
5. How Engineering Optimization Translates into Financial Metrics
These engineering measures don’t exist in isolation; through the single thread of “lowering per-token cost,” they ultimately translate into gross margin performance:
inference acceleration + tiered routing + semantic caching + scheduling optimization↓per-token inference cost drops↓lower compute spend for the same revenue↓higher margin per request → overall gross margin improves
This also explains why MaaS vendors are pouring investment into inference engines, caching infrastructure, and scheduling systems—the second half of MaaS competition is not only about the upper limit of model capability, but even more about the engineering efficiency of delivering that capability. Whoever can stably serve the same call volume at a lower unit cost can hold onto profit while scaling up.
6. Practical Advice for Developers
For teams evaluating or building MaaS-related systems, you can start from the following priorities:
- Build observability first: set up per-item monitoring and alerting for token consumption, request latency, GPU utilization, and cache hit rate—there is no optimization without measurement.
- Tackle the hard inference problems first: batching and KV Cache management are usually the optimizations with the highest return on investment.
- Then layer on routing and caching: once you have a clear cost picture, use tiered routing and semantic caching to further lower average cost.
- Judge health by ARR, not single-month revenue: focus on invocation stickiness and recurring revenue trends, not short-term book figures.
For example, using general-purpose model services and cloud infrastructure like those offered by Baidu AI Cloud, enterprises can combine general capabilities such as container platforms, object storage, message queues, logging services, and monitoring/alerting to fit their own business needs, building a MaaS service architecture that balances cost, stability, and scalability.
Conclusion
MaaS turns large models from a “one-time delivered project” into a “continuously running service.” This brings sustainable revenue represented by ARR, but also binds cost firmly to compute. Whether gross margin improves ultimately does not depend on the growth rate of book revenue, but on whether engineering capabilities such as inference acceleration, tiered model routing, semantic caching, and resource scheduling can keep pushing per-token cost down. Understanding and mastering this engineering thread is the key for developers to build economically viable, sustainable service systems in the MaaS era.
