Improving Gross Margin in MaaS: Engineering Practices for Lower LLM Inference Cost

    As MaaS (Model as a Service) replaces project-based model delivery, the key question is whether the model can escape the "revenue grows, profit does not" trap. This article explains why MaaS revenue is tightly coupled to inference compute cost, why ARR is the metric that reflects real business health, and how six engineering capabilities — inference optimization, caching and reuse, scheduling and dynamic batching, model routing, elastic capacity planning, and cost observability — work together to lower the cost of serving a token and raise gross margin. It also lists practical measures for developers and enterprise users building on top of MaaS platforms.

    From Project-Based Delivery to MaaS: A Structural Shift in the Revenue Model

    In the past, LLM vendors largely earned revenue from one customized delivery project at a time: train, fine-tune, and deploy a model for a particular customer, and the engagement ends at handover. This model produced “one-off” revenue that was hard to compound.

    MaaS (Model as a Service) abstracts model capability into standardized APIs. Customers and developers obtain intelligent services by calling those interfaces and consuming tokens (the basic billing unit for text processed by the model).

    The essence of this shift is moving from “selling a deliverable” to “selling ongoing usage behavior.” Revenue no longer depends on how many projects were signed, but on how many times the platform is called and how many tokens are consumed. For vendors, this significantly increases the predictability and continuity of revenue. For customers, it means no need to build expensive compute clusters, pay-as-you-go pricing, and fast integration.

    ARR: The Core Metric for Measuring MaaS Business Health

    In the MaaS context, Annual Recurring Revenue (ARR) is a much better indicator of business quality than “total revenue.” Its technical meaning is this: ARR measures whether customers keep calling the model and steadily consuming tokens, rather than relying on an occasional large one-off project to prop up the quarterly number.

    Consider the difference between the two. A project-based company may see revenue spike in a quarter because of a single big contract and drop to zero the next. A healthy MaaS platform, by contrast, accumulates revenue day after day from thousands of developers — a steady, continuous stream. When ARR grows rapidly, it shows that the platform’s model capability is genuinely in sustained use by the market and has built stickiness.

    This also explains why the industry increasingly values ARR over raw revenue growth: it strips out short-term fluctuations and points directly to the sustainable foundation of the business.

    The Root Cause of “Revenue Growth Without Profit Growth”: Revenue Tightly Coupled to Compute

    The biggest criticism leveled at the MaaS model is that it easily falls into the trap of “revenue grows, profit does not.” The root cause lies in the fact that MaaS revenue and compute cost are tied to the same chain:

    1. Token consumption ↑ → Revenue ↑
    2. Token consumption ↑ → Inference compute consumption ↑ → Cost ↑

    For every additional token served, the vendor both charges the customer and pays the corresponding GPU inference compute cost. If the selling price per token falls more slowly than the compute cost per token rises — for example, because redundant compute was reserved to handle traffic peaks, or because inefficient inference causes each request to consume excessive resources — then the larger the scale, the more the profit margin may be compressed.

    Therefore, the key to improving gross margin is not simply to raise prices or cut services, but to systematically reduce the inference cost per token on the engineering side. Six dimensions are covered below.

    Six Engineering Paths to Higher Gross Margin

    1. Deep Inference Optimization

    Inference is the most cost-intensive stage in the entire MaaS pipeline, and therefore offers the largest optimization opportunity.

    • Model Quantization: Compress model weights from high precision (such as FP16) to low precision (such as INT8 or INT4), which significantly reduces memory usage and improves throughput while keeping accuracy loss under control.
    • Knowledge Distillation: Use the outputs of a large model to train smaller models, letting simple tasks be handled by the small model and reserving scarce compute for the complex requests that genuinely need the large model’s capability.
    • KV Cache Reuse and Operator Fusion: During autoregressive decoding, cache historical Key/Value pairs to avoid repeated computation; at the same time, fuse operators to reduce memory read/write overhead.

    The common goal of these techniques is to let each GPU serve more requests without sacrificing output quality.

    2. Caching and Reuse: Making Repeated Computation Free

    A large share of real-world requests exhibit significant redundancy, and caching is one of the most cost-effective ways to reduce cost.

    • Prefix Cache: Many requests share the same system prompt or context prefix. Caching the KV computation result for that prefix lets subsequent requests with the same prefix hit directly, with no need to recompute it.
    • Semantic Cache: For questions that are highly similar in meaning, existing answers can be cached and reused.

    A simplified prefix cache decision logic:

    1. def handle_request(prompt, prefix_cache):
    2. system_prefix = extract_system_prompt(prompt)
    3. if system_prefix in prefix_cache:
    4. # Cache hit: reuse the cached KV state for the prefix and skip repeated prefill
    5. cached_kv = prefix_cache.get(system_prefix)
    6. return decode_with_prefix(prompt, cached_kv)
    7. else:
    8. # Cache miss: compute normally and write to the cache
    9. kv = prefill(system_prefix)
    10. prefix_cache.put(system_prefix, kv)
    11. return decode_with_prefix(prompt, kv)

    In enterprise scenarios with long prompts and fixed system instructions, prefix caching often delivers substantial savings in prefill overhead.

    3. Request Scheduling and Dynamic Batching

    Compute wasted while a GPU waits is compute thrown away. With dynamic batching, a scheduler can combine multiple concurrent requests within a time window into a single batch sent to the GPU, greatly improving hardware utilization. Combined with continuous batching, the system can insert a new request as soon as a request in the batch finishes, instead of waiting for the whole batch to complete, which reduces queuing latency and increases throughput.

    Scheduling strategy must balance two conflicting metrics: throughput (larger batches are more efficient) and latency (how long users can tolerate waiting). One typical trade-off is to adjust batch size dynamically according to the real-time queue length:

    1. def dynamic_batch_size(queue, max_batch, latency_budget):
    2. # Increase batch size under backlog for throughput; shrink it when latency is sensitive
    3. backlog = len(queue)
    4. if backlog > threshold_high:
    5. return max_batch
    6. elif latency_pressure() > latency_budget:
    7. return max_batch // 2
    8. return max(1, backlog)

    4. Model Routing: Coordinating Small and Large Models

    Not every request needs the most powerful model. By introducing a router layer that grades request difficulty, simple tasks can be routed to lightweight models and complex tasks to more capable models, cutting cost significantly while preserving overall quality.

    1. def route_request(prompt):
    2. difficulty = classify_difficulty(prompt)
    3. if difficulty == "simple":
    4. return small_model.generate(prompt) # Low cost
    5. elif difficulty == "medium":
    6. return medium_model.generate(prompt)
    7. else:
    8. return large_model.generate(prompt) # Complex requests only

    This tiered-service approach essentially establishes a continuously adjustable spectrum between cost and quality, rather than applying a one-size-fits-all rule.

    5. Elastic Resources and Capacity Planning

    A large portion of compute cost comes from redundancy reserved for peak traffic. Using general-purpose container platforms and autoscaling capabilities on the cloud, a MaaS platform can scale inference instances in and out automatically according to real-time traffic: scale out during peaks to meet SLAs, scale in during troughs to save money. At the same time, separating stateless inference services from the stateful cache layer, and using a message queue to smooth traffic spikes and fill valleys, prevents sudden bursts from overwhelming the system. Sound capacity planning pushes the proportion of idle compute down to a minimum.

    6. Cost Observability and Granular Operations

    “Costs you cannot see are costs you cannot optimize.” A mature MaaS platform needs a complete logging service and monitoring and alerting system to track token consumption, compute usage, cache hit rate, and cost per request at a fine-grained level for every call. By archiving historical call data in object storage for offline analysis, the platform can continuously locate cost hotspots, validate the effect of optimizations, and form a closed loop of “monitor — analyze — optimize — monitor again.”

    Practical Recommendations for Developers and Enterprise Users

    For developers and enterprise users building on MaaS platforms, there are also several measures worth taking to optimize their own costs:

    • Trim prompts: Redundant system prompts add token consumption on every round, so controlling context length is the most direct way to save money.
    • Use cache-friendly structures: Separate fixed instructions from dynamic content to maximize the chance of hitting the prefix cache.
    • Tier your needs: Identify which steps can accept a lightweight model, and concentrate the budget on the critical paths that truly affect experience.
    • Make your calls observable: Connect your own application to logging and monitoring, quantify the model cost of each functional module, and avoid “hidden spend.”

    Conclusion

    Whether MaaS can escape the criticism of “revenue growth without profit growth” ultimately depends on whether vendors can keep pushing down the inference cost per token. Rapid ARR growth proves that market demand is real, while improved gross margin relies on six engineering capabilities working in concert: inference optimization, cache reuse, intelligent scheduling, model routing, elastic resources, and cost observability. For teams that want to build sustainable competitiveness in this direction, whoever codifies these engineering practices into standardized platform capabilities first will hold the cost advantage in the competition of the token economy.