Reducing LLM Inference Costs: MoE and KV Cache Optimization Practices

1. Inference Costs: The Transition from Usable to Practical LLMs

Over the past two years, the focus of the LLM industry has shifted from “training more powerful models” to “deploying models at an acceptable cost.” For most businesses, inference is a continuous operational expense: every conversation, every Agent tool-calling loop, and every batch of document parsing consumes compute power. From an engineering perspective, inference costs consist of three main components:

• Compute costs: The compute power required for forward passes, which correlates positively with the scale of activated parameters and sequence length.
• Memory and storage costs: Model weights reside permanently in VRAM, while the KV Cache expands linearly with context length. In long-context and high-concurrency scenarios, its footprint can even exceed that of the weights themselves.
• Scheduling costs: Hidden waste caused by insufficient GPU utilization and uneven peak-valley load distributions.

New-generation model architectures can offer significantly lower serving prices compared to models of similar scale, fundamentally because they apply systematic optimizations across these three areas. Understanding the mechanics behind these optimizations provides direct value for developers in model selection and tuning.

2. MoE and Sparse Activation: Decoupling Parameter Scale from Compute Costs

The core philosophy of MoE (Mixture of Experts) is “large total parameters, small per-token activation.” The router selects only a few experts for each token’s computation. Thus, the model can “hold” vast amounts of knowledge (total parameters determine the capability ceiling) while paying a very low compute cost per forward pass (activated parameters determine the per-token cost).

However, traditional MoE faces practical challenges on the inference side: although experts are sparsely activated, their weights must fully reside in VRAM. Routing also introduces additional communication and scheduling overhead. Therefore, new-generation architectures generally evolve in the following directions:

1. Finer-grained expert partitioning and shared expert design to improve the utilization efficiency per parameter.
2. Asymmetric input/output compute allocation. The concept is straightforward: “reading” (Prefill, understanding long context) and “writing” (Decode, generating tokens token-by-token) have entirely different compute characteristics. Prefill is highly parallel and compute-friendly; Decode is memory-bound and limited by VRAM bandwidth. Configuring different scales of activated parameters for input and output ensures that compute power is spent exactly where needed—this is the value of asymmetric architectures.
3. Scalability for larger parameter scales, incorporating “whether it can scale smoothly” as a constraint during the architectural design phase.

An easily overlooked detail is that small activated parameters do not equate to a small memory footprint. Full weight residency, multi-replica deployment, and expert parallelism communication are all hidden costs. When evaluating whether an MoE model is “truly cheap,” you should look at the end-to-end cost per token, not just the parameter ratio.

3. KV Cache: The Bottleneck and Compression Paths in the Long-Context Era

3.1 Why KV Cache Spirals Out of Control

During autoregressive generation, to avoid recalculating the entire historical attention for every newly generated token, inference frameworks cache the Key/Value at each layer. Its size can be estimated using the general formula:

KV Cache ≈ 2 × num_layers × num_kv_heads × head_dim × seq_len × precision_bytes

Taking a typical configuration (32 layers, 32 KV heads, 128 dimensions per head, FP16) as an example, each token generates about 512KB of KV cache. When the context reaches 32K, a single request consumes roughly 16GB of VRAM—and this excludes the model weights. This is why, in long-context and multi-turn conversation scenarios, VRAM is often exhausted before compute power.

3.2 Common Industry Compression Paths

• Structural compression: MQA/GQA allows multiple query heads to share a single set of KV heads, reducing the cache linearly with the number of KV heads (e.g., GQA with 8 groups drops the footprint directly to 1/4 compared to 32 heads).
• Low-rank latent compression: Projecting KV into a lower-dimensional space and restoring it during computation, trading a small amount of compute for a substantial reduction in cache size.
• Quantized storage: Saving KV in FP8/INT8. For most tasks, the precision loss is acceptable, and the cache is reduced by more than half.
• Sparsification and windowing: Retaining only important tokens or the KV within a sliding window, discarding the rest or recalculating on demand.
• Paged memory management: Allocating memory on demand, much like an operating system manages memory pages, to eliminate fragmentation and boost concurrency.
• Tiered storage: Keeping hot data in HBM and offloading cold data to system memory or even SSDs. After multiple architectural iterations, the KV storage requirement per unit of context can be reduced by several to dozens of times compared to early versions.

4. Cost Structure of Agent Workloads and Selection Criteria

Agents represent the most typical inference workload today: a single task encompasses multiple loops, including planning, tool calling, result backfilling, and reflection, with the context growing continuously as steps progress. Its cost characteristics are: high cache hit ratio (system prompts and tool definitions are repeated in every request), input tokens far exceeding output tokens (heavy compute spent on re-understanding context), and latency sensitivity (multiple loops amplify single-turn latency).

This explains why the combination of “asymmetric input/output design + KV cache compression + prefix caching billing” is particularly effective for Agents. If your business is primarily Agent-driven, you should focus on these metrics during model selection, rather than solely relying on general leaderboards.

5. Multimodal Capabilities and Version Migration: What Callers Should Note

New-generation models natively support visual understanding, meaning images no longer rely on bolted-on external solutions. In practice, note that images are encoded into a specific number of visual tokens; the higher the resolution, the greater the overhead, so this should be controlled based on business needs. Stable multimodal prefixes can also hit the cache.

Regarding version migration, providers often use “compatibility routing” to smoothly deprecate older versions (temporarily pointing the old model name to the new model). While this facilitates migration, it also means behavior might have subtle differences. For critical businesses, explicitly switch to the new model name and conduct regression testing.

6. Peak/Off-Peak Pricing and Task Scheduling: Coding Cost Savings into Your Workflow

Many API services adopt peak/off-peak pricing, where off-peak prices are roughly half of peak hours. This is a leverage point to save money purely through scheduling:

# Pseudocode: Off-peak batch task scheduling
import datetime
def is_off_peak(now) -> bool:
    # Definition of off-peak hours depends on the provider's announcement
    return 0 <= now.hour < 8
if is_off_peak(datetime.datetime.now()):
    for task in offline_tasks:   # Document parsing, data cleaning, batch evaluation, etc.
        submit(task)

Supporting recommendations: Centralize model names in a configuration center so new version releases only require changing one place; build a minimal regression set of a few dozen representative cases to automatically compare before and after version switches; log prompt/completion token counts and cache hit counts at the gateway layer to allocate costs by business line; queue non-real-time tasks that can be delayed and consume them during off-peak hours.

7. A Rational Checklist for Evaluating “Stronger and Cheaper” New Models

1. Define your workload profile: input/output ratio, context length, concurrency, and whether it is multimodal.
2. Run regressions using a self-built evaluation set, focusing on the success rate and average steps of Agent-like tasks.
3. Test end-to-end costs: total expenditure (including cache hits and misses) for the same task across different models and time periods.
4. Observe stability metrics: throughput, time-to-first-token (TTFT), and timeout rate.
5. Conduct small-traffic canary testing before fully switching over.

8. Conclusion

Reducing LLM inference costs is not a victory of a single technology, but the叠加 of three layers of optimization: “architectural innovation (MoE, asymmetric structures) + cache governance (compression, tiering, prefix reuse) + scheduling strategies (peak/off-peak, batching).” For developers, understanding the principles behind these mechanisms holds more long-term value than chasing the parameter sheets of individual releases. It determines whether you can continuously deliver stable and reliable intelligent applications at a reasonable cost amidst the rapid iteration of models.