MoE Model Selection Guide: Inference Cost, KV Cache, and Security Governance

    Selecting a Mixture-of-Experts (MoE) model requires more than comparing parameter counts. This guide explains how total and active parameters affect inference cost and VRAM, why expert routing balance matters, and how to budget for long context using KV cache and prefix caching. It also compares closed-source APIs, self-hosted open-source deployments, and edge-cloud collaboration, and provides a security and cost governance checklist covering keys, cache isolation, code sandboxes, cost alerts, and multimodal validation.

    1. Start with the Conclusions: Total Parameters Are Not Inference Cost, and Active Parameters Are Not VRAM Demand

    A Mixture-of-Experts (MoE) model splits the feed-forward layer into multiple expert subnetworks. For each token, a router selects only a small number of experts to participate in computation. When selecting a model, separate two concepts. Total parameters determine model capacity. Active parameters per token determine per-token computation. VRAM should be estimated based on all expert weights being resident.

    As a result, looking only at total parameters overestimates inference cost, while looking only at active parameters underestimates VRAM requirements. For example, a model may have a very large total parameter count but activate only a small subset of experts per token, so per-token computation can be significantly lower than that of a dense model with the same total parameter count. However, all expert weights usually still need to reside in VRAM or high-speed storage, so the hardware threshold for deployment remains high.

    Public materials for some flagship models have disclosed configurations with trillion-level total parameters, hundreds of experts, and more than ten experts activated per token. Such numbers apply only to specific models and versions, and different models vary widely. When citing them, specify the model name, version, and statistical definitions, and record total parameters, active parameters per token, total experts, and experts activated per token separately. Do not put unverifiable numbers into a model selection document; use official technical documentation as the source.

    Routing imbalance can also affect deployment. When a small number of experts are overloaded, latency concentrates on the hot experts. During training, load balancing loss is commonly used. Deployment teams can monitor the load distribution across experts to determine whether parallel strategies or the number of replicas need adjustment.

    2. How to Use a One-Million-Token Context: Window, KV Cache, and Caching Cost

    A context window of 1 million tokens means the model can accept a large codebase, long technical documents, or a large volume of logs. However, the window limit only indicates how much content the model can accept; it does not guarantee that the model can effectively use all of it.

    Long context mainly introduces two types of cost. First, KV cache grows with sequence length, increasing both VRAM usage and inference latency. The following approximation can be used to estimate KV cache size:

    KV cache bytes ≈ 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per element × concurrent requests

    Second, the model’s ability to extract information from the middle of the context may decline, often called the “lost in the middle” phenomenon. Key information should be placed at the beginning or end of the context, and irrelevant content can be removed through retrieval or segmented summaries.

    Cost Formula for Cache Hits

    Many vendors offer prefix caching for input tokens, with cached portions billed at a lower unit price. The cost of a single request can be calculated as follows:

    Cost per request = cached tokens × cache-read unit price + uncached tokens × input unit price + output tokens × output unit price

    Cached tokens plus uncached tokens equal the total input tokens for the request. Cache hit rate = cached tokens ÷ total input tokens.

    The unit prices below are hypothetical and are used only to demonstrate the calculation method. Actual calculations should use the vendor’s current prices. Assume CNY 4 per million input tokens, CNY 0.4 per million cached-read tokens, and CNY 16 per million output tokens. A request has 100,000 input tokens, of which 80,000 hit the cache and 20,000 do not, with 2,000 output tokens.

    • With cache: 80,000 × CNY 0.4/million + 20,000 × CNY 4/million + 2,000 × CNY 16/million = 0.032 + 0.080 + 0.032 = CNY 0.144
    • Without cache: 100,000 × CNY 4/million + 2,000 × CNY 16/million = 0.400 + 0.032 = CNY 0.432

    In this example, the cost per request drops by about 67%. If the endpoint is called 10,000 times per day, the daily costs for the two approaches are CNY 1,440 and CNY 4,320, respectively.

    Cost reduction recommendations:

    • Place system instructions, tool definitions, and fixed knowledge at the request prefix, and put dynamic content later to improve the hit rate.
    • Continuously monitor the hit rate. If it drops suddenly, first check whether the prefix has been modified.
    • Set an output limit to prevent unlimited model generation.
    • Calibrate estimates with measured data instead of planning budgets solely based on window size.

    3. How to Choose Among Three Access Models

    Dimension Closed-source API Open-source self-hosted Edge-cloud collaboration
    Cost structure Pay-per-use with low upfront investment Fixed hardware and operations costs; unit cost may be lower at high concurrency Cloud pay-as-you-go; edge hardware already exists, but split-processing development costs are required
    Latency Affected by network and server-side queuing Can be deployed nearby, making latency easier to control Simple tasks respond quickly on the edge; complex tasks fall back to the cloud
    Data compliance Data leaving the domain requires contract and terms assessment Data stays in the own environment with a clear compliance boundary Sensitive data can remain on the edge; routing rules must be clearly defined
    Operations Minimal operations required Requires maintaining GPUs, inference frameworks, autoscaling, and model upgrades Requires maintaining both edge and cloud paths
    Scalability Constrained by quotas and rate limits Depends on the scale of the own cluster Can scale separately by task tier
    Use cases Prototyping, highly fluctuating traffic, small teams High compliance requirements, stable and large call volume Frequent terminal interactions, privacy-sensitive scenarios, offline or weak-network availability

    Model selection steps:

    1. List the tasks and mark the input length, output length, latency requirements, and data sensitivity level for each task type.
    2. Use the same sample set to test all three access models, recording accuracy, P95 latency, and cost per request.
    3. First eliminate options that do not meet compliance requirements based on data sensitivity level.
    4. Compare monthly total costs among the remaining options, and include operations labor in self-hosted costs.
    5. After selecting the primary option, keep a backup option and re-evaluate regularly.

    4. Security and Cost Governance Checklist

    Key management:

    • Store keys in a KMS or key vault; do not write them into code, configuration files, or logs.
    • Rotate keys regularly and assign a separate key to each service.
    • Authorize by least privilege, exposing only the required models and interfaces.

    Cache isolation:

    • Include tenant ID and session ID in cache keys to avoid cross-tenant cache hits.
    • Do not cache content containing sensitive fields such as government ID numbers, phone numbers, or bank card numbers.
    • Set expiration times for caches and support purging by tenant.

    Code sandbox:

    • Run model-generated code in an isolated container.
    • Set CPU, memory, execution time, and disk quotas.
    • Disable internet access by default; when access is required, use an allowlist.

    Cost alerts:

    • Set daily and monthly budget thresholds for each project, with alerts at 80% and 100%.
    • Monitor cache hit rate, average cost per request, and abnormally high output token counts.
    • Set call limits per user or per interface.

    5. Boundaries of Multimodal Capabilities

    Native multimodal models learn images and text together during training, so mixed image-text reasoning is often smoother. However, being able to understand images does not mean the model is accurate under all conditions. Before launch, confirm the following parameters and test them with real samples:

    • Supported image formats, maximum size per image, and number of images accepted per request.
    • Whether video is sampled at a fixed frame rate or input as a whole clip, and the duration limit.
    • Whether outputs include coordinates or structured information. If coordinates are required, verify the coordinate system definition and error range.

    Common failure examples include incorrect recognition of small-font text, row-column misalignment after tables span pages, missed detection of heavily occluded objects, and fast actions in videos being missed by frame sampling.

    Common mistakes include:

    • Treating the maximum window size as the effective utilization limit.
    • Estimating cost only by total parameters.
    • Failing to isolate caches by tenant, leading to data leakage between tenants.
    • Not setting cost alerts and discovering overspending only when the monthly bill arrives.