Cutting Agent Costs and Latency: MoE, Caching, and Cloud-Edge Collaboration

    Free trial quotas lower the barrier to trying an AI agent, but long-term costs are determined by model architecture, caching strategy, and deployment topology. This article explains from a general technical perspective how Mixture-of-Experts (MoE), prefix caching, and cloud-edge collaboration each reduce cost and latency, what prerequisites they require, and what to verify before production deployment—including TTFT decomposition, a caching savings formula, a long-context vs. RAG comparison, and a go-live checklist.

    An open-source agent project recently set a domestic large language model as its default recommended model, bundled with a free usage quota for a limited period. Free quotas lower the barrier to experimentation, but what determines long-term cost is the model architecture, caching strategy, and deployment approach. This article explains, from a general technical perspective, what each of these three key technologies solves, what prerequisites they require, and what to verify before going live.

    1. MoE: Reducing Per-Token Compute with Sparse Activation

    A dense model uses all of its parameters on every inference pass—the larger the parameter count, the higher the compute per token. A Mixture-of-Experts (MoE) architecture places multiple expert subnetworks inside the Transformer’s feed-forward layers, and a routing network selects the top-scoring few experts for each token. The remaining experts sit out that computation, so the number of parameters actually active is far smaller than the total parameter count.

    Take a hypothetical model as an example: total parameters might reach hundreds of billions or even trillions, but each token passes through only a small fraction of the experts, with active parameters in the range of a few billion to tens of billions. Since compute scales with active parameters, the compute cost of a single forward pass drops significantly, while the model retains a large knowledge capacity.

    Common routing configurations are top-2 or top-8, meaning each token is routed to 2 or 8 experts. This value is fixed during model training—users should not treat it as a universal optimum. The capacity factor limits how many tokens each expert can process in one pass; overflow tokens are dropped or rerouted to other experts, so it affects both output quality and throughput. Higher expert parallelism reduces per-GPU memory pressure but increases cross-device communication. Inference frameworks differ considerably in how they implement routing and communication, and the same model can show noticeably different latency on two frameworks—so only measurements taken in your own environment are authoritative.

    Three caveats when working with MoE. First, MoE reduces compute per token, not the storage requirement for all parameters. Every expert still needs to fit into GPU memory or be distributed across multiple cards, so the deployment bar is not low. Second, routing tends to produce load imbalance—some experts run hot while others idle—requiring load-balancing losses or capacity constraints. Third, when experts are spread across devices, tokens must travel across cards, and communication latency can offset part of the compute savings. When evaluating options, focus on measured throughput and latency rather than the active-parameter headline number.

    2. Time to First Token (TTFT): MoE Is Only One Factor

    Time to First Token (TTFT) is the elapsed time from when a user sends a request to when the first output token arrives. In interactive agents, it directly determines how fast the system feels.

    TTFT typically consists of four components:

    Component Main influencing factors How to measure Optimization approaches
    Network round trip Link quality, gateway location Time from client sending the request to the server receiving it Nearby ingress points, persistent connection reuse
    Queueing Server concurrency, scheduling policy Time from request arrival to start of computation Rate limiting, priority scheduling, scaling out
    Prefill computation Input length, active parameter count Time from start of computation to the model finishing all input Shorter inputs, smaller models, MoE
    KV cache reuse Whether prefixes match, whether cache has expired Number of cache-hit tokens returned by the server Fixed system-prompt prefixes, keeping sessions alive

    MoE mainly affects the Prefill component. If the network is jittery, queues are long, or inputs are very long, TTFT can remain high even with an MoE model. Therefore, before optimizing, measure each of these four segments separately, identify the bottleneck, and only then decide where to invest.

    Record both P50 and P95 latency. P50 reflects the typical experience; P95 reflects the tail, which often determines whether users perceive stutter. Aim for at least a few hundred requests per measurement window, and compute statistics separately for peak and off-peak periods, since tail latency can differ substantially between the two.

    3. Long Context and Caching: Use Prefix Reuse, Know Its Limits

    256K is the context window of one example model; other models have different window sizes—refer to the relevant provider’s documentation. Long context lets a model ingest the full system prompt, conversation summaries, and tool definitions in one pass, reducing reliance on external retrieval and making it easier to maintain instruction consistency over long conversations. But long context is not free. Attention compute grows faster than input length, and KV cache memory grows linearly with input length. If every request reprocesses the same long prefix, cost and latency accumulate continuously.

    Context caching (prefix caching, typically based on the KV cache mechanism) targets exactly this problem. When the prefix content is fixed, the server can store its intermediate computation results, and subsequent requests only compute the new portion. The benefit depends on the following conditions:

    • Hit rate: The request prefix must match the cached content character-for-character. If a timestamp or random ID is embedded in the system prompt, the cache misses.
    • Prefix share: The higher the proportion of reusable content in the total input, the greater the savings.
    • Time-to-live (TTL): Expired cache entries must be recomputed; sessions idle for a long time may miss.
    • Billing rules: Providers price cache hits and cache storage differently; calculate against the actual price list.

    A rough estimate of the savings ratio:

    Savings ratio ≈ hit rate × reusable prefix share × (1 − cache unit price ÷ original input unit price) − cache storage cost share

    Definitions: hit rate is the proportion of cache-eligible requests that actually hit the cache over a given period. Reusable prefix share is the average ratio of reusable prefix tokens to total input tokens per request. Cache unit price and original input unit price come from the provider’s price list, and their ratio reflects the post-hit discount. Cache storage cost share is the storage fee as a proportion of the original input cost. The formula assumes cache misses are still billed at the original rate, so the result is an approximation.

    Example: assume a hit rate of 80%, a reusable prefix share of 70%, a cache unit price at 10% of the original input price, and a cache storage cost share of 5%. Plugging in: 0.8 × 0.7 × (1 − 0.1) − 0.05 ≈ 0.45, meaning roughly 45% savings on input costs. These numbers only demonstrate the calculation; replace them with your own business data.

    In multi-tenant scenarios, the cache key must include a tenant identifier to prevent different users from sharing prefixes containing private content. A common design concatenates tenant ID, model version, system-prompt hash, and tool-definition hash into the cache key. Any change to one of these produces a new key, and the old cache naturally becomes unreachable. Caches may also store users’ personal information, so set a reasonable TTL, proactively purge the relevant cache when a tenant is deleted, and encrypt any persisted cache data.

    For choosing between long context and retrieval-augmented generation (RAG), refer to this comparison:

    Dimension Long-context direct input RAG retrieval injection
    Cost Longer input per request; relies on caching to reduce repeated cost Only a few relevant snippets injected; shorter input
    Latency Longer input means slower Prefill One extra retrieval step, but less injected content
    Accuracy and recall Coherent reasoning when material is complete; details may be missed when material is too long Depends on retrieval quality; missed retrieval directly hurts results
    Update frequency Content changes require full recomputation; cache invalidates frequently Only the changed fragments need re-indexing
    Privacy and tenant isolation Entire material enters model context; high isolation requirements Permission filtering at the retrieval layer, finer granularity
    Implementation complexity Simple engineering; focus on cache design Requires maintaining vector indexes, chunking, and the retrieval pipeline
    Suitable scenarios One-shot reasoning over long material, strong instruction-consistency requirements Knowledge-base Q&A, frequently updated knowledge

    Selection rules: prefer RAG when knowledge updates frequently, since update costs are low and injected inputs are shorter. Choose long context plus caching only when strong instruction consistency is needed or when processing long material in one shot. The two can also be combined—for example, route fixed system prompts and tool definitions through the cache, and business knowledge through retrieval.

    4. Cloud-Edge Collaboration: Applicability and Limitations

    Cloud-edge collaboration means running part of the inference on the user’s device or a local gateway, while handing the rest to cloud models. It addresses three problems: sensitive data that should not be uploaded, the need to respond under unstable network conditions, and the desire to reduce call costs for simple requests.

    Prerequisites include: the device has enough memory and compute to run a small model or classifier; the business allows some data to stay local; and cloud call cost or network latency has already become a clear bottleneck. If none of these hold, introducing cloud-edge collaboration only adds system complexity.

    The limitations are equally clear. Edge models are generally weaker than large cloud models, so complex reasoning still goes to the cloud. Edge and cloud model versions must stay in sync—version mismatch causes behavioral divergence. If context and user memory are split across both sides, state consistency is hard to guarantee. Updating, rolling back, and compatibility-testing edge models is also more cumbersome than on the cloud.

    A common division of labor: the edge handles intent recognition, sensitive-data masking, and simple commands, while complex tasks go to the cloud. Degradation strategy must be designed in advance—when the cloud is unavailable, the edge can fall back to a local small model or return cached results, clearly informing the user that the system is in a limited mode. Pre-upload masking rules, log retention periods, and data-in-transit encryption should also be part of the design.

    5. Go-Live Checklist

    1. TTFT: Instrument the four segments (network, queueing, Prefill, KV cache) separately; record P50 and P95; compute statistics by time period.
    2. MoE: Log active parameters, routing load distribution, GPU memory usage, cross-card communication time, and actual throughput—don’t rely on the active-parameter number alone.
    3. Caching: Track hit rate, reusable prefix share, and hit decay within the TTL; calculate cache unit price and storage fees against the provider’s price list.
    4. Prefix stability: Check whether timestamps, random IDs, or session numbers are embedded in the system prompt, and move them outside the prefix.
    5. Tenant isolation: Confirm the cache key contains tenant ID, model version, and prompt hash, and test that cross-tenant requests cannot hit another tenant’s cache.
    6. Data lifecycle: Set cache TTLs, define the purge procedure for tenant deletion, and confirm persisted data is encrypted.
    7. Cloud-edge collaboration: Define which requests go to the cloud and which are handled at the edge; verify the edge-cloud model version sync mechanism; test the degradation path when the cloud is unavailable.
    8. Selection review: Using your own business samples, compare long context plus caching against RAG on cost, latency, and accuracy before deciding on the primary approach.