Long-Context LLM Serving and Capacity Planning: A Practical Guide

    A practical guide to deploying long-context LLM services: GPU memory budgets (weights, KV cache, runtime, headroom), request-length tiering, max context vs. max active sequences, load-test-based tuning, POC checks, and a go-live checklist.

    Long-context LLM serving needs its own plan

    LLM context windows keep growing; million-token inputs already appear in some model specs. A window size describes model capability, not the default service configuration. How much text a request can accept, how much GPU memory the server allocates, how long requests wait, and whether concurrency stays stable are three different questions. Before production, separate “the model can process this” from “the service can deliver it reliably.”

    GPU memory budget: more than weights

    Many estimates look only at parameter count. Mixture-of-experts (MoE) models have a large total parameter count, but each token activates only a subset of experts, so per-step compute is closer to the activated parameter size. That does not cut GPU memory the same way: all expert weights still need to reside in GPU memory or be quickly accessible. A full budget usually has four parts:

    • Model weights: estimate by precision. FP16 is about 2 bytes per parameter, FP8 about 1 byte, 4-bit quantization about 0.5 byte, plus quantization metadata.
    • KV cache: grows linearly with sequence length, concurrent sequences, layers, KV heads, and data type.
    • Runtime overhead: compute graph, communication buffers, temporary activation tensors, and similar.
    • Safety margin: fragmentation and peak spikes.

    KV cache can be estimated as follows. The factor of 2 covers Key and Value:

    1. KV cache bytes per sequence ≈ 2 × layers × KV heads × head dim × sequence length × bytes per element

    Suppose total weights are on the order of 700 GB (illustrative only; not a specific model). Even if a single GPU has enough memory, after multi-GPU sharding weights still take most of the memory; only the remainder is available for KV cache. Plan by totaling weights first, then how many tokens the leftover memory can hold, then the concurrency cap.

    Govern by request length

    KV cache is roughly proportional to sequence length. One near-million-token request can use as much cache as seven or eight 128K requests. If every request shares one queue, a few ultra-long requests starve resources and raise latency for most short requests.

    Measure the live length distribution—P50, P90, P99, and input vs. output separately—then tier:

    1. Up to 32K: shared pool as main traffic. RAG Q&A and short-document summarization usually land here.
    2. 32K–128K: per-user or per-tenant quotas, a separate queue, and a cap on in-flight requests.
    3. Above 128K: dedicated service instances with independent scale-out so tail latency of the main pool is not affected.
    4. Near the maximum window: open only to a few jobs, a low-concurrency dedicated queue, longer timeouts, and a degradation path.

    The point is to match resource use to business value. Do not put all traffic on a max-window config just because the model supports ultra-long input.

    Two parameters that are easy to mix up

    Two common serving settings are often misunderstood:

    • Max context length: the per-request token cap (usually input plus output). It drives worst-case KV cache reservation.
    • Max active sequences: how many sequences the scheduler may keep active at once. It affects batch size and memory peaks.

    Neither equals online user count. Real concurrency also depends on request and output length, queueing, rate limits, and continuous batching. One hundred users sending short questions may keep few sequences active; the same users sending long documents can exhaust GPU memory quickly.

    Illustrative config (names are structural only; use your serving framework’s docs):

    1. # Illustrative config for a layered setup
    2. serving:
    3. max_context_tokens: 131072 # per-request context cap
    4. max_active_sequences: 32 # scheduler active-sequence cap
    5. long_context_queue:
    6. max_concurrency: 2 # dedicated queue for ultra-long requests
    7. timeout_seconds: 600 # clear error and logging after timeout

    Tune from load tests, not guesswork. Fix a representative request set, raise concurrency stepwise, and record latency and GPU memory. The concurrency where latency starts rising nonlinearly is the capacity knee for that config. If long requests dominate, lower max active sequences; if short requests dominate and memory remains, raise it gradually and re-validate each step.

    Four questions a POC must answer

    Starting the service once does not prove it is production-ready. An effective POC covers four dimensions.

    First, is quality stable?

    • Task completion rate, human scores, and failure-type mix.
    • Tool-call success rate and parameter-error rate.
    • Run a fixed eval set multiple times and look at variance, not a single run.

    Second, does performance meet the bar?

    • TTFT (time to first token) and TPOT (time per output token).
    • P95 and P99 latency bucketed by input length.
    • End-to-end task time, not only a single model call.

    Third, where is the capacity boundary?

    • Max stable concurrency at different context lengths.
    • GPU memory, host RAM, local NVMe, and network bandwidth.
    • Find the knee and set rate-limit thresholds from it.

    Fourth, can the system recover?

    • Rehearse restarts, isolation of bad requests, and node failover.
    • Confirm rollback can finish within the allowed window.
    • Confirm request-level logs and tracing can locate issues.

    Go-live checklist

    • Request-length distribution measured and tiering rules designed.
    • GPU memory budget complete: weights, KV cache, runtime, safety margin.
    • Max context length and max active sequences set with load-test evidence.
    • Ultra-long requests have a dedicated queue, timeout, and degradation path.
    • All four POC dimensions completed with data and a report.
    • Rollback and failover rehearsed.

    Conclusion

    Treat long context as a tiered resource, not a single setting for every request. Production readiness depends on capacity planning from real length distributions, parameter limits from load tests, and a full POC of quality, performance, capacity, and recovery.