Deploying Very Large Language Models: A Practical Guide to Long Context, Reinforcement Learning, and Inference

    This guide covers the main bottlenecks in deploying large dense and MoE models: KV cache memory, long-context evaluation, MoE All-to-All communication, reinforcement learning scheduling and sandbox isolation, fault handling for long-running agents, and consistent inference latency metrics. It gives formulas, measurement rules, and review checklists, using a hypothetical configuration that readers can replace with their own parameters.

    This article uses a hypothetical configuration as a running example. Each number comes with its assumptions, so you can substitute your own parameters and recalculate. The examples use dense models. Differences introduced by MoE architectures and larger parameter counts are noted in the relevant sections.

    1. KV Cache Memory for Long Contexts

    The KV cache memory can be estimated with the following formula:

    KV cache memory (bytes) = 2 × layers × KV heads × head_dim × bytes per element × sequence length × concurrency

    The factor of 2 accounts for storing both K and V.

    The example configuration is as follows:

    Parameter Example value Notes
    Layers 80 Replace with the value for your target model
    KV heads 8 Number of KV heads in grouped-query attention
    head_dim 128 Dimension of each attention head
    Precision BF16, 2 bytes FP8 uses 1 byte
    Sequence length 128K (131,072 tokens) Replace based on your workload
    Concurrency 1 Each additional concurrent request increases usage linearly

    Substituting these values, the KV cache for a single 128K sequence is 2 × 80 × 8 × 128 × 2 × 131,072 bytes, or about 40 GiB. This figure excludes model weights, activations, and framework overhead. At a concurrency of 8, the KV cache grows to about 320 GiB, which requires splitting across multiple GPUs.

    Inference services often use paged memory management, such as PagedAttention, which divides the KV cache into fixed-size blocks and allocates them on demand. This reduces memory fragmentation and increases usable capacity. Paging does not reduce the theoretical memory needed for a single sequence. It only improves the share of memory that can actually be used.

    Do not validate long-context support based only on the maximum length listed on the model card. Measure it as follows:

    1. Fix the model, framework version, hardware model, and precision, then record peak memory usage.
    2. Send requests according to an input length distribution (for example, 4K, 32K, and 128K, each at a set proportion). Record the P50 and P99 time to first token (TTFT) and the throughput in tokens/s.
    3. Keep peak memory usage within 90% of the usable capacity of a single GPU to leave headroom for bursts.
    4. Set the P99 TTFT threshold according to your business needs. For example, 3 seconds. If the threshold is exceeded, reduce concurrency or shorten the maximum length.

    Long-context quality must be tested separately. A common method is the needle-in-a-haystack test. Place key information at positions such as 10%, 50%, and 90% of a long document, then measure recall at each position. Results on short texts do not represent performance on long texts.

    Sliding window attention and sparse attention reduce computation but lose long-range dependencies. Before choosing one, compare them on your target task:

    Approach Memory and computation Long-range recall Main risk
    Full attention Computation grows quadratically with sequence length Best High cost for long sequences
    Sliding window Computation close to linear Information outside the window is lost Errors when key information falls outside the window
    Sparse attention Depends on the sparsity pattern Depends on pattern design Complex to implement and requires specialized kernels

    Extension methods for rotary position embedding (RoPE) also need validation on long texts. Beyond the training length, the performance gap can be large. Do not assume that performance beyond the training length matches performance within it.

    2. Training Parallelism and MoE Communication

    Training very large models usually combines several parallelism strategies:

    • Tensor parallelism (TP): splits the matrices of a single layer across multiple GPUs. It requires frequent communication, so it works best within the high-speed interconnect of one machine.
    • Pipeline parallelism (PP): splits the model by layer and passes activations between layers. Pipeline bubbles must be monitored.
    • Data parallelism (DP): each GPU holds a full copy of the model, processes different data batches, and synchronizes gradients.
    • Expert parallelism (EP): in MoE models, different experts are placed on different GPUs.

    In each MoE layer, tokens are routed, which produces All-to-All communication. The data volume of a single All-to-All operation in one layer can be roughly estimated as:

    Communication volume (bytes) ≈ number of tokens × hidden dimension × bytes per element × top-k

    Dispatch and combine each require one All-to-All operation, for two in total. Network bandwidth and topology directly affect this cost. Record the share of training step time spent on All-to-All. If the share exceeds a set threshold, check the network and parallelism configuration first.

    Load balancing across routes is equally important. Monitor it with the following metric:

    Expert load imbalance = tokens received by the most loaded expert / average tokens received per expert

    If this value stays above 1.5 for a long period, some experts are overloaded while others sit idle, and overall throughput drops. In that case, review the routing policy and the weight of the auxiliary loss. The threshold of 1.5 is an example value. Use your own training records to set the actual threshold.

    3. Reinforcement Learning Post-Training: Scheduling and Sandboxing

    Reinforcement learning in post-training usually involves a policy model, a reward model, and several downstream domains at the same time. The scheduling system should ensure that a problem in one domain does not block the others.

    Allocate resources as follows:

    1. Set separate GPU quotas for the policy model, the reward model, and each domain. The total of all quotas must not exceed the cluster capacity.
    2. Reserve a guaranteed minimum share for each domain so that no domain is preempted for long periods.
    3. Give each domain its own sampling queue, and run sampling and training asynchronously.

    Monitor the following metrics continuously:

    Metric Definition Example alert threshold
    Sample production rate Number of samples completed per second Below 50% of the 1-hour average
    Queue backlog Number of samples waiting for training More than 3 times the size of one batch
    Sampling timeout rate Timed-out requests / total requests Above 2%
    Reward computation failure rate Failed reward computations / total computations Above 1%
    Policy version lag Step difference between the policy used for sampling and the current policy Exceeds the configured limit

    If sampling stalls in one domain, handle it in this order:

    1. Timeout: a single sampling request that exceeds the configured duration is marked as timed out.
    2. Retry: retry only timeouts and rate-limit errors, up to 3 times, with exponential backoff.
    3. Degrade: after retries fail, skip that domain’s samples for the current round, record the reason, and let other domains continue training.
    4. Alert: when degradations exceed a threshold in a row, notify the owner. The owner decides whether to roll back or repair.

    Sandbox isolation checklist:

    • Code or tool calls generated by the model run in an isolated container or lightweight virtual machine and cannot access the host file system.
    • The sandbox has no network access by default and allows only allowlisted addresses.
    • Each execution has limits on CPU, memory, runtime, and output size.
    • Credentials do not enter the sandbox or appear in logs.
    • Tool permissions follow the principle of least privilege for each domain. Write, delete, and external send operations require approval.
    • The output of the reward model cannot directly change sandbox permissions or configuration.
    • Each execution records the input hash, tool calls, exit code, and resource usage. Audit log retention follows compliance requirements.
    • Personal information in logs is redacted.

    4. Failure Handling for Long-Horizon Agents

    Long-horizon agents usually execute many tool calls in sequence. A failure in one step should not leave the outcome of the whole task unknown. Model the task as a state machine with these states: planning, executing, validating, waiting for human input, completed, and failed.

    The key failure-handling designs are as follows:

    1. Checkpoints: save the state after each successful tool call. After a failure, resume from the latest checkpoint instead of starting over.
    2. Idempotency: assign idempotency keys to write operations to prevent duplicate charges, duplicate sends, or duplicate resource creation during retries.
    3. Retries: retry only retryable errors (timeouts, rate limits, and temporary unavailability), up to 3 times, with exponential backoff. Fail immediately on parameter and permission errors.
    4. Rollback: save a snapshot before each write operation. On failure, run compensating operations in reverse order. Mark operations that cannot be compensated as requiring manual handling, and record them in the task log.
    5. Budget limits: set caps on total tokens, tool call count, and runtime for each task. When a limit is exceeded, stop the task and preserve its state.
    6. Human intervention conditions: pause and wait for confirmation when consecutive failures exceed a threshold, when budget consumption exceeds 80%, or before operations such as deletion, payment, sending, or permission changes.

    5. Metric Definitions for Inference Acceleration

    Inference speedups can only be compared under consistent definitions. Define the following metrics explicitly:

    • Time to first token (TTFT) = time of the first output token − time the request arrived
    • Time per output token (TPOT) = (end-to-end latency − TTFT) / (number of output tokens − 1)
    • Throughput (tokens/s) = total output tokens in the measurement window / window duration in seconds
    • P99 latency: sort the latencies of all requests from lowest to highest. The value at the 99th percentile is the P99 latency.

    Record the following conditions during measurement. Without them, the numbers cannot be compared:

    • Hardware model, number of GPUs, and interconnect type
    • Framework and version, and model precision (BF16, FP8, or INT4)
    • Distribution of input and output lengths, and concurrency
    • Whether speculative decoding is enabled. If it is, record the acceptance rate.
    • Warm-up duration. Data collected during warm-up is excluded from results.

    Report both P50 and P99. Reporting only the average hides tail latency problems. When comparing two approaches, measure them on the same set of requests at the same concurrency.

    6. Technical Review Checklist

    • Memory: KV cache was calculated for the target concurrency and sequence length, and peak usage is documented.
    • Long context: recall tests were completed at positions such as 10%, 50%, and 90%.
    • Attention approach: sliding window or sparse attention was compared on the target task.
    • MoE: expert load imbalance and the share of time spent on All-to-All were recorded.
    • Reinforcement learning: quotas, guaranteed minimum shares, and timeout and degradation policies are documented.
    • Sandbox: network is disabled by default, credentials are kept out of the sandbox, and audit logs are searchable.
    • Agents: checkpoints, idempotency keys, compensating operations, and human intervention conditions are defined.
    • Budget: limits on tokens, tool call count, and runtime are set.
    • Metrics: measurement conditions for TTFT, TPOT, throughput, and P99 match the baseline version, and both P50 and P99 are reported.