6 Blind Spots in LLM Production Deployment: From Running to Production-Ready
A large language model that starts successfully only proves its inference pipeline is wired up correctly—it doesn't mean the service is ready for production. Focusing on long-context LLM inference services, this article breaks down six commonly overlooked deployment gaps: KV Cache capacity planning, accelerator selection, scheduling parameter semantics, POC acceptance, quantization trade-offs, and model/task routing. It explains why "running" and "production-ready" are far apart, offering practical guidance on tiering requests by context length, reading metrics like TTFT and TPOT correctly, and building reproducible POCs, and closes with a deployment readiness checklist plus the essential production infrastructure.
In a recent private LLM deployment project, the model service started successfully, the API returned results as expected, and the acceptance meeting went smoothly with a long-document Q&A demo. The project looked nearly complete. One week after going live, problems clustered together: several extra-long document requests were submitted at once, GPU memory was exhausted, ordinary Q&A started queuing, and response time jumped from around 1 second to over ten seconds. Then, after a version upgrade, the service kept restarting. When the team tried to roll back to the old version, they discovered that neither the old image nor its configuration had been retained.
This isn’t an isolated case; it’s a common scenario when enterprises bring LLMs into production. An inference service that “runs” only proves the hardware-software pipeline is connected. Being “production-ready” is still a clear distance away. This article takes long-context LLM inference services as its subject and breaks down the six things most easily overlooked before entering a production environment—covering capacity, selection, parameters, acceptance, quantization, and routing—along with a deployment checklist.
1. First, a clear question: what does “running” actually prove
When a model starts successfully, it usually means these steps are complete:
- Accelerator card drivers and the inference framework work correctly;
- Weights load completely, and partitioning configurations such as tensor parallelism or pipeline parallelism take effect;
- The basic inference API is usable and can handle a single request.
These only show that the inference pipeline is technically connected. They don’t answer: Can real business tasks be completed reliably? What is the latency when dozens or hundreds of users work at once? Do long and short requests slow each other down? Are permissions, auditing, and monitoring in place? Can the system recover quickly during an upgrade or failure?
The “six things” discussed in this article:
- KV Cache capacity planning: determines how much real concurrency the service can carry.
- Accelerator selection: determines the throughput ceiling per unit cost.
- Scheduling parameter semantics: determines whether these capabilities can be correctly released.
- POC acceptance: determines whether the go-live decision is backed by data.
- Quantization trade-offs: determines whether accuracy and cost can be balanced.
- Model and task routing: determines whether total cost of ownership is reasonable.
A service that has only passed startup validation is, strictly speaking, just a “test service.” Going from a test service to a production service means crossing at least the following four thresholds:
| Capability dimension | Question to answer | Corresponding gaps |
|---|---|---|
| Service quality | Real task completion rate; stability of structured output and tool calls | Gaps 4, 5 |
| Capacity & performance | Whether TTFT and TPOT meet targets at target concurrency; whether long context is usable | Gaps 1, 2, 3 |
| Ops & security | Whether permissions, auditing, monitoring, release, and rollback are in place | Gap 4, checklist |
| Cost control | Whether per-request cost is reasonable; whether small tasks actually use the LLM | Gap 6 |
Let’s go through each one.
2. The six things most easily overlooked
Gap 1: Long context really consumes KV Cache, not “a switch”
Long-context models can have context windows up to the million-token level, but that doesn’t mean every request can be processed at maximum length directly.
Weights are a static cost; KV Cache is a dynamic cost. After the model loads, weights occupy a fixed amount; KV Cache grows with context length and concurrency. It can be roughly expressed as:
KV Cache memory ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_dtype × num_tokenswhere num_tokens = input length + generated output length so far
Note two points:
- KV head count is not the same as attention head count. Mechanisms like grouped-query attention (GQA) let each group of attention heads share one copy of KV, reducing KV Cache overhead.
- The formula doesn’t count extra overhead such as paged management and memory alignment; actual usage is usually higher.
Take a common ~64 billion-parameter dense model as an example, assuming FP16 precision, 48 layers, 8 KV heads, and head dimension 128. Plugging into the formula gives the KV usage per token:
def kv_bytes_per_token(layers: int, kv_heads: int, head_dim: int, dtype_bytes: int = 2) -> int:"""Estimate KV Cache bytes per token (excluding paging, alignment, and other overhead)"""return 2 * layers * kv_heads * head_dim * dtype_bytes# 48 × 8 × 128 × 2 (FP16) × 2 (K and V) = 196608 bytes, about 0.1875 MiBper_token = kv_bytes_per_token(layers=48, kv_heads=8, head_dim=128)print(per_token / 1024 / 1024) # 0.1875
Estimating at this scale:
- A request close to 1 million tokens needs over 180 GiB for KV Cache alone;
- A 128K request needs about 24 GiB;
- A 32K request needs about 6 GiB.
An ordinary Q&A request has only one or two thousand tokens, with KV usage under 0.5 GiB. For the same “one request,” the memory cost can differ by hundreds of times.
Output length also counts toward KV Cache. During generation, KV Cache grows with each token output. Requests with very long outputs—like long-form writing and multi-turn agent tasks—see KV usage keep growing during the decoding phase.
Concurrency scales the cost linearly. Processing 10 concurrent 32K requests needs about 60 GiB of KV memory; switch to 10 concurrent 128K requests and the requirement jumps to about 240 GiB. Since each decoding step must re-read the KV Cache, the longer the context, the more data a single forward pass has to move, and the system easily shifts from “compute-bound” to “bandwidth-bound.”
Long requests also drag down short ones. Under continuous batching, a long request’s prefill phase consumes a large amount of compute, during which short requests’ first responses are delayed. When memory is insufficient, the scheduler may also preempt or swap out sequences, further amplifying latency jitter and even triggering OOM.
A more reasonable approach is to tier by context length:
| Tier | Context scale | Resource strategy | Goal |
|---|---|---|---|
| Short requests | Regular context | Shared pool, dynamic batching allowed | Low latency, high throughput |
| Medium-long requests | 32K–128K | Quota pool, cap max active sequences and per-tenant quota | Predictable capacity |
| Long context | Above 128K | Dedicated resource pool, lower concurrency ceiling | Stable availability |
| Extreme context | Close to million-level | Low-concurrency dedicated queue; consider switching to async tasks | Usable without dragging down the whole |
Three supporting measures:
- Estimate first, then route at the ingress. The gateway should estimate request size by “input tokens + reserved output tokens,” not split pools by user role or API path.
- Make extra-long requests asynchronous. For extreme long requests that can take tens of seconds, use a “submit task—poll/callback for result” async pattern to avoid holding a connection for a long time and triggering gateway or load balancer timeout disconnects.
- Use caching to cut redundant computation. Enable prefix caching for reusable common prefixes such as system prompts and knowledge-base documents to reduce repeated prefill; if the business allows multi-turn conversations to share history, combine with context compression to lower the marginal cost of long context.
# Illustration: pick a scheduling pool by context length (specific thresholds and pool counts must be set from load-test results)def pick_pool(input_tokens: int, reserved_output: int, limits: dict) -> str:total = input_tokens + reserved_outputif total <= limits["short_max"]: # e.g., 8192return "shared" # shared poolif total <= limits["mid_max"]: # e.g., 131072 (128K)return "quota" # quota poolif total <= limits["long_max"]: # e.g., 524288 (512K)return "long_ctx" # long-context poolreturn "ultra_long" # extreme length; route to async task queue
Gap 2: Accelerator selection can’t look only at memory capacity
Different generations and classes of data-center accelerators each have their strengths: some have large memory and high bandwidth, others have strong single-card compute but relatively smaller memory. Following a common industry reference recipe, the single-node 8-card homogeneous form is often used as the serving-side solution for long-context models; to fully carry near-million-level context, you often need a card type with larger memory, more cards, or a combination of both. But these are just a starting point. Selection must at least consider the following dimensions:
(1) Pool by purpose first, then decide “who replaces whom.” Cards with large memory and high bandwidth are better suited to long context and high-concurrency decoding; cards with strong compute and smaller memory are better for shorter-context, throughput-focused tasks. Expecting one card type to cover all scenarios often pleases neither side.
(2) “Fits in memory” doesn’t equal “runs fast enough.” Available memory must deduct weights, KV Cache, runtime activations, communication buffers, and fragmentation; the portion allocatable to KV Cache is usually noticeably less than the nominal capacity. Take an 8-card single node with 80 GiB per card:
- Nominal memory: 640 GiB;
- At 90% utilization, about 576 GiB usable;
- Assume a model with weights of about 320 GiB, leaving only about 250 GiB;
- Long requests’ KV usage is clearly higher than ordinary requests; with activations and fragmentation counted, a single 128K request needs over 20 GiB;
- At this point, processing 4 concurrent 128K requests will nearly exhaust the KV Cache margin.
This example shows directly: in long-context scenarios, a concurrency gap of one to two orders of magnitude may come from card selection rather than parameter tuning.
(3) Task type determines where the bottleneck sits. The prefill phase is mainly compute-bound; the decoding phase is mainly memory-bandwidth-bound. Long-document “read-heavy, write-light” tasks bottleneck at prefill; long-form generation and multi-turn agent tasks that “write a lot” put more pressure on decoding and are more sensitive to memory bandwidth.
(4) Parallelism strategy is constrained by intra-node interconnect. Tensor parallelism does collective communication at every layer and is very sensitive to intra-node interconnect bandwidth. So prioritize keeping the parallel group within the same node’s high-speed interconnect domain, and avoid cross-node tensor parallelism that slows the whole down; when cross-node scaling is needed, prefer pipeline parallelism and other schemes with more controllable communication patterns.
(5) Data-center conditions are also a hard constraint. High-power accelerator cards have clear requirements for power capacity, rack power density, and cooling. If the data center still uses traditional air cooling, evaluate cooling upgrades and power expansion before buying high-power models, otherwise the equipment can’t run at full load once it arrives.
(6) Use benchmark data as the decision basis. Theoretical compute on a spec sheet doesn’t equal business throughput. Run the same benchmark (see Gap 4) on 2–3 candidate configurations and compare “effective throughput per unit hardware cost” and “max concurrency at target latency,” rather than peak metrics.
Gap 3: max-model-len and max-num-seqs are not “user count limits”
When starting an inference service, two parameters often appear in the config: maximum model length (max-model-len) and maximum number of sequences (max-num-seqs). They are often misread as “how long a document is supported” and “how many people can be served,” but their actual meaning is far more precise.
max-model-len: the context window ceiling, and also the ceiling for memory planning. It caps the total token count of a single request’s “input + output” and doesn’t guarantee that any length of request can be handled smoothly under high concurrency. If this value is far larger than what memory can actually support, long requests will hog large amounts of KV Cache, causing queuing, preemption, and even OOM.
max-num-seqs: the number of active sequences in one scheduling round, not the number of online users. It limits how many sequences the scheduler processes simultaneously per iteration, and works together with the following factors to determine actual capacity:
- Number of online users;
- Each user’s question frequency;
- Single-request input and output length;
- Waiting queue and rate-limiting strategy.
By Little’s Law, the average number of requests in the system is about the request arrival rate multiplied by average dwell time:
L = λ × WL: average number of requests in the system (including queued requests)λ: request arrival rate (req/s)W: average dwell time per request (seconds, including queue time)Example: λ = 0.5 req/s, W = 40 s, so L = 20That is, at a steady request rate of 0.5 req/s, there are on average about 20 requests in the system at any moment.
This means “500 online users” and “20 concurrent requests” can be true at the same time. Therefore:
- When the share of long requests is high, keep max-num-seqs conservative. Each long request occupies many KV blocks; once active sequences pile up, it triggers preemption and swapping, and throughput drops instead of rising.
- When short requests dominate, you can gradually raise it. Short requests have small KV usage; moderately increasing active sequences helps improve GPU utilization and overall throughput.
- Tuning must be driven by load tests. Fix the input/output length distribution, increase concurrency step by step, and watch the inflection points of throughput, P95/P99 latency, and memory level—rather than setting values by gut feeling.
# Illustrative config: actual parameter names should follow the inference framework's docs# Short-request shared pool: pursue throughputmax-model-len: 32768max-num-seqs: 256# Long-context pool: pursue stabilitymax-model-len: 131072 # 128Kmax-num-seqs: 16
Gap 4: A POC must leave reproducible data, not screenshots
Many POCs’ acceptance materials are a chat screenshot: the model answered the question, the format was correct, and it looked “pretty good.” But a screenshot only proves “the model can return results”; it can’t prove “the service can carry production traffic.” A proper POC should cover at least these 8 types of results:
| Acceptance item | What to collect | Focus |
|---|---|---|
| Task completion rate | The share of correct completions on the business task set | Use the business’s own test set, not a general leaderboard |
| Latency metrics | P50/P95/P99 of TTFT, TPOT, and end-to-end latency | Collected under target concurrency pressure, not single requests |
| Context tier testing | Verify 32K, 128K, 512K, near-million level tier by tier | Success rate, latency, and memory level at each tier |
| Long/short request mix | Fixed-ratio (e.g., 8:2) mixed load test and burst traffic | Whether short-request latency is dragged down by long requests |
| Memory level | Peak memory usage at each tier and concurrency | Whether the margin is enough for bursts |
| Tool call success rate | Structured output validity, parameter accuracy, multi-turn call success | Count format compliance and semantic correctness separately |
| Failure recovery | Recovery time after service restart or single-node failure | Whether requests are lost; how to degrade during recovery |
| Quantization comparison | Effect gap between full precision and low-bit quantization on the same task set | Focus on sensitive scenarios like long context and structured output |
Two of these are most easily ignored.
One is the measurement basis for latency metrics. TTFT (Time To First Token) reflects how long users wait before starting to see an answer; TPOT (Time Per Output Token) reflects the smoothness of generation. Both must be collected at P95/P99 under target concurrency pressure; single-request data has no reference value.
The other is the “availability boundary” at each tier. As context length rises tier by tier, you need to clearly answer: at which tier must concurrency be lowered? At which tier must it switch to async tasks? The boundary should be set by data, not by fiat.
Additionally, each test round should record the model version, hardware config, parallelism strategy, quantization method, load script, and test-set version. Otherwise the conclusions can’t be reproduced and can’t be compared across versions.
Gap 5: Quantization isn’t “try it first and see”; let the task set speak
Low-bit quantization (e.g., 4-bit weights, FP8 precision) can significantly reduce memory usage and improve throughput, and is a common cost-reduction approach. But quantization is lossy compression, and the loss distribution isn’t uniform. The following scenarios are often more sensitive to quantization:
- Long-context information retrieval. It needs to locate details among a large number of tokens; a subtle loss of numerical precision may amplify into localization errors.
- Structured output. Bracket matching and field hierarchy in formats like JSON are prone to errors.
- Tool calls. Parameter types, enum values, and numeric boundaries are more error-prone.
- Multi-step reasoning and code generation. Single-step deviations accumulate along the chain.
So the correct order isn’t “quantize first, deploy, and fix problems later,” but rather:
- First run the business task set on the full-precision or high-precision version to establish an effectiveness baseline;
- Then run the same task set on the low-bit version, comparing task completion rate, structured output validity, and tool call success rate;
- Focus on the degree of degradation in sensitive scenarios like long context and long output;
- Only enable the quantized version in a given scenario if the degradation is within a business-acceptable range.
At the same time, consider the limitations of the quantization method itself. Common approaches fall into two categories:
- Weight-only quantization: activations keep higher precision, so quality loss is relatively small, but memory savings—and compute acceleration—are limited;
- Weight and activation quantization together: has hardware support requirements, offers bigger gains, but higher numerical sensitivity.
Actual results also depend on the specific implementation and hardware characteristics, and must be validated by real measurement.
Gap 6: Defaulting to “the biggest model serves all requests” isn’t economical
Long-context large models excel at complex reasoning, long-document understanding, and multi-tool coordination, but a large share of online requests are actually relatively simple tasks like ordinary Q&A, information extraction, and text classification. Routing all requests to the largest-context, highest-precision model brings three problems:
- High per-request cost;
- Shared inference resources get heavily occupied by simple requests, so complex requests wait longer;
- Capacity planning gets harder.
A more robust approach is layered routing:
| Request type | Typical characteristics | Recommended strategy |
|---|---|---|
| Simple requests | FAQ, chit-chat, text classification, field extraction | Handled by a smaller model; templated or rule-based when necessary |
| Medium requests | Multi-turn Q&A, routine document summarization | Medium-sized model; enable quantization when necessary |
| Complex requests | Long-document analysis, cross-document reasoning, multi-tool orchestration | Large model + long context; full precision when necessary |
Three points when implementing:
- The routing basis must be observable. Prioritize parameters explicitly declared in the request (e.g., business scenario tag, context length); when the model must decide itself, set a confidence threshold and fallback strategy, auto-escalating low-confidence requests to a stronger model.
- Effectiveness must be regression-testable. The routing strategy itself is business logic; run regressions on the task set before and after changes to avoid “cutting cost into a higher error rate.”
- Cost must be accountable. Track call volume, token consumption, and resource usage by request type, and periodically review whether routing boundaries are reasonable.
One more point: long-context capability itself doesn’t need to be “on by default everywhere.” If the P99 context of business requests is only a few thousand tokens, planning resources for a million-level window is clearly over-investment. Capacity should be planned against the real distribution, with some margin.
3. Quick reference for the six things: a deployment checklist
| Gap | Common misjudgment | Must answer before go-live |
|---|---|---|
| KV Cache capacity | Million-context support = every request can use a million | What’s the target concurrency per context tier? Is the memory margin sufficient? |
| Accelerator selection | Big enough memory = it will run | What’s the effective throughput per unit cost at target latency? Do power and cooling suffice? |
| Scheduling parameters | max-num-seqs = user count | What’s the actual concurrency at target QPS and dwell time? How to pool long/short requests? |
| POC acceptance | Screenshots returning = validation done | Did you leave data on completion rate, per-tier latency, mixed load, failure recovery? |
| Quantization trade-offs | Quantization saves memory, so just deploy it | How much does the quantized version degrade on your own task set? Is it acceptable? |
| Model & routing | All requests use the strongest model | Are simple requests offloaded? Are routing effect and cost observable? |
Beyond these six things, the following production infrastructure is equally indispensable:
- Unified API gateway: handles authentication, tenant quotas, and rate limiting. The model service should not be exposed directly to business parties.
- Full-chain logging and auditing: records request metadata and model version, desensitizes sensitive content, and retains data per compliance requirements.
- Monitoring and alerting: covers four signal types—traffic, latency, errors, and saturation—for example QPS, P99 latency, error rate, memory level, and queue length.
- Release and rollback: supports canary release and version rollback, versions weights, configs, and images together, ensuring the old version can be restored quickly.
Whether deployed in your own data center or on a cloud platform, these general capabilities don’t differ fundamentally. Taking Baidu AI Cloud as an example, enterprises can combine products such as container service, object storage, log service, and monitoring/alerting on demand to reduce duplicated infrastructure work and focus effort on model effectiveness and capacity planning.
Conclusion
A model that runs only means the inference pipeline is connected. Production-ready means capacity is predictable, latency is guaranteed, permissions are controllable, problems are traceable, failures are recoverable, and cost is accountable. Answer these six things clearly one by one, and long-context large models can truly move from “demo-usable” to “production-usable.”
