LLM Deployment: Six Easy-to-Miss Gaps Between Running and Production-Ready
A large language model that starts successfully, loads weights, and returns responses only proves that the technical chain is viable, not that the service is production-ready. This article examines six common gaps that teams often overlook during LLM deployment: confirming production readiness, managing KV cache pressure from long contexts, choosing appropriate hardware, understanding max-num-seqs and concurrency limits, validating a POC with measurable results, and avoiding unnecessary use of overly large models. It also highlights checks for real task quality, concurrency, mixed request lengths, access control, auditability, monitoring, and resilience.
1. Model startup success is only the first step
In many enterprise LLM rollout processes, a common mistake is treating “the service can start” as equivalent to “it is ready for production.” In reality, a model process launching normally, the inference framework loading successfully, weights being read into GPU memory, and a basic inference API returning results only prove that the technical chain is working. These checks validate feasibility, not usability.
A production-oriented inference service must also pass several checks:
- Real task performance: On real business data, answers must be accurate and meet formatting requirements.
- Concurrency capacity: Latency and throughput must remain controllable when multiple users access the service at the same time.
- Mixed long and short requests: The system must remain stable when very long contexts and short questions are interleaved.
- API access control: Authentication, quotas, and rate limiting must be in place.
- Logging and auditability: Requests and responses must be traceable and retainable.
- Monitoring and alerting: Latency, GPU memory usage, and error rates must be continuously observed.
- Service resilience: Process restarts, version rollback, and exception recovery must be verified.
Without these checks, it is still only a “test service,” not a “production service.”
2. Long context mainly stresses KV Cache in GPU memory
Many teams assume that if a model supports million-token context, every request can simply use the full window. This is an expensive and dangerous misunderstanding. To understand it, it is important to distinguish two types of GPU memory usage:
- Weight usage: The model parameters themselves; this is mostly fixed after loading.
- KV Cache: Dynamically allocated during inference to store historical attention keys and values; it grows with input length, output length, and the number of concurrent requests.
A request approaching one million tokens can therefore consume a large amount of cache. When multiple long requests arrive at the same time, GPU memory can be exhausted quickly, leading to queuing, sharp drops in throughput, and even OOM, which may crash the service.
For production environments, a more reasonable approach is tiered scheduling:
- Regular short requests: Use a shared resource pool and prioritize high throughput.
- Medium context, approximately 32K–128K: Set separate quotas to avoid crowding out the shared pool.
- Long context, above 128K: Route requests to a dedicated long-context pool.
- Near the limit, close to 1M: Use a low-concurrency dedicated queue, trading throughput for stability.
The essence of this tiering is to isolate scarce GPU memory resources by request value and size, preventing a few extremely long requests from dragging down the entire service.
3. Hardware selection is not simply “replace old with new”
Different data-center GPUs are often misunderstood as simply “newer cards replacing older cards.” In long-context inference scenarios, selection requires more dimensions. Single-card or single-node memory capacity determines the model size and context limit that can be supported, but production performance also depends on:
- Task type: General question answering, long-document understanding, or code generation.
- Context length distribution: Whether requests are generally long or short.
- Concurrency scale: The number of active requests at peak times.
- Output length: The generation phase also consumes GPU memory and compute.
- Tool calls: Function calling and retrieval-augmented generation can introduce additional overhead.
- Inference framework: Different frameworks have noticeably different memory management and scheduling strategies.
- Intra-node interconnect: Communication bandwidth between GPUs directly affects tensor parallel efficiency.
- Data center conditions: Power supply and cooling capacity are often overlooked hard constraints.
In other words, selection cannot be based only on “whether GPU memory is enough.” It must ask whether latency and throughput can be stably met under the target workload. The same GPU may perform very differently when serving short requests versus million-token contexts.
4. max-num-seqs is not the number of online users
During deployment, teams often need to configure two key parameters: maximum context length and maximum number of concurrent sequences. Many teams intuitively treat the latter as “how many users can be online at the same time,” which is a typical conceptual mistake.
- Maximum context length: Controls the longest token window a single request can use.
- Maximum number of concurrent sequences: Controls how many sequences can be active at the same time within one scheduling batch.
It measures how many sequences the scheduler can process in one batch, not how many users are accessing the service. Real concurrency also depends on queue length, average request length, output length, and rate-limiting policies.
In practice, the tuning logic is roughly as follows:
- If long requests make up a large proportion, set the concurrency limit conservatively; otherwise GPU memory can be overwhelmed.
- If short requests dominate, gradually increase concurrency to push throughput higher.
Below is an illustrative configuration snippet. Different frameworks use different parameter names; this only expresses the concept.
# Illustrative configuration; not specific to a product; conceptual onlymax_context_length: 131072 # Maximum context window per requestmax_active_sequences: 64 # Upper limit of active sequences per scheduling batch (not online users)memory_utilization_target: 0.90 # GPU memory utilization target
The key is to understand that these parameters constrain one another. Increasing one often comes at the cost of another.
5. A POC cannot rely on screenshots
During the proof-of-concept stage, many teams are satisfied with screenshots showing that “the model returns correct results.” This is far from proving that the system is ready for production. A qualified POC should leave behind quantifiable and reproducible conclusions, including at least:
- Real task completion rate: Accuracy and usable rate on the business data set.
- Performance metrics: Time to first token (TTFT), time per output token (TPOT), P95 latency, and P99 latency.
- Tiered stress testing: Test at context levels such as 32K, 128K, 512K, and close to 1M.
- Mixed long and short requests: Simulate mixed workloads similar to real traffic.
- GPU memory level: GPU memory usage curves under different workloads.
- Tool-call success rate: Whether function calls and structured outputs are correct.
- Resilience testing: Whether service restarts and exception recovery behave as expected.
- Precision comparison: Quality differences between quantization precisions, such as FP8 and lower-bit quantization.
Only when these results are documented into reports can the POC be considered truly “validated.” Screenshots prove that it can run; data proves that it can handle load.
6. Complex models are not suitable for every task
The final easily overlooked point is that a powerful model with long context does not mean it should be used for every scenario. For many high-frequency, simple tasks—such as ordinary question answering, intent recognition, and format conversion—using a very large model often leads to three problems: high cost, high latency, and inefficient use of resources.
A more reliable engineering practice is task tiering:
- Ordinary question answering and simple classification should be handled by lighter models.
- Complex reasoning, long-document understanding, and multi-step tool calls should be handled by more capable large models.
Through routing and tiered scheduling, expensive long-context capabilities can be reserved for requests that truly need them. This can significantly reduce costs while improving overall system throughput and response speed.
Conclusion
The gap between “can run” and “can go live” is not a piece of code, but a set of engineering capabilities: production readiness checks, KV cache management, hardware selection, concurrency tuning, POC validation, and task tiering. Implementing these items one by one is what allows a model to move from a “lab demonstration” to a “production service.”
