Running an LLM Isn't the Same as Shipping It to Production: 6 Things Enterprises Most Often Miss in Deployment

    A model that starts up successfully has only proven its inference path works—not that it's ready for production. This article walks through six aspects that enterprise LLM deployments most often overlook: defining real launch criteria beyond a working inference call, understanding the true cost of long context (KV Cache), selecting compute cards, correctly interpreting concurrency parameters, running a POC that produces evidence rather than screenshots, and routing simple tasks away from expensive models. It closes with a seven-point production readiness checklist so teams can take a model from "it runs" to "it can handle load, stay under control, and roll back safely."

    In many teams’ day-to-day work, “getting the model running” is treated as the finish line: the weights finish loading, the endpoint returns normally, a screenshot is shared with the team, and deployment is declared done. But once the model is actually wired into the business, the real problems surface. A single long request drags down the whole instance, P99 latency climbs several-fold the moment concurrency ramps up, restarts take ten-plus minutes, and there are no logs to check when something breaks.

    A successful startup only proves the pipeline is connected. Production is where you find out whether the service can reliably deliver under real traffic, real tasks, and real failures.

    The six areas below each tend to get missed, and they aren’t isolated—they interlock. Context tiering sets the concurrency ceiling, the concurrency ceiling drives selection and capacity planning, and all of it ultimately has to show up in your POC evidence and go-live checklist.

    1. Launch Criteria: A Working Inference Path Is Just the Starting Line

    Getting the service to start usually means only four things: the driver and operator-library versions match, the framework correctly detects the hardware, the weights load in full, and the basic inference endpoint returns normally. All of this is connectivity verification. Production demands at least seven more checks:

    • Real task quality: Evaluate completion rate against ground-truth answers using real business data—not just whether the model returns some text.
    • Multi-user concurrency: Run many requests in parallel and confirm latency stays under control.
    • Mixed long and short requests: Verify that short Q&A and long-document requests don’t drag each other down when they run together.
    • API access control: Ensure authentication, quotas, and tenant isolation are in place. The API should never be left wide open beyond the internal network.
    • Logging and audit: Every call—who invoked what, when, how many tokens it consumed, and what the result was—must be traceable.
    • Monitoring and alerting: Latency, error rate, GPU memory watermark, and queue depth all need metrics and thresholds.
    • Restart and rollback: A failed instance must recover quickly, and a bad new version must be able to roll back to the previous one in time.

    The bar is simple: if any one of these has no closed loop, the service is still just a test service. One more caution—loading a large model’s weights often takes several minutes. If the service doesn’t shut down gracefully, rolling updates will produce obvious request failures during the transition.

    2. The Real Cost of Long Context: KV Cache

    “Supports a very long context” does not mean “every request should be pushed straight to the limit.” The key to understanding this is separating two kinds of GPU memory overhead:

    • Weights are fixed overhead. Once the model is loaded, they occupy GPU memory regardless of how many requests arrive.
    • KV Cache is dynamic overhead. The inference engine caches the Key/Value of historical tokens to avoid recomputation. Cache usage grows roughly linearly with the number of tokens, then stacks up again with the number of concurrent sequences.

    You can estimate the order of magnitude with the formula below:

    1. KV Cache bytes ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_element × num_tokens × num_concurrent_sequences
    2. # The 2 stands for one copy each of K and V; with half precision, bytes_per_element is 2

    This is only illustrative. Some models use grouped-query attention, so the actual number of KV heads is smaller—always plug in values based on your own model’s structure. Even so, the problem becomes obvious. Suppose a model has 60 layers, 8 KV heads, a head dimension of 128, and uses half precision: a single token needs about 240KB. A single near-limit, very-long-context request can consume cache on the order of hundreds of GB, and that’s before concurrency stacks on top. So the typical failure modes emerge:

    1. One very long request monopolizes cache and compute while other requests wait in line, and P99 spikes;
    2. Multiple long requests arrive at once, the cache fills up almost immediately, and the scheduler preempts or even terminates sequences;
    3. Cache fragmentation accumulates, the GPU memory watermark keeps climbing, and it finally hits OOM.

    The more sensible approach is tiered management:

    Tier Context length Queue strategy Concurrency target
    Shared pool Normal short requests Shared queue, oriented toward high throughput As high as possible
    Quota tier 32K–128K Quota allocated per tenant Medium, evaluated dynamically against memory
    Long-context pool Above 128K Dedicated instances, dedicated queue Low
    Dedicated queue Near the limit Reservation-based or queued, time- and volume-limited Very low, typically single digits

    After tiering, instance isolation becomes the first line of defense: ordinary requests won’t be slowed down by a single very long input. Quotas and queuing become the second line: long requests over the quota enter a queue or get degraded instead of crushing the instance outright. The shift from “how long can we open it” to “how many concurrent streams can each tier serve stably” is the dividing line between long context as a demo and long context in production.

    3. Choosing Compute Cards: It’s Not Simply One Replacing Another

    A “single-node, 8-card GPU with large memory” is a reasonable starting reference for medium-precision, long-context deployments. To fully cover near-limit context along with higher concurrency, you need a higher tier of cards with more memory capacity, more bandwidth, and stronger intra-node interconnect. But selection can’t focus only on the memory number. At minimum, you also have to answer eight questions:

    1. Task type: Code and tool-calling tasks are more sensitive to output quality and latency; summarization and document Q&A lean more heavily on long context.
    2. Context length: Long context sharply amplifies cache overhead, and whether the cache can be sharded across multiple cards directly determines which cards are viable.
    3. Concurrency scale: Concurrency determines the total cache and the compute demand.
    4. Output length: Output is generated token by token, and long outputs keep sequence slots occupied, putting heavy pressure on memory bandwidth.
    5. Tool calls: Structured-output parsing and retries on failure both amplify the real request volume.
    6. Inference framework: How mature a framework’s operator support is for newer hardware often affects real throughput more than the hardware’s peak specs.
    7. Intra-node interconnect: Both tensor parallelism and pipeline parallelism involve cross-card communication, and insufficient interconnect bandwidth noticeably degrades scaling efficiency.
    8. Data-center conditions: Per-rack power, power redundancy, and cooling capacity determine whether high-density cards can actually be deployed.

    Selection and business planning should proceed together, not the other way around: first define the task profile and capacity targets, then work backward to card type, card count, and node count, and finally validate the assumptions with POC data. If a team lacks the conditions to build its own data center, it can also use on-demand GPU compute in the cloud, together with general-purpose cloud services such as object storage, logging, and monitoring/alerting, to quickly build a full path from validation to launch and avoid a large one-time capital investment.

    4. Concurrency Parameters Aren’t Online User Counts

    Two parameters in mainstream inference engines like vLLM are often misunderstood:

    • max-model-len: the maximum context window for a single sequence—it decides “how long the input can be at most.”
    • max-num-seqs: the upper limit on active sequences in a single scheduling batch—it decides “how many sequences can be advanced at the same time.”

    The key point is this: max-num-seqs is not the number of online users, and the number of online users is not the number of concurrent requests either. The time users spend reading a page, thinking, and typing doesn’t occupy a sequence slot on the server. You can estimate the requirement with Little’s Law from queueing theory:

    1. Required sequence slots ≈ request arrival rate × average residence time
    2. Average residence time ≈ prefill time + output tokens × per-token generation time

    Take an illustrative example: 10 requests per second, an average output of 512 tokens, and 20 milliseconds per generated token gives an average residence time of about 10.2 seconds and a slot requirement of roughly 102. If the average output grows to 4096 tokens, the residence time approaches 72 seconds and the slot requirement approaches 720.

    Real concurrency is also affected by queue depth, rate-limiting policy, the distribution of long versus short requests, and gateway-layer timeout settings. A long request occupies a slot for a long time. Set the parameters too high and the cache gets filled instantly with frequent preemption; set them too low and GPU memory sits idle with insufficient throughput. The right tuning rhythm is: set conservatively and rely on queuing when long requests dominate; combine load testing and gradually raise the value when short requests dominate. The final value has to come from measurement, not from a gut feeling.

    5. A POC Isn’t Just Screenshots

    A qualified POC should leave behind at least the following evidence:

    • Real task completion rate: Evaluate on a business sample set—extraction tasks, classification tasks, code generation, and so on—and compare against the expected baseline.
    • Latency metrics: TTFT (time to first token), TPOT (per-token generation time), P95, and P99—not just the average.
    • Tiered context testing: Load-test each tier—32K, 128K, 512K, and near the limit—and record latency and throughput at different concurrency levels.
    • Long/short mix: Simulate the ratio of long to short requests seen in real traffic and observe the mutual interference.
    • GPU memory watermark: Record weights, cache, fragmentation, and peak usage separately.
    • Tool-call success rate: Structured-output parse success rate, retry rate, and timeout rate.
    • Failure recovery: Restart duration, traffic recovery after a node is drained, and the request retry strategy.
    • Precision comparison: The quality gap before and after quantization, for example the difference on task metrics between medium precision and a lower-bit quantization.

    For the test scripts, it’s best to use the service’s basic inference endpoint directly and keep the call logic simple and transparent:

    1. import asyncio
    2. import json
    3. import time
    4. def build_payload(prompt: str, max_tokens: int, stream: bool) -> dict:
    5. # Fields are illustrative; refer to the actual API
    6. return {"prompt": prompt, "max_tokens": max_tokens, "stream": stream}
    7. async def run_scenario(client, base_url: str, prompt: str, max_tokens: int, concurrency: int):
    8. # Time-to-first-token and per-token generation time can only be measured with streaming responses; this non-streaming example shows overall latency
    9. async def one_call():
    10. start = time.perf_counter()
    11. resp = await client.post(base_url + "/v1/completions",
    12. json=build_payload(prompt, max_tokens, False))
    13. data = await resp.json()
    14. latency = time.perf_counter() - start
    15. return latency, len(data.get("choices", [{}])[0].get("text", ""))
    16. start_all = time.perf_counter()
    17. results = await asyncio.gather(*[one_call() for _ in range(concurrency)])
    18. elapsed = time.perf_counter() - start_all
    19. latencies = sorted(r[0] for r in results)
    20. p95 = latencies[int(len(latencies) * 0.95)]
    21. print(json.dumps({
    22. "concurrency": concurrency,
    23. "elapsed": round(elapsed, 2),
    24. "p95_latency": round(p95, 2),
    25. "throughput_tokens_per_s": round(sum(r[1] for r in results) / elapsed, 1)
    26. }, ensure_ascii=False))

    Repeat each scenario at least three times and take the median, while collecting GPU memory and queue metrics at the same time. Proving only that “the model can return a result” is not the same as completing validation. The output of a POC should be a decision report with data and conclusions: how much concurrency this configuration can support, at what context length and latency level, and what it will cost to get there.

    6. Not Every Task Needs a Complex Model

    Large, complex models have an edge in reasoning, tool calling, and long-document understanding, but ordinary Q&A, intent classification, and templated generation don’t necessarily warrant the same investment. A more robust approach is to tier:

    1. Simple requests: FAQ Q&A, intent classification, format conversion—hand these to a lightweight model or a traditional rules engine.
    2. Medium requests: Ordinary business Q&A, document summarization—hand these to a mid-sized model.
    3. Complex requests: Multi-step reasoning, tool orchestration, very-long-document processing—only these get routed to a large, long-context model.

    The routing decision can be made at the gateway or application layer. Illustrative logic:

    1. # Pseudocode: classify first, then route by complexity
    2. def route(request):
    3. if is_faq(request) or is_format_convert(request):
    4. return "light_model"
    5. if request.need_tools or request.input_tokens > 64 * 1024:
    6. return "large_context_model"
    7. return "mid_model"

    The benefits of tiering are direct:

    • Cost: Most traffic is served by low-cost models, and expensive compute only serves the requests that truly need it.
    • Stability: Under high load, the queuing pressure on complex-model instances drops noticeably, and critical tasks get served with priority.
    • Scalability: When you later replace or upgrade a model in one tier, the blast radius stays contained.

    7. The Go-Live Readiness Checklist

    Consolidating the six areas above into one actionable checklist:

    1. Quality: The task completion rate on real samples meets the preset threshold.
    2. Capacity: Estimate slot requirements with Little’s Law, and combine them with the tiered context strategy to set a concurrency ceiling for each pool.
    3. Latency: P95/P99 meet business requirements, with no avalanche-style degradation in mixed long/short scenarios.
    4. Stability: GPU memory watermarks have alert thresholds, and restart and version-rollback drills have passed.
    5. Security: API authentication, quotas, tenant isolation, and logging/audit are all enabled.
    6. Observability: Latency, throughput, error rate, queue depth, and GPU memory watermark are all wired into monitoring and alerting.
    7. Cost: The routing strategy is live, and simple traffic no longer runs on high-cost models.

    A model that can start only proves the pipeline is connected. Going to production tests your combined command of capacity, concurrency, long-context tiering, quantization trade-offs, observability, and rollback. Close out these seven items one by one, and only then does the model move from “it runs” to “it can handle the load, stay under control, and roll back safely.”