RAG in Production: A Practical Guide to Inference Service Deployment and Reranking
RAG is often mistaken for simply "connecting an LLM to a knowledge base," but a production-grade system is a full pipeline spanning data processing, retrieval, inference, and service operations. This article breaks RAG deployment into three layers—the inference service layer, the retrieval layer, and the generation and orchestration layer—and walks through the engineering practices that matter most in production: containerized deployment, introducing a reranking model, and planning GPU memory. You will find practical guidance on launching inference services, choosing and preparing models, tuning reranking, capacity planning, performance optimization, observability, security, and cost control, along with a troubleshooting reference for common failure modes. The goal is a reusable path that takes RAG from a promising prototype to a stable production system.
When people hear about RAG, the first thing that comes to mind is often “hook up a knowledge base to a large language model.” But once you actually start building it, you realize RAG is a pipeline engineering effort spanning data processing, retrieval, inference, and service operations—not a single round of prompt tuning. Many prototypes that look impressive during a demo expose three classes of problems as soon as they reach production:
- Answers that miss the question: the retrieved document fragments are topically related but cannot answer the specific fact being asked.
- Fluent but fabricated: the model dresses up wrong information in fluent language.
- Unstable service: under a few concurrent requests, inference latency climbs, and the service can even crash due to insufficient GPU memory.
Behind these problems, inference service deployment and retrieval pipeline design are often the overlooked weak spots. This article breaks RAG deployment into three parts—“inference service layer,” “retrieval layer,” and “generation and orchestration layer”—with a focus on containerized deployment, introducing reranking models, and planning inference GPU memory. The goal is to lay out a reusable path from prototype to production.
1. From Prototype to Production: Dissecting the Full RAG Pipeline
A production-ready RAG system can usually be broken down into the following stages:
- Data processing: document parsing and cleaning, semantic chunking, vectorization, and writing vectors to the vector store.
- Retrieval: recalling candidate fragments based on the user’s question, typically combining keyword retrieval and vector retrieval across multiple paths.
- Reranking: using a finer-grained model to rescore candidate fragments and select the small set of most relevant evidence.
- Generation: organizing retrieval results into a prompt so the LLM can generate an answer and cite its sources.
- Serving and operations: a unified inference entry point, concurrency and rate limiting, monitoring and alerting, cost control, and security compliance.
Steps 3 and 4 both depend on model inference. Model deployment is infrastructure that no RAG system can avoid, and it is one of the factors that sets the ceiling on overall system quality.
2. Inference Services: The Unified Compute Foundation for RAG
2.1 Hosting Three Types of Models Under One Platform
RAG’s model requirements are not monolithic. At minimum, it needs three capabilities:
- Generation model (LLM): generates the final answer from the question, context, and instructions.
- Embedding model: converts text into vectors for building the index and serving online queries.
- Reranking model: performs fine-grained scoring on the candidate fragments returned by retrieval.
If these three model types are deployed separately with different interface styles, the adaptation cost at the orchestration layer rises quickly. A more sensible approach is to manage them centrally through a unified inference platform:
- Expose a consistent interface that follows common HTTP conventions, so the orchestration layer does not need separate adaptation logic for each model.
- Allocate GPU memory and compute resources per card to avoid resource contention.
- Provide a unified observability entry point to make performance bottlenecks easier to locate.
Tip: The open-source inference platforms and low-code orchestration platforms mentioned in the original text are just options among many. When choosing one, evaluate it across five dimensions: model support range, GPU resource management, interface compatibility, community activity, and operational cost.
2.2 Containerized Deployment: A Closer Look at Key Parameters
Deploying inference services via containers is currently a mainstream approach. Containerization brings three benefits:
- Environmental consistency: drivers, dependencies, and the model runtime are distributed together with the image, reducing “works locally, fails in production” issues.
- Resource isolation: clear, controllable boundaries for CPU, memory, and GPU device usage.
- Fast delivery: version upgrades and rollbacks happen at the image level, making canary releases easier.
A typical startup command can be abstracted as follows (the exact command may differ across inference platforms; the following only illustrates parameter semantics):
docker run \--name rag-inference \-d \-p 9997:9997 \-e HOME_DIR=/data \-v "$(pwd)":/data \--gpus all \<inference-platform-image>:<version> \<inference-platform-startup-command> -H 0.0.0.0
Each parameter is explained below:
| Parameter | Function | Engineering recommendation |
|---|---|---|
--name |
Sets the container name | Use a standardized “service name + environment” convention, e.g., rag-inference-prod |
-d |
Run in the background | Combine with process supervision or a container orchestration platform for automatic restarts |
-p host_port:container_port |
Port mapping | Map only necessary ports and use a reverse proxy for a unified entry point |
-e |
Environment variables | Used to specify the model cache directory, log level, and so on |
-v host_dir:container_dir |
Directory mount | Ensures model weights, logs, and configs are not lost when the container is rebuilt |
--gpus all |
Exposes all host GPUs to the container | In multi-GPU shared scenarios, specify exact GPU IDs to avoid resource contention |
Once deployed, the inference service usually exposes both a web console and an HTTP interface on the mapped port. You can verify service readiness in two ways:
- Open the web console to view the model list, running status, and resource usage.
- Send one minimal request through a health check or the chat interface to confirm the end-to-end path works.
2.3 Startup Modes: Interactive vs. Long-Running Services
Many inference platforms offer both an interactive startup mode and a long-running service mode:
- Interactive mode: interact with the model in a terminal-based conversation. It is well suited for validating model output quality and debugging prompts without any additional client program.
- Long-running service mode: runs continuously and listens on a specified address and port so the orchestration layer or other services can call it over HTTP.
The choice between them depends on the use case:
- For local debugging and effectiveness validation, prefer interactive mode—it is quick to start and has few dependencies.
- For business-facing production environments, always use long-running service mode, and use process supervision or container orchestration to keep it continuously online and able to self-heal from failures.
It is worth stressing that the listening address (Host) in long-running mode directly affects how widely the service is exposed:
- Set to the loopback address, the service is reachable only from the local machine—the most secure option, suitable for single-machine validation.
- Set to all network interfaces, any host that can reach that address may send requests—the least secure option.
When both need to be used together, three protective measures are mandatory:
- Add a reverse proxy in front of the inference service, and let the proxy handle authentication, access control, and TLS encryption.
- Open the service port only within an internal or private network, and restrict source addresses through security groups and firewall rules.
- Avoid exposing the inference service directly to the public internet. If it genuinely needs to be reachable externally, forward it through an API gateway that provides authentication.
3. Model Preparation: From “Finding a Model” to “Using It Well”
3.1 Three Sources for Models
There are generally three ways to obtain models:
- The inference platform’s built-in model list: most platforms maintain a curated list of adapted open-source models. Selecting the right model and completing the download finishes deployment—ideal for quick validation.
- Download channels from a model hosting platform: when the built-in list does not meet your needs, you can download weights from a public model hosting platform and import them yourself. Be mindful of connectivity and download speed in internal network environments.
- Your own model assets: for enterprises with sensitive data or deep customization needs, fine-tuned models can be stored in your own object storage or private repository and distributed over the internal network, avoiding external dependencies.
3.2 From “Downloadable” to “Usable”
Model preparation is far more than getting a successful download. It also requires attention to the following:
- Format compatibility: different models may use different weight formats and inference backends. Before importing, confirm the platform supports the corresponding loading method, and prepare the accompanying tokenizer and configuration files.
- Precision and quantization: the parameter count sets the ceiling on capability, while precision and quantization affect GPU memory usage and inference speed. These must be balanced against your memory budget and latency requirements.
- Integrity verification: interrupted downloads of large files and missing shards are common issues. Before importing, use checksums to confirm the weights are complete.
- Persistent caching: model sizes can easily reach tens of gigabytes, so repeated downloads are slow and wasteful of bandwidth. Mount the model directory to persistent storage to download once and reuse many times.
The original text points out that downloading from a model hosting platform can be faster, and this idea is worth extending. Model repositories are usually located on the public internet, while inference services are deployed in your own environment, and differences in network path show up directly as download time. In real projects, you can first pull the weights in parallel with a download tool and verify integrity, then import them into the inference platform for loading—this avoids repeated retries during online loading.
4. Reranking: The Key Link in RAG Accuracy
4.1 Why “Enough Recall” Still Produces “Wrong Answers”
In many RAG systems, the quality bottleneck appears between “retrieval recall” and “answer generation.” The reason lies in how the two model types work:
- Embedding model: typically uses a dual-tower structure, where the query and the document are each encoded into vectors and matched via a similarity calculation. Document vectors can be precomputed offline, and online retrieval only needs to encode the query and compute vector similarity. This is fast, but its accuracy is limited by “encode first, then compare.”
- Reranking model: typically uses a cross-encoder structure, concatenating the query and document and feeding them into the model together to score relevance through interaction. It captures finer-grained semantic matching, but its computational cost is far higher than that of a dual-tower model.
In practice, the candidate list from vector retrieval is often mixed with fragments that are “topically close but have no substantive answer.” Passing these directly to the LLM wastes the context window at best and induces fabrication at worst. This is exactly where reranking adds value: it does not shoulder the task of “finding candidates from massive data.” Instead, it performs fine-grained filtering among a small set of candidates, pushing the fragments that can actually answer the question to the front.
4.2 The Standard Reranking Workflow
User question↓Multi-path recall (vector retrieval / keyword retrieval, etc.) → candidate fragment set↓Result fusion (e.g., merging multiple results via reciprocal rank fusion)↓Reranking model scoring (concatenate query + candidate pairwise, score in batches)↓Sort by score and take Top-N↓Assemble the prompt → LLM generation → return with cited sources
4.3 Key Decisions Before Going Live
When introducing a reranking model, focus on these four areas:
- Number of candidates: reranking cost is proportional to “number of queries × number of candidates,” and too many candidates significantly lengthens response time. A common practice is to first recall dozens of candidates with vector retrieval, then keep only the top few after reranking for the context.
- Batch processing: submitting multiple candidate fragments as a single batched request is more efficient than calling them one by one, and it noticeably reduces overall latency.
- Model choice: reranking models also differ in parameter count and language ability. For Chinese-language scenarios, prioritize models with strong Chinese support. Larger models generally offer higher accuracy but come with greater latency and GPU memory overhead, so balance this against your tolerance for latency.
- Resource isolation: reranking requests are often dense and short. If they share a GPU with the generation model, they may contend for resources, so it is advisable to deploy them separately or allocate GPU memory independently.
5. Production Readiness: From “Runs” to “Runs Reliably”
5.1 Capacity Planning: The GPU Memory Budget Determines Service Scale
GPU memory is the primary constraint on inference services. Plan it by estimating three parts:
- Model weights: roughly “parameter count × bytes per precision unit.” With FP16, for example, each parameter takes about 2 bytes, so a model with tens of billions of parameters needs about 20 GB of GPU memory for weights alone.
- KV cache: the attention mechanism caches the key-value pairs of historical tokens, and its footprint grows with context length and the number of concurrent requests. In long-document scenarios, this often consumes more GPU memory than the weights themselves.
- Runtime overhead: memory reserved by the framework itself, the CUDA context, and fragmentation space, among others.
The practical recommendation is to run a load test with the target model and typical inputs before going live, record GPU memory usage and response time at different concurrency levels, then set the maximum concurrency and number of instances accordingly, leaving some headroom for peaks.
5.2 Performance Optimization: Latency, Throughput, and Cost
- Batching and scheduling: modern inference frameworks generally support dynamic batching to combine multiple requests for higher throughput. The orchestration layer should keep request formats uniform and length distributions relatively balanced to reduce wasted computation within a batch.
- Cache reuse: for high-frequency questions, apply semantic caching so similar questions directly reuse historical answers, avoiding repeated inference.
- Streaming output: when the generation model supports streaming returns, prefer streaming so users see the first token sooner and perceived latency drops significantly.
- Quantization and pruning: use quantized weights when accuracy is acceptable to reduce GPU memory usage and improve throughput. At the same time, monitor changes in output quality after quantization to avoid “saving memory but losing quality.”
- Context trimming: control the number and length of fragments sent to the model. This saves token cost and reduces interference from irrelevant information.
5.3 Observability: Making Problems Detectable and Diagnosable
A production RAG system needs monitoring across three layers:
- Resource layer: GPU utilization, GPU memory usage, CPU and memory, and disk I/O.
- Service layer: request volume, error rate, P95/P99 latency, queue time, and model loading time.
- Business layer: retrieval hit rate, Top-N hit rate after reranking, answer adoption rate, and user feedback.
It is advisable to generate a global trace ID at the request entry point and carry it through retrieval, reranking, and generation to make problem diagnosis easier.
5.4 Security and Compliance: The Baseline of Keeping Data In-House
Enterprise knowledge bases often contain sensitive information, so a RAG system should ensure:
- Deployment form: both the inference service and the vector store are deployed in your own environment or a private network, so model weights and business data never pass through third-party services.
- Encryption in transit and at rest: enable TLS for service-to-service calls, and encrypt sensitive documents and vector data at rest.
- Access control: unified authentication and role-based authorization, with inference interfaces open only to the internal network.
- Content safety: run compliance checks on both user input and model output to avoid generating inappropriate content.
- Audit trails: retain logs of key operations to meet compliance traceability requirements.
5.5 Cost Control: GPU Is the Biggest Variable
The cost of a RAG system is concentrated mainly in GPU resources and can be optimized in the following ways:
- Time-based reuse: release or downscale resources during off-peak hours and scale back up during peak hours.
- Model tiering: handle simple questions with a small model and route complex questions to a large model, forming a layered strategy of “small model first, large model as fallback.”
- Elastic scaling: configure autoscaling based on monitoring metrics to avoid provisioning resources for peak capacity year-round.
- Cloud hosting: if a team wants to reduce the burden of self-operations, it can build a self-hosted inference environment using the container services, object storage, logging, and monitoring and alerting capabilities of cloud providers such as Baidu AI Cloud—keeping model weights and business data in its own account at all times while benefiting from elastic compute and unified operations.
6. Common Issues and Troubleshooting
| Symptom | Possible cause | Troubleshooting direction |
|---|---|---|
| Port unreachable after the container starts | Port mapping not in effect, incorrect listen address, firewall blocking | Check startup parameters, container logs, and network rules |
| Model loading fails or times out | Incomplete weights, unsupported format, insufficient GPU memory | Verify file integrity, confirm the model format, check GPU memory usage |
| Slow generation, high time to first token | Queuing from high concurrency, overly long context, streaming not enabled | Check queue time and context length, enable streaming output |
| Answers inconsistent with the knowledge base | Poor retrieval quality, missing reranking, unconstrained prompts | Check retrieval hits, introduce reranking, strengthen “answer based on context” instructions |
| Out of memory (OOM) | Concurrency exceeds budget, KV cache growth, memory occupied by other processes | Limit maximum concurrency, compress context, dedicate the GPU |
7. Conclusion
The effectiveness of a RAG deployment depends on the weakest link across the entire pipeline. Containerized deployment of the inference service solves the problem of “how to get models running.” Introducing a reranking model solves the problem of “whether retrieval results are accurate.” Capacity planning and observability determine whether the system can run stably over the long term.
Back to the original question—“how should RAG be built?” The answer is not a single model or a single kind of prompt, but an engineering system that can be decomposed and measured: first build a solid foundation for the inference service, then use reranking to raise retrieval accuracy, and finally hold the system together with monitoring, security, and cost controls. Because these pieces interlock, RAG can genuinely move from demo to production.
