Self-Hosting Open-Source LLMs for Code Generation: Hardware, Inference, and Quality Evaluation

    Commercial LLM APIs offer transparent pricing, but bills can grow quickly as call volume rises, leading many teams to consider self-hosting open-source models. This article covers hardware planning, inference engine optimization, and evaluation practices for code generation quality. It explains how to estimate model weight and KV cache memory, when to use tensor and pipeline parallelism, how to prevent out-of-memory failures, and how to assess quantization and semantic caching. It also highlights which numbers must be measured and which conclusions require explicit conditions.

    Using closed commercial LLMs for code generation offers transparent per-token pricing, but bills can grow quickly once call volume increases. Many teams therefore consider self-hosting open-source models. This article summarizes the key steps for deployment and evaluation, and clarifies which numbers must be measured and which conclusions require stated conditions.

    Whether self-hosting is cost-effective depends on average monthly call volume, GPU utilization, and operations staffing. At low call volumes, fixed server and operations costs may exceed API fees. Estimate the monthly cost of both options based on call volume before deciding whether to proceed.

    Hardware Planning

    GPU Memory Estimation

    The GPU memory occupied by model weights can be estimated as:

    weight memory ≈ parameter count × bytes per parameter

    FP16 or BF16 uses 2 bytes per parameter, INT8 uses 1 byte, and INT4 uses about 0.5 bytes. For a 70B-parameter model, FP16 weights require about 140 GB, while INT4 weights require about 35 GB. This result does not yet include other overhead.

    KV cache memory must be calculated separately:

    KV cache ≈ 2 × number of layers × number of KV heads × head dimension × context length × number of concurrent requests × bytes per element

    The coefficient 2 accounts for the separate K and V caches. KV cache grows linearly with context length and concurrency. In long-code-context and high-concurrency scenarios, it is often the main consumer of GPU memory.

    Multi-Node Parallelism

    When a single machine does not have enough GPU memory, tensor parallelism can split computation within a layer across multiple GPUs. Tensor parallelism involves frequent communication and is best kept within one server connected by NVLink. Across nodes, pipeline parallelism is commonly used, splitting the model by layer. Inter-node communication volume is relatively lower, but pipeline bubbles can affect latency.

    Network bandwidth between nodes should be measured, for example with nccl-tests, and InfiniBand or RoCE configuration should be confirmed as active. If bandwidth is insufficient, cross-node communication becomes a bottleneck, and actual throughput can fall far below expectations.

    Preventing Out-of-Memory Failures

    • Reserve some free GPU memory per card as a starting point, for example 10% to 15%, then adjust based on measurements.
    • Limit the maximum context length and maximum concurrency. When limits are exceeded, return a clear error instead of letting requests overwhelm GPU memory.
    • Continuously record the memory watermark for each GPU and set alert thresholds.
    • Truncate or reject overly long inputs and notify users at the API layer.

    Inference Engine and Optimizations

    Inference Engine

    For example, vLLM supports PagedAttention and continuous batching. PagedAttention manages KV cache in pages, reducing memory fragmentation and waste. Continuous batching admits a new request as soon as another finishes, improving GPU utilization. Together they can improve throughput, but they do not reduce the total KV cache required by an individual request.

    During deployment, record the engine version, startup parameters such as tensor-parallel-size, max-model-len, and max-num-seqs, and the hash of the model weights so the setup can be reproduced.

    Quantization

    INT8 and INT4 quantization can reduce memory usage and may improve inference speed. The effect of quantization on code quality depends on the model, quantization method, and task type, and cannot be generalized. After quantization, compare the quantized model with the original on the same evaluation set, checking pass@1, compile pass rate, unit test pass rate, and P95 latency. Teams should define an acceptable degradation threshold in advance based on business risk, and versions that exceed it should not be released. Without comparison data, teams should not claim that quality does not drop noticeably after quantization.

    Semantic Caching

    Semantic caching can reuse results from highly similar historical requests. It fits repetitive scenarios such as common utility functions and standard library usage. Two cautions apply.

    First, similar natural-language descriptions may correspond to different code requirements. Incorrect reuse returns mismatched code, so the similarity threshold should be tuned on an evaluation set.

    Second, the cache key should include the model version, prompt template, and decoding parameters to avoid returning stale results after a model upgrade.

    On a cache hit, latency decreases, but the size of the reduction depends on the hit rate, input and output length, and concurrency.