Private LLM Deployment for AI Coding: A Practical Guide to GPU Selection, Inference Serving, Evaluation, and Cost
This guide helps engineering teams decide whether to self-host large language models for code generation instead of relying on commercial coding assistants. It explains when private deployment is worth pursuing, how to size GPUs and cope with limited VRAM, how to design inference APIs with authentication, timeouts, rate limits, and error codes, how to build an executable evaluation and acceptance process for generated code, and how to compare the total cost of self-hosting, cloud inference, and SaaS. It also includes an FAQ section and a best-practices checklist.
Lead with the conclusion: which teams should prioritize self-hosting?
If a team meets all of the following conditions, it can prioritize evaluating private deployment: stable and steadily growing call volume, sensitive code and data, existing or rentable GPU resources, and basic operations and model-tuning capability. Conversely, if the team is small, call volume fluctuates significantly, a fast time to market is required, or there is no dedicated ops staff, it is better to first validate value with cloud inference, model-hosting services, or mature SaaS tools, and then decide whether to migrate to a self-owned environment.
To be clear, private deployment is not a “natural replacement for commercial services.” Rather, it establishes a sustainable, evolvable foundational capability for the team under the premises of controllable cost, controllable data, and auditable compliance.
1. Background: procurement costs are rising — is self-hosting more cost-effective?
As intelligent coding tools become widespread across coding, review, test generation, and documentation completion, more and more teams are paying attention to how large language models affect R&D efficiency. However, some commercial coding assistants charge by subscription, per seat, or per call. As the number of developers grows and call frequency increases, costs accumulate quickly. For small and mid-sized enterprises, outsourcing teams, or organizations with strict internal code-security requirements, procurement cost directly affects how broadly a tool can be rolled out.
As a result, many engineering teams ask: can we deploy a large language model on our own GPU resources and provide stable code generation? The value of this path lies in:
- Predictable long-term call costs;
- Code and prompts that never need to leave the internal network;
- Tuning for internal frameworks, private libraries, and standards;
- Integration with the R&D pipeline, code-hosting platforms, and CI systems.
But self-hosting also has prerequisites: it requires investment in hardware planning, service deployment, model evaluation, security governance, and ongoing operations.
2. How do you choose a GPU based on model size? What if VRAM is insufficient?
The first bottleneck in LLM inference is usually not “can it run,” but whether there is enough VRAM. VRAM demand is primarily determined by model weights, the KV Cache, concurrent requests, context length, and framework overhead.
A simplified estimation method is as follows:
- FP16/BF16: roughly 2 GB of weight VRAM per billion parameters;
- INT8: roughly 1 GB of weight VRAM per billion parameters;
- INT4: roughly 0.5 GB of weight VRAM per billion parameters, though you must also account for quantization-framework overhead in practice.
For example:
| Model size | Precision | Rough weight VRAM | Recommended starting VRAM |
|---|---|---|---|
| 7B | FP16/BF16 | ~14 GB | 24 GB+ |
| 13B | FP16/BF16 | ~26 GB | 40 GB+ |
| 32B | FP16/BF16 | ~64 GB | 80 GB+ |
| 7B / 13B | INT8 / INT4 | Significantly lower | Validate with evaluation |
Note: the table above estimates weights only and does not include the KV Cache. The longer the context and the higher the concurrency, the larger the KV Cache. For example, long-context code understanding, cross-file completion, and batch code review all increase VRAM usage significantly.
When selecting a GPU, focus on four items:
- Total VRAM: determines whether you can load the target model and context;
- Memory bandwidth: affects long-text generation and multi-turn conversation experience;
- Inter-card interconnect: for multi-card tensor parallelism, interconnect bandwidth affects throughput;
- Power consumption and cooling: determine deployment feasibility for a data center, workstation, or edge node.
If VRAM is insufficient, consider:
- Using INT8/INT4 quantization, but performing code-correctness regression testing;
- Reducing context length and maximum generation length;
- Reducing concurrency or enabling a request queue;
- Using multi-card tensor parallelism;
- Switching to a small-to-medium model better suited to code tasks;
- Separating offline evaluation from online serving.
3. How do you deploy the inference service? How should you design interfaces, auth, timeouts, and error codes?
The core task of private deployment is turning the model from “loadable” into “stably callable.” The general workflow includes: installing drivers and acceleration libraries, loading model weights, starting the inference service, wrapping the API, and integrating with business systems.
3.1 Interface protocol recommendations
For internal services, you can use HTTP RESTful interfaces, and TLS is recommended in production. If internal call volume is high, you can also evaluate gRPC. Use JSON for requests and responses, and keep fields clear.
A typical request could be:
POST /v1/inference HTTP/1.1Authorization: Bearer <internal-token>Content-Type: application/json{"model": "local-code-llm","messages": [{"role": "system", "content": "You are a code assistant. Prioritize executability and security."},{"role": "user", "content": "Please implement a cached function decorator in Python."}],"temperature": 0.2,"max_tokens": 2048,"stream": false}
The response should include traceable fields:
{"request_id": "req-20260620-001","content": "generated content","finish_reason": "stop","usage": {"prompt_tokens": 320,"completion_tokens": 512,"total_tokens": 832}}
If you use an OpenAI-compatible interface, the generated content is typically located at:
data["choices"][0]["message"]["content"]
If your self-hosted inference framework returns a content field directly, you can extract it from the top level. Callers should not assume there is only one structure; an adaptation layer is recommended.
3.2 Authentication, timeouts, and error handling
A private service cannot assume that “the internal network is secure by default.” At a minimum, it should include:
- Authentication: internal tokens, OAuth, mTLS, or unified gateway authentication;
- Timeouts: a connect timeout of 1 to 3 seconds, and a read timeout set based on generation length, e.g. 60 to 120 seconds;
- Rate limiting: QPS or concurrency caps per user, team, and interface;
- Error codes: clear codes for parameter errors, unauthorized access, rate limiting, model unavailability, timeouts, and so on;
- Logging: record
request_id, latency, token counts, and error type, while avoiding logging full source code and sensitive prompts.
Common error codes to reference:
- 400: parameter error or invalid message format;
- 401: unauthenticated;
- 403: insufficient permissions;
- 429: concurrency or quota limit exceeded;
- 500: internal inference-service error;
- 503: model not ready or temporarily unavailable;
- 504: upstream or model-generation timeout.
3.3 Security deployment checklist
Before going live, check the following item by item:
- Weight verification: record and verify the SHA256 or a signature, and avoid weight files of unknown origin;
- Dependency version locking: lock driver, CUDA, inference framework, and Python package versions to avoid behavior changes caused by upgrades;
- Internal network isolation: isolate the inference service from the public internet, and allow access only from controlled network segments;
- API authentication: every call must carry valid credentials, and unauthenticated exposure is prohibited;
- Log redaction: do not log full code, keys, internal links, or user-sensitive information;
- Resource isolation: use containers or virtual environments to limit CPU, memory, VRAM, and process count;
- Audit trail: retain call audits, anomaly alerts, and version-change records.
A more robust call example is as follows:
import requestsdef call_llm(api_base, prompt, token, timeout=(3, 120)):headers = {"Authorization": f"Bearer {token}"}payload = {"model": "local-code-llm","messages": [{"role": "user", "content": prompt}],"temperature": 0.2,"max_tokens": 2048,}resp = requests.post(f"{api_base}/v1/inference",json=payload,headers=headers,timeout=timeout,)resp.raise_for_status()data = resp.json()content = data.get("content")if content is None and "choices" in data:content = data["choices"][0]["message"]["content"]if content is None:raise ValueError("No generated content found in the response")return content
4. How do you evaluate code generation quality? How do you accept it?
The biggest concern with self-hosted models is not “can it generate code,” but “can the generated code enter the R&D process.” Therefore, evaluation cannot rely on a single subjective experience; instead, you should establish a quantifiable, regression-capable evaluation system.
4.1 Building the evaluation set
The evaluation set should cover real R&D tasks, such as:
- Function completion;
- Bug fixing;
- Unit test generation;
- Code explanation;
- Writing regular expressions and scripts;
- Internal framework usage;
- Security-sensitive code generation.
Each sample should ideally include: a task description, input code, expected behavior, test cases, and acceptance criteria.
4.2 Prioritize executable verification
For code tasks, the most effective approach is not just looking at syntax or text similarity, but running tests. Different languages can use:
- Python:
pytest - JavaScript/TypeScript:
npm test,vitest - Java:
mvn test,gradle test - Go:
go test - Rust:
cargo test - C/C++: compile and then run the test binary
For example, you can run pytest in a temporary directory as follows:
import pathlibimport subprocessimport tempfiledef run_pytest(code: str, tests: str, timeout: int = 60):with tempfile.TemporaryDirectory() as td:base = pathlib.Path(td)(base / "solution.py").write_text(code, encoding="utf-8")(base / "test_solution.py").write_text(tests, encoding="utf-8")result = subprocess.run(["python", "-m", "pytest", "-q", "test_solution.py"],cwd=td,capture_output=True,text=True,timeout=timeout,check=False,)passed = result.returncode == 0detail = result.stdout + result.stderrreturn passed, detail
Note: generated code may execute arbitrary logic, so it must not be run directly on a production server or a developer machine. Run it in a container and set:
- A read-only file system;
- No external network access;
- A least-privilege user;
- CPU, memory, and execution-time limits;
- No access to keys, databases, or internal services;
- Retain redacted logs for failed test cases.
4.3 Multi-dimensional scoring
Score across the following dimensions:
- Correctness: whether it passes tests;
- Security: whether it introduces issues such as command injection, path traversal, or hardcoded keys;
- Readability: whether naming, structure, and comments are clear;
- Style consistency: whether it complies with team lint rules;
- Latency and cost: how long it takes to generate a usable answer;
- Stability: whether repeated generations all pass tests.
4.4 Acceptance criteria should be conditional
When the evaluation set, quantization, and concurrency tuning all meet the team’s acceptance criteria, a private model can usually satisfy everyday assisted-programming needs, though real-world testing is still required for the specific business scenario. High-risk scenarios in particular — such as payments, permissions, data deletion, and low-level protocol implementation — must add human review and dedicated testing.
4.5 Evaluation report template
It is recommended to output a report after every model or inference-configuration change, with fields including:
- Model version;
- Quantization method;
- Context length;
- Evaluation set version;
- Task type;
- Pass rate;
- Average latency;
- Average token count;
- Failed samples;
- Number of security issues;
- Regression comparison against the previous version.
5. Cost and cost-effectiveness: how do you compare self-hosting, cloud inference, and SaaS?
Cost cannot be judged by GPU price alone; you must look at total cost of ownership and cost per call.
5.1 Cost quantification table
| Dimension | Self-hosted private | Cloud inference / model hosting | Commercial SaaS |
|---|---|---|---|
| Hardware investment | High, needs GPUs and a data center | Low, rent on demand | Very low |
| Initial delivery | Slow, needs deployment and tuning | Relatively fast | Fastest |
| Marginal cost | Drops as utilization rises | Rises with usage | Rises with seats or calls |
| Ops requirement | High | Medium | Low |
| Data control | Strong | Medium, depends on the cloud service | Weak to medium, depends on the vendor |
| Customization | Strong | Medium | Weak |
| Cost predictability | Good at high utilization | Medium | High but expensive per unit |
5.2 Self-hosting cost formula
You can use:
Monthly self-hosting cost = hardware depreciation + electricity + bandwidth + labor + model-iteration costUnit cost = monthly self-hosting cost / effective calls per month
Where:
- Hardware depreciation: GPUs and servers are typically depreciated over 3 to 4 years;
- Electricity: full-load and idle power draw differ greatly;
- Labor: deployment, monitoring, tuning, and upgrades all require investment;
- Model iteration: evaluation, fine-tuning, prompt optimization, and knowledge-base maintenance should also be counted.
5.3 Common cost pitfalls
- Idle GPUs: used only for demos, with no stable call volume formed;
- Overestimated concurrency: buying cards for peak load, leaving low utilization the rest of the time;
- Overly long context: long context significantly increases the KV Cache and VRAM;
- Driver mismatch: repeated troubleshooting consumes labor;
- No evaluation system: frequent model switching makes cost uncontrollable;
- Ignoring storage and logging: code retrieval, audits, and vector databases also incur costs.
5.4 Selection recommendations
- Low call volume, quick pilot: use cloud inference or SaaS first;
- Sensitive data, many internal frameworks: self-host privately;
- High call fluctuation: hybrid deployment, using cloud inference to absorb peaks;
- Stable and large-scale calls: the marginal-cost advantage of self-hosting becomes more pronounced;
- No ops capability on the team: be cautious about self-hosting, and prioritize hosted services.
6. FAQ
What should I do if a high temperature makes code unstable?
For code generation, set the temperature between 0 and 0.3. A higher temperature makes output more divergent and prone to introducing unnecessary changes. For tasks such as code completion, unit test generation, and bug fixing, prioritize executability and stability.
What should I do if high concurrency causes latency to rise?
You can add a queuing mechanism, rate limiting, dynamic batching, and replica scaling. At the same time, monitor first-token latency, total generation time, and queue length. Do not look only at average latency; also examine P95 and P99.
What should I do if quantization increases syntax errors?
Do not deploy it directly. First compare the FP16, INT8, and INT4 versions on the evaluation set. If INT4 causes a noticeable drop in pass rate, you can quantize only some layers, use mixed precision, or fall back to a higher precision.
What should I do if an overly long context causes VRAM to explode?
Set maximum context length, maximum generation length, and a concurrency cap. For long code-repository tasks, you can combine retrieval augmentation to put only relevant snippets into the context, rather than stuffing in all files at once.
What should I do if the model weights come from an untrusted source?
Download weights only from trusted channels, and record their hashes. Verify file integrity before going live, and avoid using unsigned model files or files with an unclear training source.
7. Best-practices checklist
- Validate at small scale first: use a single machine with multiple cards to run through the full loop of “model loading, API calls, code evaluation, and cost tracking”;
- Build an evaluation loop: accept all code generation tasks using executable methods such as pytest, npm test, and compile commands;
- Security first: inference APIs must be authenticated, and generated code must run in a sandbox;
- Cost transparency: continuously record GPU utilization, call volume, latency, token cost, and labor investment;
- Version control: version the model, drivers, frameworks, dependencies, and prompts;
- Continuous iteration: periodically re-evaluate based on the model ecosystem, business scenarios, and call data, rather than deploying once and leaving it unchanged for a long time.
Overall, deploying a large language model on your own GPU resources for intelligent coding assistance is a viable path. But it is better suited as a controllable, auditable, and long-term evolvable capability foundation than as an unconditional replacement for all commercial tools. Teams should first validate the results with a small-scale evaluation, then gradually expand the scope of usage.
