Moving open-weight LLMs from a demo environment into a production system takes more than swapping in a new inference endpoint. The business questions are whether the model can complete internal tasks reliably at acceptable latency and availability, whether call costs are predictable, whether model weights and logs can stay within a controlled boundary, and whether the team can quickly roll back if output quality regresses. Lower cost, a clear deployment path, and greater technical autonomy are the three most common reasons companies evaluate open-weight options. The sections below follow the engineering sequence and do not tie the discussion to any specific model family or commercial assistant brand.
This article covers the engineering work needed to run open-source large language models in production. It explains how to define production readiness, how to choose between closed APIs, self-hosted open-weight inference, and managed cloud inference, and how to structure a gateway, inference layer, context layer, and observability stack. It also covers evaluation with internal tasks, cost control, a staged migration from a closed coding assistant, and the security and compliance review needed before launch. Readers get a launch checklist they can use to confirm each item with evidence.
1. Define Production Readiness Before Evaluating the Model
Many pilots fail because their acceptance criteria stay at “it feels useful.” Production readiness requires five things at the same time.
First, the task boundary must be clear. Code completion, ticket summarization, retrieval-augmented question answering, and structured extraction each fail in different ways, so one subjective “looks good” standard cannot cover them all. Second, quality must be measurable. Use an internal question set to track pass rates, agreement with manual spot checks, and the share of tickets that contain errors caused by hallucination. Do not rely only on public leaderboards. Third, latency and throughput must be commitments the team can keep. Interactive coding assistants usually need both the time to first token and the full response time to fall within a range engineers can tolerate. Offline batch processing depends more on throughput and queue buildup. Fourth, cost must be accountable. Track spending per request, per token, per GPU hour, or per concurrency tier, and attribute it to teams and use cases. Fifth, failures must be reversible. Version the model, prompts, retrieval index, and gateway policy, and be able to switch back to the previous stable version or a backup route within minutes.
Only when these five items appear in the launch checklist do model selection and migration have a basis for decisions.
2. Model Selection: Capability, Licensing, and Running Cost Together
Selection should start from the task distribution, not from parameter count. Analyze real requests from a recent period and measure the share of short completions, long-context reconstruction, multi-file understanding, tool calls, and multi-turn clarification. Short completions usually depend on latency and concurrency. Long-context work depends on GPU memory and prefill cost. Tool calls depend on format stability and timeout policy.
On that basis, compare three sources of supply: closed remote APIs, self-hosted inference on open weights, and managed inference for general-purpose models on a cloud platform. Closed APIs are quick to integrate, but pricing, data residency, and product changes are not fully under your control. Self-hosting requires upfront investment in GPUs, images, observability, and on-call support. It suits high call volumes, sensitive data, or deep customization. Managed inference sits between the two. It suits teams that want to validate a use case before deciding whether to move workloads onto their own cluster.
Open-weight models also need a separate review of their license and weight source. Confirm that commercial use and redistribution are permitted, that the intellectual property terms for training data and outputs fit the business, and that the weight files have checksums that can be traced. Technical autonomy means that prompts, adaptation data, logs, vector indexes, and inference configuration stay under your own access controls. It does not mean that external services are never used. Hybrid architectures are often more realistic: high-sensitivity or high-concurrency workloads run on internal inference, while long-tail requests or peak overflow go to a backup route.
If a company already runs containers, object storage, and log monitoring on Baidu AI Cloud or another comparable cloud, bring the inference service under the same identity, network isolation, and audit controls. Do not create a separate account structure that cannot be traced. This section uses only general-purpose capabilities: a container platform schedules instances, object storage holds weights and evaluation sets, and logging and monitoring alerts handle tracing. No vendor-specific API names are required.
3. Reference Architecture: Gateway, Inference, Context, and Observability
A workable pipeline can be organized into four layers.
The access layer handles authentication, quotas, redaction of sensitive fields, and request routing. Tag requests by team, application, and use case. When a quota is exceeded, queue or reject the request instead of retrying without limit, which would saturate the GPUs. Routing policy should distinguish at least three cases: the default model, a high-accuracy fallback, and a rule-based degraded mode. Degradation can mean a smaller model, a cache hit, or returning only the retrieved passages without generation.
The inference layer handles batching, continuous batching, context length limits, and timeouts. Separate the queues. Interactive requests use short timeouts and small batches. Offline tasks use long timeouts and high throughput. Record the weight version, quantization method, and maximum context length in configuration, and change them through a release ticket. This prevents weights from being swapped silently in production.
The context layer determines what the model sees. Coding scenarios typically combine repository snippets, symbol definitions, and recent edits. Knowledge scenarios use retrieved passages with a constraint to cite sources. Retrieval augmentation and fine-tuning do not exclude each other. When knowledge changes often and sources must be shown, prioritize retrieval. When output format and domain tone stay stable over time, consider small-scale adaptation. Do not put all internal documents into the context to “make the model smarter.” That raises latency and cost and widens the surface for data leakage.
The observability layer records request IDs, model version, prompt template version, retrieval hits, token usage, latency percentiles, error codes, and human feedback. Content logs should be redacted by default and retained according to compliance requirements. Without these fields, you cannot explain why latency rose this week or why a certain type of defect suddenly increased.
Routing logic can be written as a policy, without binding it to a particular framework:
on_request(req):req = redact(req)if over_quota(req.team): return degrade_or_reject(req)route = policy(req.scene, req.sensitivity, req.latency_slo)if cache_hit(req, route.model): return cache_get(req)resp = infer(route, budget_ms=req.timeout)if resp.failed or low_confidence(resp):resp = fallback(req) # smaller model, backup route, or retrieval onlyemit_metrics(req, route, resp)return resp
The policy itself should live in a configuration center, released separately from the model weights. That way, you can roll back only the routing without rolling back the weights.
4. Evaluation: Use Internal Tasks Instead of Demos
Before launch, build a frozen evaluation set that covers high-frequency cases, high-risk cases, and long-tail cases. For a coding assistant, include whether unit tests pass, whether patches modify files outside their scope, whether they leak strings that look like secrets, and the share of dangerous operations the assistant correctly refuses. For knowledge question answering, include whether citations come from permitted documents, whether answers go beyond the passages and invent content, and whether numbers and proper names match the sources.
Comparison experiments must fix the prompt template, decoding parameters, context assembly, and evaluation scripts. Change only one variable at a time, such as the model, quantization, retrieval top-k, or temperature. Report results as ranges and percentiles, for example P50 and P95 latency, task pass rate, and the rate of disagreement in manual review, and state the sample size. Do not use unrepeatable pilot figures such as “about this much faster” or “this much cheaper” as a basis for capacity planning. Recalculate capacity from peak QPS, average context length, and the acceptable queue time.
For rollout, use shadow traffic: copy production requests to the candidate model, score the results, and do not return them to users. When the pass rate is no lower than the current baseline, P95 latency does not hurt the user experience, and the cost per valid result is acceptable, increase traffic by team percentage. If a safety or correctness regression appears, the gateway should switch that use case back to the baseline route in one step.
5. Cost and Performance: Cut Wasted Tokens Before Buying Hardware
Spending usually comes from four sources: prefill for long contexts, output length, retries after failures, and idle GPU time. The following methods reduce cost without depending on a specific hardware brand, and they should be applied first:
- Trim prompt templates and remove repeated system instructions. A stable prefix also makes prefix caching more effective.
- Deduplicate retrieved passages, truncate them by relevance, and cap the maximum context length.
- Lower randomness for deterministic subtasks to avoid unnecessary long outputs.
- Cache identical requests for a short time. For completion scenarios, you can cache intermediate results for recently edited files, but invalidate them when the code changes.
- Separate pools for interactive and batch workloads so offline tasks do not raise online P95 latency.
- When accuracy is acceptable, use common low-bit quantization, and confirm the size of any regression on the same evaluation set.
Cost reductions that come at the expense of traceability or safety filtering do not count as production optimization. The billing data should answer which use case, which prompt version, and which context length tier account for most of the consumption.
6. Migrating from a Closed Coding Assistant
The goal of migration is to make the default path replaceable, not to shut off the old interface in one day. Use four steps.
Step one: inventory every call site, including IDE plugins, patch suggestions in CI, code review bots, and internal document Q&A. Mark which calls must stay on the internal network and which can leave the domain. Step two: match capabilities, not brands. Use the same set of internal repository tasks to compare patch usability, average number of turns, and the rate at which humans take over. Step three: run both systems in parallel. Make the new model an optional engine first. Collect refusal reasons and negative-feedback samples, and use them to improve prompts and retrieval. Do not change the default yet. Step four: switch the default and keep a fallback window. After the switch, keep monitoring for at least one release cycle to confirm there are no hidden license or secret-scanning regressions.
Assign owners for each area. The platform team owns the gateway and inference. The business teams own the evaluation sets and prompts. The security team owns redaction, access rights, and licensing. A model without an evaluation-set owner should not become the default for everyone.
7. Security, Compliance, and the Supply Chain
In a production system, the model is one link in the software supply chain. Lock versions and verify checksums for weights, tokenizers, inference images, and evaluation scripts. Threat-model prompt injection, unauthorized tool calls, and personal information in training data or logs, just as you would for any traditional application. Outputs shown outside the company should exclude internal paths, secrets, and unpublished data by default. Decide cross-border transfers based on data classification. Keep raw code and customer data in designated regions, and when necessary, sync only redacted aggregate metrics.
Open source does not mean exempt from review. A launch review should cover at least the license, weight source, evaluation report, capacity and degradation plan, log retention, and emergency contacts. During a change window, do not modify weights, the retrieval index, and gateway policy at the same time. Otherwise, you cannot identify the cause of a regression.
8. Launch Checklist
Before launch, use the following checklist, and back every item with evidence instead of verbal confirmation:
- The task and non-goals are documented.
- The evaluation set is frozen and includes high-risk samples.
- The baseline comparison covers quality, latency, and unit cost.
- Routing, quotas, redaction, and fallback have been rehearsed.
- Weights and images can be verified by checksum.
- Logs can be linked to versions.
- Permissions are granted at the minimum scope.
- Capacity is estimated from peak load, not average load.
- On-call and rollback steps are documented and executable.
Adopting open-weight LLMs in industry is, at its core, a matter of applying the same engineering discipline used for databases and message queues to generation capabilities. The system should be observable, reversible, accountable for cost, and auditable. If a team meets these requirements, cost savings and autonomy can become lasting advantages. If it does not, even a strong demo becomes one more unstable dependency. Teams should start with one use case with a clear boundary, use shadow traffic to show that results are no worse than the current baseline, and then expand default coverage step by step.
