Practicing Private Deployment and Intranet Evaluation for Open-Source LLMs

1. Background: Open-Source Weights Re-enter Enterprise Technology Selections

Over the past two years, the iteration speed of open-source large language models has accelerated significantly. Weights with parameter scales entering the hundreds of billions and context windows expanding to the million-token level have emerged. The performance gap between these open-source weights and top-tier closed-source models on public benchmarks for coding, agent operations, and log understanding has been compressed to single-digit percentages. For enterprise engineering teams, this provides a new solution to a long-standing problem: intelligent transformation in sensitive data scenarios no longer relies solely on external APIs, making private deployment a viable option within budget.

The value of open-source weights goes beyond simply “saving API call costs” and is reflected in at least three aspects:

• Data Sovereignty: Weights are downloaded to the intranet, and inference requests do not pass through third-party networks. Sensitive assets like test data, production logs, and requirement documents remain within your own security boundary.
• Predictable Cost Structure: Shifts from “variable token-based costs” to “relatively fixed infrastructure costs.” After scaled adoption, the marginal cost approaches the cost of electricity and O&M labor.
• Auditable and Customizable Behavior: You can perform continued training or fine-tuning based on business corpora to internalize coding standards, domain glossaries, and compliance requirements into the model, rather than injecting them into prompts every time.

2. Compute and Deployment Evaluation: Calculating the “Within Budget” Constraint

The first hurdle of private deployment is compute. Before going live, it is recommended to perform quantitative estimations using the following steps, rather than estimating the budget blindly.

1. VRAM Requirement Estimation (General Empirical Method)

• Model weights VRAM ≈ Parameter count × Bytes per parameter. Taking a 70-billion parameter model as an example: under BF16 precision, the weights require about 140GB of VRAM; under INT8 quantization, about 70GB; and under INT4 quantization, about 35GB—this is merely the lower limit for the weights.
• After factoring in the KV Cache, activation values, and inference framework overhead, the actual requirement typically increases by 20% to 50%. In long-context scenarios, the KV Cache scales almost linearly with context length and concurrency, often surpassing the weights themselves. Therefore, targeted stress testing based on “target concurrency × average context length” is necessary.
• Note: The above is a general estimation approach. Exact figures must be based on the chosen model’s release notes and actual testing; do not apply them directly without verification.

2. Deployment Pattern Selection

• Persistent Inference Cluster: Suitable for internal platform scenarios with stable concurrency, aiming for deterministic throughput and latency.
• Containerized Elastic Scaling: Leverages container platforms to scale based on business hours (e.g., batch processing at night, online during the day) to improve GPU utilization.
• Quantization Tiering: Core pipelines use high-precision tiers to guarantee quality, while throughput-intensive tasks like offline test case generation and log summarization use low-precision tiers to maximize throughput, trading tier differences for cost differences.

3. Separate Evaluation for Online and Offline

Online interactive scenarios focus primarily on first-token latency and queueing probability. Offline batch scenarios focus on total throughput and cost per task. Mixing these two sets of metrics and capacity plans is a common source of budgeting errors.

3. Typical Intranet Pipeline Integration Scenarios

Getting the weights into the intranet is just the beginning; the real value lies in specific scenarios within the pipeline.

Scenario 1: Test Case Generation

Feed requirement documents and API definitions into the intranet model to generate drafts for boundary conditions and exception branches, which are then reviewed and refined by humans. The key to the prompt template is forcing the model to “acknowledge uncertainty”:

Role: Senior Test Engineer
Input: API documentation + parameter constraints
Output: Equivalence class / boundary value / exception branch test cases in JSON format,
      Fields include case name, preconditions, steps, expected results
Constraints: Mark uncertain fields as ASSUMPTION; do not fabricate API behavior

Scenario 2: Log Analysis

Perform structured splitting and desensitization on massive logs first, then have the model perform anomaly clustering summaries and generate root cause hypotheses. The output is connected to the alerting system and handled by humans after review. The model’s role here is to “compress information and propose hypotheses,” not to “draw conclusions.”

General Architecture Recommendations

• Isolate the network between the model service zone and the business zone, placing an inference gateway in the middle to handle unified authentication, rate limiting, and auditing.
• Log all requests and responses (which can be integrated with logging services). Sensitive fields must be desensitized at the gateway layer before entering the model.
• Use object storage to host weights, corpora, and evaluation sets. Use a container platform to host inference services. Monitoring and alerting should cover metrics like latency, VRAM usage, and queue depth. If choosing a cloud deployment, general capabilities like container services, object storage, log services, and monitoring alerting from cloud providers (such as Baidu AI Cloud) can cover these categories, saving teams from reinventing the wheel.

At the API call level, use the internally agreed-upon REST interface:

import json, urllib.request
GATEWAY = "http://llm-gateway.internal/v1/generate"
def generate(prompt: str, max_tokens: int = 1024) -> str:
    body = json.dumps({"model": "internal-llm", "prompt": prompt,
                       "max_tokens": max_tokens,
                       "temperature": 0.2}).encode()
    req = urllib.request.Request(GATEWAY, data=body,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.loads(resp.read())["text"]

4. Official Metrics ≠ Business Performance: Custom Evaluation Set Methodology

This is the most common pitfall in private deployment. Officially published hallucination rates and benchmark scores represent an “official perspective”: the test set distribution, task definitions, and scoring methods differ from your business. An official single-digit hallucination rate can easily double or more when applied to your specific log formats and domain terminology. Making go-live decisions based directly on official numbers is equivalent to grading yourself using someone else’s test paper.

The correct approach is to build an evaluation set using your own business samples:

1. Sampling: Perform stratified sampling from real production traffic, historical tickets, and requirement documents, covering both high-frequency and long-tail tasks.
2. Golden Standard Annotation: Provide reference answers and judgment rules. For fact-based tasks, annotate verifiable sources to facilitate subsequent automated comparison.
3. Define Metrics: Factuality (hallucination rate), usability (whether manual major edits are needed), format compliance rate, and task success rate. Metrics must be translatable into business language.
4. Evaluation Method Combination: Rule validation (structure validation, fact comparison with database tables) + Model-as-a-judge (using a stronger model for scoring, noting the judge model’s own biases and stances) + Manual sampling calibration. These three mutually correct each other.
5. Continuous Regression: Integrate the evaluation set into CI. Run it every time the model version is changed, the quantization tier is adjusted, or the prompt is modified, forming a regression gate.

The evaluation set can be organized in JSONL, with a script skeleton as follows:

def fact_check(pred: str, golden: str, evidence: list) -> bool:
    # First perform rule/fact comparison; route uncertain samples to the judge model and manual review
    ...
def run_eval(path="eval_set.jsonl"):
    total = hallucinated = 0
    for case in map(json.loads, open(path, encoding="utf-8")):
        pred = generate(case["prompt"])
        total += 1
        if not fact_check(pred, case["golden"], case["evidence"]):
            hallucinated += 1
    print(f"Business hallucination rate: {hallucinated / total:.2%}")

Only when the business hallucination rate is compressed to an acceptable threshold for the team should you proceed to connect the pipeline and scale up traffic. This order cannot be reversed.

5. Risk Control and Engineering Checklist

• Supply Chain Security: Verify weight hashes and sources, lock model versions, and prohibit casually introducing unaudited third-party variants.
• Canary Strategy: Run new models on shadow traffic first, comparing them with the live version in a dual-run setup. Switch traffic only after metrics meet the standard.
• Rollback Capability: Retain the previous version’s weights and configuration to support one-click rollback.
• Cost Dashboard: Include token throughput, GPU utilization, and cost per task in monitoring and alerting to avoid “bleeding money upon launch.”
• Hallucination Fallback: Run fact checks and source verification before persisting generated content. High-risk actions (such as automatically modifying configurations) must require manual confirmation.

6. Conclusion

The fact that open-source weights are approaching the capabilities of top-tier closed-source models changes the decision cost of “whether to privatize.” However, what determines the success of the deployment is whether the evaluation set reflects real business needs, whether the pipeline has auditing and rollback mechanisms, and whether metrics have shifted from benchmark scores to business language. Building the evaluation set first, then connecting the pipeline, and finally scaling up traffic is a robust and reproducible engineering path.