Open-Source LLM Enterprise Spending in China: A TCO Guide and Selection Checklist

    This article argues that "enterprise spending share" figures for open-source LLMs deserve a cautious read, and shifts the focus from leaderboards to total cost of ownership (TCO) and a practical selection framework. It explains what enterprises actually pay for, how to estimate effective token cost, how VRAM and KV cache drive self-hosted costs, and how prefix caching reduces spend. It also covers deployment options (elastic, dedicated, private, and edge), tiered-routing architecture, production engineering practices such as load testing, priority scheduling, observability, and regression testing, ten key questions for selection, and an FAQ—helping technical decision-makers judge whether an open-source model can carry business workloads cost-effectively over the long term.

    1. Background: Why a “90%” Figure Deserves a Cautious Read

    Recently, some media coverage has cited industry surveys suggesting that open-source model vendors account for a high share of enterprise spending on open-source models. Because the discussion lacks verifiable details—report name, publishing organization, release date, measurement period, denominator definition, and whether private deployment or toolchains are included—this article does not treat any specific percentage as a basis for decisions. Instead, it uses the figure as background for a broader shift: when enterprises procure open-source large models, their focus is moving away from “parameters, leaderboards, and community buzz” toward “total cost of ownership, stability, and delivery certainty.”

    For technical decision-makers, the questions that actually matter are not a single ranking but the following:

    • What are enterprises really paying for?
    • What hidden costs remain between “free weights” and “production-ready”?
    • How do you judge whether a model is suitable to carry a business over the long term?
    • How can architecture and engineering reduce unit business cost?

    This article addresses these questions with a reusable evaluation framework.

    2. What Spending Share Actually Signals

    If the cited reports can later be verified, readers should pay close attention to their measurement definitions. Enterprise spending on open-source models is usually not equivalent to:

    • Download counts: cannot distinguish personal trials, one-off evaluations, and long-term production use.
    • GitHub stars: reflect attention, not procurement or investment.
    • Leaderboard scores: heavily influenced by evaluation sets, prompts, and methodology.
    • API call volume: may include test traffic and does not necessarily reflect revenue.

    Enterprise spending is closer to an indirect signal of “production readiness,” because enterprises typically pay on an ongoing basis only after completing security review, performance load testing, cost accounting, and service evaluation. Spending usually covers five categories:

    1. Model inference services: managed inference billed by token or compute-hours, dedicated instances, and so on.
    2. Private deployment: bringing the model into an enterprise’s own data center or private cloud, involving licensing, deployment, and migration.
    3. Fine-tuning and scenario adaptation: domain continued pre-training, instruction fine-tuning, preference alignment, RAG, tool-calling integration, and more.
    4. Operations and support: SLAs, incident response, version upgrades, technical support.
    5. Toolchain support: model registry, evaluation, gateway, observability, security auditing, and other foundational capabilities.

    So what enterprises compare is not just the “unit price of a model” but the combined delivery capability of “model + service + operations + compliance.”

    3. From “Free” to “Paid”: What Enterprises Really Buy

    Open-source model weights can be obtained for free, but once they enter production, they often incur substantial hidden costs:

    • Purchasing GPUs, servers, and networking for the inference cluster;
    • Deploying, tuning, and upgrading the inference engine;
    • 24/7 monitoring, troubleshooting, and capacity planning;
    • Security and compliance review and data isolation;
    • Effect tuning, business integration, and regression evaluation;
    • Failure fallback, canary releases, and version rollback.

    Open source lowers the “acquisition cost” but does not automatically lower the “operating cost.” Enterprises should ultimately judge by total cost of ownership (TCO):

    1. TCO = direct compute cost + infrastructure cost + labor cost + migration/integration cost + quality risk cost

    When a capability can be obtained on demand through a mature service, the marginal benefit of building the entire capability in-house can decline quickly. When evaluating candidate models, six dimensions are recommended:

    Dimension Key question Validation method
    License compliance Does it allow commercial use, fine-tuning, and distribution? Are derivative naming rules restricted? Check license terms clause by clause; conduct legal review if needed
    Chinese and business fit How does it perform on Chinese semantics, long documents, and technical terms? Build a small evaluation set from real business data
    Inference efficiency At the same hardware, what are throughput and time-to-first-token? How much does enabling prefix caching improve them? Load-test with inputs close to real lengths
    Deployment mode Does it support private deployment, private cloud, and elastic hosting? Run end-to-end validation in the target environment
    Toolchain and ecosystem Is it compatible with mainstream inference engines, fine-tuning frameworks, and application frameworks? Run a full pipeline from data to service
    Long-term maintenance Does it keep releasing versions, security patches, and technical support? Check release cadence and support history

    4. Cost Perspective: What Really Counts Behind the Token Price

    4.1 Unit Token Cost Is More Than the Sticker Price

    Many teams look only at “price per million tokens,” but actual spending is also affected by input/output structure, retries, cache hit rate, and other factors. A more realistic cost formula can be written as:

    1. effective cost ≈ {
    2. non-cacheable input tokens × input price
    3. + cacheable input tokens × input price × [(1 - h) + h × (1 - d)]
    4. + output tokens × output price
    5. } × (1 + r) + fixed resource cost

    where:

    • h: hit rate of cacheable input tokens;
    • d: discount ratio applied after a cache hit;
    • r: retry factor from timeouts, rate limiting, substandard output, and so on;
    • the cache discount typically applies only to cacheable input tokens, not automatically to output tokens or non-cacheable content.

    The following example only illustrates the method and does not represent real pricing:

    1. Assumptions:
    2. Input total 10M tokens, of which 60% cacheable;
    3. Cache hit rate h = 0.8, cache discount d = 0.5;
    4. Output 2M tokens;
    5. Input price 0.4 CNY per million tokens, output price 1.2 CNY per million tokens;
    6. Retry factor r = 0.10.
    7. Non-cacheable input = 4M = 4 units of a million
    8. Cacheable input = 6M = 6 units of a million
    9. Cacheable input billing factor = (1 - 0.8) + 0.8 × (1 - 0.5) = 0.6
    10. Input cost = 4 × 0.4 + 6 × 0.4 × 0.6 = 3.04 CNY
    11. Output cost = 2 × 1.2 = 2.4 CNY
    12. Subtotal before retry = 5.44 CNY
    13. After retry ≈ 5.98 CNY

    For actual decisions, ask the provider to define the billing boundaries: whether input and output are priced separately, how cache hits are determined, the scope of the cache discount, whether failed retries are billed, and whether dedicated instances include operational support.

    4.2 VRAM Usage and KV Cache: No One-Size-Fits-All Answer

    For self-hosted or dedicated deployment, cost depends heavily on hardware utilization. VRAM is mainly consumed by:

    1. Model weights: roughly “parameter count × bytes per parameter”;
    2. KV cache: grows with context length and number of concurrent requests;
    3. Activations and temporary buffers;
    4. The inference framework’s own overhead.

    KV cache can be estimated approximately as:

    1. KV cache VRAM ≈ 2 × layers × KV heads × head dim × sequence length × concurrent requests × bytes per value

    For a 7B-parameter model at FP16 precision, the weights need about 14GB. The KV cache must be judged by the network architecture. Assuming 32 layers, head dim 128, and FP16:

    • MHA, 32 KV heads: about 0.5MB per token. An 8K context single request is about 4GB, and 32 concurrent requests are about 128GB.
    • GQA, 8 KV heads: about 0.125MB per token. An 8K context single request is about 1GB, and 32 concurrent requests are about 32GB.

    Using INT8 KV cache can reduce VRAM further, but the accuracy impact must be validated on the business evaluation set. Real deployment must also reserve space for activations, framework overhead, and fragmentation. So a vague statement like “a few GB” is unsuitable for direct capacity planning; you must specify layers, KV head count, precision, MHA/GQA, and other conditions.

    4.3 Three Technical Variables That Determine Cost Structure

    1. Sparse architectures: MoE models have a large total parameter count but activate only some experts per token. Compute is determined by activated parameters, while VRAM usage is determined by total parameters, so the two must be accounted for separately during deployment.
    2. Quantization and compression: INT8, INT4, low-precision KV cache, and similar techniques can reduce VRAM and bandwidth usage and increase per-GPU concurrency, but they introduce accuracy loss that must be validated with business evaluation.
    3. Prefill/Decode asymmetry: prefill is compute-intensive and determines time-to-first-token; decode is memory-access-intensive and determines the inter-token output interval. Mixed workloads easily interfere with each other and must be confirmed by load testing.

    4.4 The Overlooked Cost Lever: Prefix Caching

    In scenarios like customer-service Q&A, document summarization, and code assistance, system prompts, tool definitions, and few-shot examples are often repeated at scale. Recomputing them on every request is effectively paying twice. Keeping the prompt structure stable, placing variable content as late as possible, and prioritizing services that support prefix caching is often more effective than simply switching to a cheaper model.

    1. import hashlib
    2. import json
    3. def build_prompt(system_prompt: str, tools: list, docs: list, question: str) -> str:
    4. # Reusable prefix: system instruction + tool definitions, kept byte-for-byte identical across all requests
    5. prefix = system_prompt + "\n" + json.dumps(tools, ensure_ascii=False, sort_keys=True)
    6. # Place the variable part as late as possible to improve the prefix-cache hit rate
    7. return prefix + "\n[Reference]\n" + "\n".join(docs) + "\n[Question]\n" + question
    8. def cache_stats(hits: int, misses: int) -> dict:
    9. total = hits + misses
    10. hit_rate = hits / total if total else 0.0
    11. return {"hit_rate": round(hit_rate, 3), "requests": total}
    12. print(cache_stats(hits=8600, misses=1400)) # {'hit_rate': 0.86, 'requests': 10000}

    Two engineering notes: servers usually identify reusable prefixes by content hash, so an extra space, reordered keys, or a timestamp inserted at the beginning can all break the cache. Hit rate must be continuously measured and used as an input to capacity planning and cost accounting.

    5. Deployment Models: Choosing the Supply Mode by Scenario

    Deployment mode Typical scenarios Advantages Main trade-offs
    Elastic managed inference Highly variable traffic, short time-to-launch No ops, pay by usage, elastic scaling Data may leave the enterprise boundary; unit cost must be recomputed at scale
    Dedicated deployment Stable baseline traffic, performance-isolation requirements Dedicated resources, more controllable long-term unit cost You bear capacity planning and operations
    Private deployment Strong-compliance scenarios such as finance, healthcare, and government Data stays within the domain, easy to meet audit requirements High upfront investment, heavy upgrade and operations burden
    On-device / edge lightweight models Low latency, privacy-sensitive, weak-network environments No network dependency, near-zero marginal inference cost Limited by device compute, limited capability ceiling

    Mainstream cloud platforms typically offer public-cloud APIs, dedicated instances, private deployment, and other modes. When selecting, enterprises should not look only at model capability; they should also put data classification, compliance requirements, traffic curves, and operational capability into the same decision table.

    There is usually a “crossover point”: when traffic is small and highly variable, pay-as-you-go managed models have a lower total cost; once call volume grows steadily to a certain level, dedicated or private resources may be more economical. The specific inflection point should be based on the business curve, quotes, and measured utilization, not on intuition.

    6. Architecture: Tiered Routing to Spend Where It Matters

    Enterprise request difficulty typically follows a long-tail distribution: many requests are simple Q&A, format conversion, or information extraction, and only a minority truly need strong reasoning. Routing everything to the most expensive model is wasteful. A common approach is “small model as the baseline, large model as the fallback”:

    1. def route(prompt: str, privacy_level: str, meta: dict) -> str:
    2. # 1. Compliance first: sensitive data may only go through the private channel
    3. if privacy_level in ("sensitive", "confidential"):
    4. return "private"
    5. # 2. Rule fallback: hand structured extraction to the small model to lower unit cost
    6. if meta.get("task") in ("extract", "classify", "rewrite"):
    7. return "small"
    8. # 3. Route by complexity: multi-step reasoning and long-document analysis go to the stronger model
    9. if meta.get("complexity", 0) >= 3 or meta.get("doc_length", 0) > 20000:
    10. return "large"
    11. return "small"

    Routing decisions should be based on observable signals—task type, input length, business confidence, historical failure rate—rather than subjective experience. For safety-critical scenarios, a cascaded verification approach can be used: a lightweight model produces an answer first, then another model or rules cross-verify it; if the results disagree, escalate to a stronger model or hand off to a human.

    In the early stage after launch, run the “routing approach” and the “single-model approach” in parallel, compare quality, latency, and cost, and switch only after the data stabilizes.

    7. Engineering: Turning Open-Source Models into Reliable Production Services

    7.1 Capacity Planning and Load Testing

    Before any model goes live, three types of load tests should be completed:

    1. Single-concurrency latency test: record time-to-first-token and the inter-token output interval at different input lengths.
    2. Graduated concurrency test: gradually increase concurrency to find the throughput saturation point and the latency inflection point.
    3. Mixed-load test: run long-text and short requests simultaneously and observe mutual interference.

    Capacity estimation can refer to Little’s Law:

    1. required concurrency ≈ target QPS × average per-request latency (seconds)

    Actual configuration should also account for the peak-to-trough ratio, redundancy, and failover capability. It is recommended to establish at least the following load-test template:

    Input length Concurrency TTFT TPOT Throughput Error rate Cost metric
    1K 8 TBD TBD TBD TBD CNY per thousand requests
    4K 16 TBD TBD TBD TBD CNY per thousand requests
    8K 32 TBD TBD TBD TBD CNY per thousand requests

    7.2 Stability Design and Priority Scheduling

    The production inference path should include at least: timeout control, bounded retries, exponential backoff with jitter, circuit breaking and fast failure, multi-endpoint switching, and fallback degradation.

    When resources are constrained, requests can be scheduled by priority. The code below uses (priority, seq) to guarantee first-in-first-out within the same priority:

    1. import heapq
    2. import itertools
    3. from dataclasses import dataclass, field
    4. @dataclass(order=True)
    5. class Request:
    6. priority: int = field(compare=True) # Lower value = higher priority
    7. seq: int = field(compare=True) # Arrival sequence, guarantees FIFO within the same priority
    8. max_tokens: int = field(compare=False, default=256)
    9. class InferenceScheduler:
    10. def __init__(self):
    11. self._heap = []
    12. self._seq = itertools.count()
    13. def submit(self, priority: int, max_tokens: int = 256) -> None:
    14. heapq.heappush(self._heap, Request(priority, next(self._seq), max_tokens))
    15. def next_request(self):
    16. return heapq.heappop(self._heap) if self._heap else None
    17. if __name__ == "__main__":
    18. s = InferenceScheduler()
    19. s.submit(priority=1, max_tokens=1024) # High priority, arrives first
    20. s.submit(priority=2, max_tokens=256) # Normal priority
    21. s.submit(priority=1, max_tokens=512) # Same high priority, arrives later
    22. while (req := s.next_request()) is not None:
    23. print(req.priority, req.seq, req.max_tokens)
    24. # Output: 1 0 1024 -> 1 1 512 -> 2 2 256

    Two points matter: first, the arrival sequence number provides first-come-first-served within the same priority, avoiding unfairness; second, high-priority requests can cap output length to prevent long outputs from blocking subsequent requests.

    7.3 Observability and Cost Governance

    Three groups of metrics are recommended:

    • Business metrics: task success rate, human-takeover rate, user satisfaction.
    • Technical metrics: time-to-first-token, inter-token output interval, error rate, rate-limit count.
    • Cost metrics: token consumption trends, cache hit rate, cost allocation across business lines.

    Logs should desensitize user data and record model version, prompt version, and routing-strategy version, so that issues can be traced and costs can be accounted for.

    7.4 Version Upgrades and Regression Evaluation

    Open-source models iterate quickly, and enterprises must guard against “silent regressions”: after a model version update, output style changes and goes unnoticed until users complain. The recommended process:

    1. Pin the model version used in production;
    2. Run regression tests with an evaluation set that covers real business;
    3. Compare old and new versions via shadow traffic;
    4. Release gradually by proportion and set automatic rollback thresholds.

    During a canary rollback, keep the previous version’s weights, image, configuration files, inference parameters, and dependency versions to ensure fast recovery if anomalies occur.

    Minimal viable evaluation framework example:

    1. import re
    2. def exact_match(pred: str, gold: str) -> bool:
    3. normalize = lambda s: re.sub(r"\s+", "", s).lower()
    4. return normalize(pred) == normalize(gold)
    5. class EvalCase:
    6. def __init__(self, case_id: str, prompt: str, expected: str, tags: tuple = ()):
    7. self.case_id, self.prompt = case_id, prompt
    8. self.expected, self.tags = expected, tags
    9. def evaluate(model_call, cases: list) -> dict:
    10. records = []
    11. for c in cases:
    12. records.append({
    13. "id": c.case_id,
    14. "ok": exact_match(model_call(c.prompt), c.expected),
    15. "tags": c.tags
    16. })
    17. total = len(records)
    18. if total == 0:
    19. return {"overall_pass_rate": 0.0, "pass_rate_by_tag": {}, "failed_cases": []}
    20. by_tag = {}
    21. for r in records:
    22. for tag in r["tags"]:
    23. bucket = by_tag.setdefault(tag, [0, 0])
    24. bucket[1] += 1
    25. bucket[0] += int(r["ok"])
    26. failed = [r["id"] for r in records if not r["ok"]]
    27. return {
    28. "overall_pass_rate": round(sum(r["ok"] for r in records) / total, 3),
    29. "pass_rate_by_tag": {t: round(v[0] / v[1], 3) for t, v in by_tag.items()},
    30. "failed_cases": failed,
    31. }

    model_call can be a local inference function or a function that calls an inference service. Per-tag statistics quickly locate capability gaps, and failed cases accumulate over time to become input for the next iteration.

    8. Ten Key Questions for Enterprise Selection and Deployment

    1. Does the license allow commercial use, fine-tuning, and distribution? Are there naming restrictions on derivatives?
    2. On your own business evaluation set, how large is the performance gap between models?
    3. Do throughput and time-to-first-token on the target hardware meet the business SLA?
    4. After enabling prefix caching, how much can cost drop? How is cache hit rate measured?
    5. How does data classification map to deployment mode? Must sensitive data stay in a private environment?
    6. Does the inference service have timeout, retry, circuit-breaking, and fallback mechanisms?
    7. Have you established a tiered “rules + small model + large model” routing?
    8. Is cost observable and allocable by business line?
    9. Do model upgrades have version pinning, regression evaluation, and canary rollback processes?
    10. If a single model or single vendor fails, is there a fallback plan and switching strategy?

    9. FAQ

    How is open-source model TCO calculated?
    It should include at least compute or API fees, operational labor, migration and integration, evaluation and tuning, failure fallback, and compliance costs. Use “cost per business request” or “cost per thousand successful requests” instead of token unit price alone.

    How do you validate prefix caching?
    First build a batch of requests with a fixed system prompt and tool definitions, then gradually change the suffix content, and compare the provider’s returned cache-hit metrics, billing changes, and latency changes. The key is not to change the byte content of the prefix.

    How do you choose a private deployment?
    First check whether data is allowed to leave the domain, then look at model weight size, hardware conditions, inference framework compatibility, upgrade mechanism, and the operations team’s capability. If there is no dedicated inference operations team, prioritize managed or dedicated instances.

    Should a small team self-host or go managed?
    If traffic is small, highly variable, and the team lacks GPU operations experience, managed inference is usually more stable; if the business already has a stable baseline, sensitive data, or a calculable long-term cost, gradually validate dedicated or private deployment.

    10. Conclusion

    The discussion of “enterprise spending share” essentially reminds enterprises that open-source model competition is entering a phase of delivery and engineering. For enterprises, the rational approach is not to chase a particular leaderboard but to build their own evaluation baselines, cost models, and engineering standards.

    Open-source models have lowered the barrier to acquiring AI capabilities, but what truly determines real-world results is the engineering system built around the model: clear cost definitions, reliable inference services, observable quality metrics, a rollback-capable version process, and architecture designed to scale with the business.