LLM Selection: Closed-Source, Open-Weight, and Truly Open Source Models

    This article explains how to choose large language models beyond leaderboard rankings. It covers the three main types of model providers, the limits of one-dimensional capability comparisons, the distinction between closed-source, open-weight, and truly open source models, what self-hosting actually means for data, versioning, cost, and compliance, and a practical selection framework with a weighted scoring example and guidance for interpreting evaluation results.

    1. Start with the Landscape, Then Discuss Selection

    The most common mistake in large language model selection is drawing conclusions from a single leaderboard ranking. Model capability, price, delivery model, and licensing terms are interconnected. Looking at only one of these factors makes reliable decisions difficult. To select well, you first need to understand what the main players in the market are doing and what their products mean for users.

    2. Three Types of Participants

    From a practical usage perspective, today’s model suppliers can be roughly divided into three categories.

    1. Comprehensive foundation model developers: These organizations typically invest substantial compute in training general-purpose models covering text, code, image, speech, and other modalities, and they provide services through APIs and subscription products. Their strengths are high capability ceilings and fast iteration, but the models themselves are usually not downloadable. Users can only pay by usage volume or subscription.
    2. Cloud service and platform vendors: These vendors host third-party or self-developed models and provide supporting capabilities such as inference, fine-tuning, vector retrieval, and security governance. For enterprises, the value of such platforms lies in integrating models into existing cloud infrastructure, which simplifies unified authentication, billing, and auditing.
    3. Open source communities and research institutions: These groups release downloadable model weights, training scripts, or technical reports, and some use permissive licenses. Such models allow teams to deploy in their own environments, but teams must take on operations, optimization, and security hardening themselves.

    These three types of participants are not mutually exclusive. The same organization may release both closed-source products and open-weight versions, and the same platform may host models from multiple sources. Therefore, whether a model is usable should be judged by its delivery form, not just by its reputation.

    3. Why Capability Gaps Are Hard to Summarize in One Sentence

    Comparisons between models from different regions or vendors are often reduced to “who is ahead.” This framing ignores three issues.

    First, capability is multidimensional. Reasoning, long-context understanding, code generation, multilingual ability, tool calling, and expression in Chinese-language contexts may each produce a different ranking. A model that leads on math competition problems may not have an advantage in customer service ticket classification.

    Second, evaluation results depend on methodology. Different leaderboards vary in question distribution, prompt templates, sampling parameters, and whether external tools are allowed, all of which can change scores. Vendor-published results are often self-reported, while third-party re-tests frequently use different settings. Directly subtracting one score from another often leads to unreliable conclusions.

    Third, deployment scenarios determine the actual gap. When a business requires low latency, offline operation, or local data that cannot leave its domain, a model with a slightly lower benchmark score may still be the more appropriate choice.

    Therefore, public rankings can only serve as an initial screening reference. What is truly valuable is evaluating models with your own business data.

    4. Three Meanings of “Open Source”

    In the context of large language models, “open source” is often used loosely. At least three layers of meaning can be distinguished.

    4.1 Closed-Source Service

    Neither model weights nor training details are publicly disclosed. Users call the model through an API, and data is processed on the provider’s servers. Whether the model can be modified or deployed locally depends entirely on the terms of service.

    4.2 Open Weight

    Model parameters can be downloaded, and users can run inference on their own hardware or even fine-tune the model. However, the composition of training data, data cleaning process, training code, and hyperparameter configuration may not be disclosed. The license may restrict commercial use, user scale, or redistribution, require attribution, or prohibit use in specific domains.

    4.3 Truly Open Source

    In addition to weights, the training code, data source descriptions, and processing methods are also disclosed, under an open source license that meets commonly accepted industry definitions. Ideally, an independent team can reproduce the training process based on the public materials and inspect the model for bias and risk. Currently, few models fully meet this standard.

    These three layers directly affect users’ rights. Open weights let you put the model into your own data center, but weights alone cannot tell you whether the training data is compliant. Truly open source lets you trace and modify the model, but maintenance costs are also higher.

    5. What It Means to Hold the Model Yourself

    “Holding the model yourself” can be broken down into four specific capabilities.

    • Data control: Whether sensitive data must remain on the internal network, and whether inference requests may be used by the provider for training or retention.
    • Version control: Whether model upgrades are decided by you. When a server-side model is updated, its output style and behavior may change, and existing prompts and downstream logic need to be revalidated.
    • Cost predictability: Pay-per-call billing is flexible, but the cost curve becomes steep as traffic grows. Self-hosting may have a lower unit cost at scale, but it requires upfront investment in hardware and personnel.
    • Compliance and audit: In areas involving personal information, finance, or healthcare, organizations need to explain model provenance, the basis for training data, and license terms. This information is often difficult to obtain in closed-source services.

    These four capabilities do not all need to be perfect. An internal knowledge question-answering system may have high requirements for data control but lower requirements for version stability. A copywriting generation feature serving a large user base may place more emphasis on cost.

    6. An Executable Selection Framework

    Selection can proceed through the following steps.

    1. Define tasks: List specific business tasks, and define the input, output, and acceptance criteria for each task.
    2. Set constraints: Confirm data sensitivity levels, latency limits, budget range, deployment environment, and compliance requirements. Once constraints are clear, many options will be eliminated immediately.
    3. Build an evaluation set: Sample from real business data, cover common cases and edge cases, and have business staff annotate reference answers. The evaluation set should be updated regularly to avoid overlap with model training data, which can inflate scores.
    4. Run a small-scale pilot: Run the same evaluation set against candidate models and record accuracy, latency, cost per call, and failure types.
    5. Make a weighted decision: Set weights based on constraints and business priorities, then calculate a composite score.

    Below is illustrative code for aggregating scores by weight:

    1. WEIGHTS = {
    2. "accuracy": 0.35,
    3. "latency": 0.20,
    4. "cost": 0.20,
    5. "data_control": 0.15,
    6. "license": 0.10,
    7. }
    8. def weighted_score(scores: dict[str, float]) -> float:
    9. """Each dimension in scores ranges from 0 to 1; higher is better."""
    10. return sum(scores[k] * w for k, w in WEIGHTS.items())
    11. candidates = {
    12. "option_a": {"accuracy": 0.82, "latency": 0.70, "cost": 0.55,
    13. "data_control": 0.40, "license": 0.90},
    14. "option_b": {"accuracy": 0.78, "latency": 0.85, "cost": 0.80,
    15. "data_control": 0.95, "license": 0.85},
    16. }
    17. ranked = sorted(candidates.items(), key=lambda kv: weighted_score(kv[1]), reverse=True)
    18. for name, s in ranked:
    19. print(name, round(weighted_score(s), 3))

    Weights and scores should be determined by the team according to business context. The numbers in the code are for demonstration only. If a dimension contains a veto condition—for example, data must remain within a particular jurisdiction or on-premises—it should be filtered out before scoring. A high score should not offset a hard constraint.

    7. How to Interpret Evaluation Results

    When using public leaderboards, consider the following rules.

    • Check the evaluation setup: Confirm the prompt format, number of few-shot examples, decoding parameters, and whether external tools are allowed.
    • Distinguish self-reported results from re-tests: Prefer independent third-party reproduction results, and pay attention to the corresponding model version number.
    • Watch the variance: Small-sample evaluations can fluctuate.