How to Choose and Evaluate Open-Source LLMs: A Practical Deployment Guide
A practical guide to selecting and evaluating open-source large language models for deployment. It explains why there is no single "best model," breaks selection into four hard constraints—effectiveness, cost, compliance, and operations—and walks through six evaluation dimensions, from parameter count and MoE activation strategy to context length, modality, licensing, and inference-stack compatibility. It then presents a three-tier evaluation approach (public benchmarks, internal evaluation sets, and concurrency load testing), two memory and cost estimation formulas, a prioritized inference-optimization checklist, seven common pitfalls, and a weighted scorecard with a step-by-step rollout cadence.
1. First, Accept That “the Best Model” Doesn’t Exist
The most common pitfall in model selection is treating a “model capability ranking” as the selection conclusion. The reason is simple: different models are optimized under different constraints. Some pack the budget into total parameters to buy knowledge breadth, some into the reasoning stage to buy math and code ability, and some use small parameters to buy low latency and on-device deployment. These orientations are not better or worse than one another—only a matter of whether they match your scenario.
So the first step is not to look up a table, but to break your business requirements into four types of hard constraints:
- Effectiveness constraints: task type (general Q&A / long-document understanding / structured extraction / math reasoning / code generation), acceptable error rate, and whether errors are reversible.
- Cost constraints: acceptable cost per request, monthly total budget, and the scaling ceiling at peak concurrency.
- Compliance constraints: whether data can leave the internal network, whether the open-source license permits commercial use and redistribution, whether private deployment is required, and whether model outputs need review and audit trails.
- Operations constraints: how many accelerator cards you have, whether you have staff for parallel inference and quantization tuning, and whether you accept the latency and rate limits of third-party hosted services.
Only once the constraints are written down does the evaluation become meaningful. Otherwise, you’ll end up with a beautiful leaderboard and a system that crashes on launch.
2. Six Evaluation Dimensions: The Real Cost Behind Each
1. Parameter Count and Activation Strategy
Distinguish between total parameters and activated parameters. In a dense architecture the two are equal; in a mixture-of-experts (MoE) architecture, a single forward pass activates only a small subset of experts, so activated parameters are usually a small fraction of the total (commonly single digits to low double-digit percentages).
This leads to a key inference: memory usage is determined by total parameters, while inference speed is determined by activated parameters. It follows that the idea of MoE models “running fast and cheap” is an illusion—they save compute, not memory. If your cards can’t hold the total parameters, MoE’s high throughput is moot. When selecting a model, calculate memory first, then talk about speed.
Rules of thumb:
- Dense small models (hundreds of millions to billions of parameters): low latency, small memory footprint, suitable for high-concurrency online scenarios and on-device/edge deployment.
- Medium-activation MoE: clear throughput-cost advantage, suitable for high-concurrency centralized services with sufficient memory.
- Ultra-large total-parameter models: strong capability but a high deployment barrier, usually better accessed through hosted services than self-hosted.
2. Inference Mode: General Chat vs. Reasoning-Specialized vs. Distilled
A single model generation often spawns several branches, and mixing them up is the most common mistake in evaluation:
- General chat version: balances capability and latency, suitable for the vast majority of business Q&A, summarization, and extraction.
- Reasoning-specialized version: heavy investment in chain-of-thought (CoT) training, notably higher scores on math competition and complex code benchmarks, at the cost of a large increase in output tokens—note that this can multiply your inference cost, and you often can’t turn off the thinking process, only cap the budget through backend parameters.
- Distilled version: transfers the reasoning behavior of a large model to a smaller one with hundreds of millions to tens of billions of parameters, approximating part of the large model’s ability on a math reasoning benchmark, and can run on a single card or even a CPU. Its weakness is narrow knowledge coverage and instruction-following that is less stable than the original.
Selection advice: don’t expect one model to do everything. A more pragmatic architecture is routing—send simple requests to a small model or cache, and escalate only complex requests to the reasoning-specialized version.
3. Context Length: Nominal Value vs. Effective Value
The context window advertised by vendors (32K / 128K / 256K, or even million-level) is a different thing from the effective length usable in business. When evaluating, verify three things:
- Mid-context recall: place key facts at the head, middle, and tail of a long input, ask questions about each, and observe whether accuracy collapses (often called “lost in the middle”).
- Actual gains from long-context tasks: long context is often more expensive than “chunking + retrieval + reranking.” If your corpus exceeds hundreds of thousands of tokens, retrieval-augmented generation (RAG) is almost mandatory, while long context is better suited as a fallback and cross-check.
- KV Cache memory cost: context length linearly scales the KV Cache, which in turn limits concurrency. This is the most easily overlooked hidden cost of long context. Sparse attention and similar mechanisms can ease it, but check whether the specific implementation is fully supported by the inference engine.
4. Input Modality
Multimodal capability cannot be compared alongside pure-text capability. A model’s score on a text benchmark says nothing about its performance on image understanding, chart parsing, or video frame extraction. If the business genuinely needs visual input, you must build a separate multimodal evaluation set (one batch each of invoices, screenshots, scanned documents, surveillance footage), and evaluate both the OCR pipeline and image-preprocessing failure rates.
Conversely, if the business only needs text, choosing a text-only model is the more economical option—multimodal weights usually mean larger memory usage, and you won’t use that overhead.
5. Open-Source License and Compliance
“Open weights” does not mean “unrestricted commercial use.” During review, confirm at least the following:
- License type: permissive licenses (allowing commercial use, modification, and redistribution, with only notice retention required) versus licenses with additional terms (which may limit commercial scale, require attribution, restrict specific uses, or require derivative models to carry the same license) differ enormously.
- Whether distillation and secondary training are allowed, and how the resulting artifacts are attributed.
- How the model’s training data sources and content-safety responsibilities are divided.
- Whether weights, configuration, and inference code are all complete—a model that ships only weights without code significantly raises reproduction cost.
Compliance review should be moved ahead of the POC, not filled in before launch.
6. Deployment Compatibility and Inference Efficiency
Whether a model “can run at low cost” depends on how well it fits your existing inference stack:
- Engine and format support: mainstream open-source inference engines often take time to adapt to new architectures, and Day-1 support for new models is frequently incomplete (e.g., a certain attention implementation, quantization format, or parallel strategy not yet wired up).
- Quantization feasibility: weight quantization and low-bit activation quantization can significantly reduce memory, but they require calibration data and measured accuracy loss. Not all architectures can be quantized losslessly.
- Parallelism and scheduling capability: tensor parallelism, pipeline parallelism, expert parallelism, prefill/decode disaggregation (PD separation), prefix-cache reuse, continuous batching, and speculative decoding—these optimizations determine your per-token cost under real concurrency.
- Observability: whether it can connect to unified logging, metrics, tracing, and alerting directly determines whether a fault can be located within five minutes.
3. Three-Tier Evaluation: From Public Leaderboards to Real Business
Tier 1: Public Benchmarks (for coarse filtering, not for conclusions)
Common public benchmarks can be understood in four categories: multidisciplinary knowledge Q&A (the MMLU family), math applications and competition problems (GSM8K, AIME family), code and repository-level repair (the SWE-bench family), and human-preference battle rankings (the Arena ELO family).
When reading leaderboards, pay close attention to three limitations:
- Data contamination: the training corpus may already contain the test questions, inflating scores.
- Prompt sensitivity: the same model can fluctuate by several percentage points across different prompt templates or sampling parameters. Evaluation must fix temperature, top-p, max output length, and the prompt template, and record the versions.
- Distribution mismatch: leaderboards test general capability, but your business might be 80% structured field extraction. In that case, the leaderboard ranking may be almost unrelated to actual performance.
Tier 2: Internal Business Evaluation Set (This Is the Decision Basis)
Construction method:
- Sampling: extract 300–1000 real requests from online logs, stratified by business type, keeping edge cases and failure samples.
- Annotation: define the judgment criteria for each task family. Extraction tasks can use field-level exact matching; for generation tasks, combine “rule validation + model judge + manual spot-check,” where the model judge must first be calibrated for consistency against manual annotations.
- Metrics: task success rate, field-level F1, hallucination rate (proportion of unsupported statements), format compliance rate, refusal rate, and average output tokens.
- Regression: solidify the evaluation set into a CI task, and run it whenever the model or prompt changes, to prevent silent regression.
An evaluation skeleton you can apply directly (pseudocode, with the interface pointing to your own internal inference gateway):
import requests, json, statisticsGATEWAY = "http://llm-gateway.internal/v1/chat"def call(model: str, messages: list[dict], **params) -> dict:payload = {"model": model, "messages": messages, **params}r = requests.post(GATEWAY, json=payload, timeout=120)r.raise_for_status()return r.json()def evaluate(candidates: list[str], dataset: list[dict], judge) -> dict:report = {}for model in candidates:rows = []for case in dataset:resp = call(model, case["messages"],temperature=0.0, max_tokens=case.get("max_tokens", 1024))usage = resp.get("usage", {})rows.append({"ok": judge(case, resp["choices"][0]["message"]["content"]),"out_tokens": usage.get("completion_tokens", 0),})report[model] = {"success_rate": round(sum(r["ok"] for r in rows) / len(rows), 4),"avg_out_tokens": round(statistics.mean(r["out_tokens"] for r in rows), 1),}return report# Key constraint: temperature, template, and version must be recorded in the report,# or the results cannot be reproducedprint(json.dumps(evaluate(["candidate-a", "candidate-b"], load_dataset()), indent=2))
Tier 3: Online Performance and Cost Load Testing
After the offline evaluation is done, you must run real-concurrency load testing; testing only latency and not throughput leads to wrong conclusions. Collect at least four metrics:
- TTFT (time to first token): determines the interactive feel, and is mainly driven by the prefill stage and queue length.
- TPOT (time per output token): determines long-text reading speed.
- System throughput (tokens/s, or requests completed per second): determines unit cost.
- Error rate and queue-timeout rate: observed under 1.5x peak business load, to find the latency inflection point rather than the crash point.
Do the load testing in tiers: run one round each at concurrency 1 / 4 / 16 / 64, recording both P50 and P99. P99 is the number users actually perceive.
4. Memory and Cost Estimation: Two Formulas Are Enough
Memory requirement (single instance, rough estimate)
Memory ≈ weight bytes + KV Cache bytes + activation and framework overheadWeight bytes ≈ total parameters × bytes per parameter (FP16=2, INT8=1, INT4≈0.5, plus quantization metadata overhead)KV Cache ≈ concurrency × sequence length × layers × 2 (K and V) × hidden dimension × bytes per elementFramework overhead ≈ 15%–30% of (weights + KV) (CUDA context, temporary buffers, paging fragmentation, etc.)
Example: a MoE model with roughly 700B total parameters loaded in FP16 needs about 1.4TB for the weights alone, which already exceeds the common memory total of a single eight-card machine, and must rely on multi-machine tensor parallelism plus weight quantization, or simply switch to calling a hosted service. This example shows one thing: calculating before selecting can save you half of the wasted evaluation.
Cost per request
Cost per request = instance card-hour cost / (effective throughput tokens/s × 3600) × total tokens per requestWhere total tokens per request = input + output + retries + repeated overhead of system prompt and tool definitions
The easiest things to miss here are retries and the system prompt. A 2000-token system prompt combined with a 20% retry rate pushes actual cost well above the theoretical value. Prefix caching can eliminate the repeated computation of the system prompt, and it is the first priority for cost optimization.
5. Inference Optimization Checklist (Ranked by ROI)
- Prefix caching: place the fixed system prompt, few-shot examples, and tool definitions all up front and in a fixed order; the gains are largest when the hit rate is high.
- Quantization: prefer low-bit weights + low-bit activations under an acceptable accuracy loss; after quantization, you must regress against the internal evaluation set—don’t just look at perplexity.
- Dynamic routing: let small models handle simple intents and escalate complex requests; combined with rule-based fallbacks, this cuts both cost and latency.
- Request scheduling: continuous batching plus separating long and short requests, to avoid long-output requests blocking the short-request queue.
- Parallelism and disaggregation: use tensor parallelism when memory is insufficient; at larger scale, consider disaggregated prefill/decode deployment, scaling each at different ratios.
- Output budget: explicitly set max output length, and cap the thinking budget for reasoning tasks—this is the most direct cost-reduction measure.
- Architecture-level substitution: for ultra-long context, prefer retrieval + reranking + summary compression over “stuffing in the full text.”
If your team doesn’t have the conditions to self-host a multi-machine inference cluster, hosted inference services are one option: cloud providers such as Baidu AI Cloud typically offer both model inference services and general capabilities like container platforms, object storage, logging, and monitoring/alerting, letting you offload scaling, rate limiting, auditing, and observability to the platform—at the cost of accepting external constraints on latency and quotas. Whether to self-host depends on where your concurrency scale, data-egress requirements, and long-term cost curve cross.
6. Seven Common Pitfalls
- Making a final decision from public leaderboards: leaderboards are only responsible for filtering out obviously unsuitable candidates.
- Treating activated parameters as total parameters for memory: MoE saves compute, not memory.
- Treating nominal context as effective context: without testing mid-context recall, you haven’t tested at all.
- Ignoring license terms: legal must review before commercial use, especially terms that limit commercial scale or require attribution.
- Testing only offline effectiveness and not concurrency: latency inflection points and queue timeouts only surface under pressure.
- Underestimating output-token cost: the thinking process of reasoning-specialized versions can multiply cost several times over.
- No regression baseline: any change to the model, prompt, or quantization configuration causes silent regression.
7. An Actionable Selection Scorecard and Rollout Cadence
Use a weighted scorecard to converge the discussion, scoring each item from 1 to 5, with weights adjusted to the business:
| Dimension | Observed Metric | Suggested Weight |
|---|---|---|
| Business effectiveness | Internal evaluation set task success rate, hallucination rate | 35% |
| Latency experience | TTFT P99, TPOT P99 | 15% |
| Throughput cost | Cost per request, peak sustainable concurrency | 20% |
| Deployment feasibility | Memory usage, quantization and parallelism support, engine adaptation completeness | 15% |
| Compliance | License type, data-egress requirements, audit capability | 10% |
| Ecosystem and operations | Version iteration cadence, documentation and community support, observability integration | 5% |
A suggested rollout cadence in four steps:
- POC (1–2 weeks): 3–5 candidates, using only the internal evaluation set plus single-request latency, eliminating half of them.
- Load testing (1 week): the remaining candidates get concurrency load testing and memory accounting, yielding the true unit cost.
- Canary (2–4 weeks): launch at 5% traffic, enable full-link logging and metrics dashboards, and manually spot-check hallucinations and refusals.
- Scale-up: solidify the routing strategy, cache hit rate, and fallback plans, and connect the evaluation set to CI for continuous regression.
Conclusion
“Which model is best” is hard to answer precisely because it has no answer detached from constraints. What really determines success or failure is three things: breaking requirements into measurable constraints, using the internal evaluation set instead of public leaderboards to make decisions, and confirming cost on the load-testing bench rather than the configuration sheet. Do these three things, and no matter how fast models iterate, your selection process won’t have to be redone from scratch—because what changes is only the names in the candidate pool, while the evaluation framework and launch standards remain the same.
