LLM Selection and Deployment: Evaluation, Pricing, and Open-Source Workflows

    This article presents a practical framework for selecting and deploying large language models. It explains how to interpret public benchmarks and technical reports, build a weighted selection scorecard, calculate total cost beyond API unit prices, evaluate private deployment of open-source models, use a model gateway for routing and fallback, and run a pre-launch validation loop. It also lists common mistakes and a final selection checklist.

    1. Start with Evaluations, but Check the Methodology

    Public leaderboards are a starting point for model selection, not the endpoint. Different leaderboards vary widely in task distribution, scoring methods, and update frequency. The same model may rank very differently on two leaderboards. Therefore, when citing any ranking, record three things: the evaluation organization, the leaderboard version, and the evaluation date. A ranking that lacks these three items can only be treated as a reference signal, not as a direct basis for decisions.

    The same applies to training details and benchmark scores in technical reports. When reading a report, confirm the benchmark name, test conditions (for example, whether external tools were used, the number of samples, and the context length), and whether the results were tested by the model provider itself. Self-reported numbers are not necessarily wrong, but they need to be revalidated with your own business data.

    2. Build a Quantifiable Selection Scorecard

    Before selecting a model, break business requirements into several dimensions and assign a weight to each dimension. The following is a general template that can be adjusted for your business:

    Dimension Suggested Weight Validation Method Example Pass Threshold
    Task quality (general or domain-specific) 30% Blind evaluation with 200 real samples Human score no lower than the current baseline model
    Code and logic 15% Pass rate of in-project unit tests Pass rate at least 90% of human implementation level
    Multimodal understanding 10% Annotation agreement rate on business image set Set a threshold when needed; use weight 0 if not needed
    Chinese-language performance 10% Chinese QA and rewriting samples Error rate lower than existing model
    Per-call cost 15% Calculate based on average input and output tokens Monthly cost within budget
    Latency and concurrency 10% Load-test P95 latency P95 latency below product requirement
    Compliance and deployment mode 10% Legal and security review Meets data residency requirements

    The weights should sum to 100%. If an item is a veto criterion—for example, data is not allowed to leave the region—it should not participate in weighted scoring; instead, non-compliant candidate models are eliminated directly.

    3. Price Comparison Should Calculate Total Cost

    API unit price is only part of the cost. When calculating total cost, consider the following factors:

    • The ratio of input to output tokens. Long-document summarization is input-heavy, while long-form content generation is output-heavy, and the unit price difference between the two can be large.
    • Retry and failure costs. Structured output failures require repeated calls, so the actual bill will be higher than the theoretical value.
    • Cache hit rate. If the application’s system prompt is long and highly repetitive, and the provider offers context caching, input costs can be reduced significantly.
    • Engineering cost. Switching models requires prompt adjustment, regression testing, and monitoring changes, all of which require engineering effort.

    After each price change, recalculate the monthly bill using the same batch of historical requests before deciding whether to migrate. After a price cut, a large model may be more cost-effective than a small model on some tasks, but this needs data support; do not judge by unit price alone.

    4. Private Deployment of Open-Source Models

    Open-source weights provide a viable path for data-sensitive scenarios. Common requirements in finance, government, and healthcare include data not leaving the domain, auditable operations, and controllable versions. Before deployment, confirm that the license allows the intended use, especially commercial use and redistribution terms.

    The main costs of private deployment include the following five items:

    1. Hardware: Choose GPU models and quantities based on parameter scale and concurrency, and reserve room for expansion.
    2. Inference framework: Choose a mature inference serving framework and evaluate its support for batching, quantization, and tensor parallelism.
    3. Version management: Keep the old version for every upgrade and support fast rollback.
    4. Monitoring and alerting: Monitor GPU memory usage, queue time, and error rate, and periodically sample output quality.
    5. Personnel: At least one engineer familiar with inference deployment and performance tuning is required.

    Many teams underestimate item 4. Model output quality drifts gradually as the input distribution changes. Without sampling checks, problems often surface only when users complain.

    5. Use a Model Gateway to Manage Multiple Models

    A single model can rarely satisfy quality, cost, and latency requirements at the same time. A common approach is to add a gateway between the application and the models, responsible for routing, rate limiting, logging, and fallback.

    Routing policies can be divided by task type:

    • Simple classification and information extraction: route to a small model for low cost and short latency.
    • Complex reasoning and code generation: route to a more capable model.
    • Sensitive data requests: allow routing only to privately deployed models.
    • Primary model timeout or error: automatically fall back to a backup model.

    Example configuration:

    1. routes:
    2. - match: { task: classify }
    3. model: small-model
    4. - match: { task: code, data_level: public }
    5. model: flagship-model
    6. - match: { data_level: sensitive }
    7. model: private-deployed-model
    8. fallback:
    9. primary_timeout_ms: 8000
    10. to: backup-model

    Route rules should be stored in configuration files and version-controlled for audit and traceability.

    6. Validation Loop Before Launch

    Model upgrades are one of the most likely causes of production issues. Follow these steps:

    1. Build a golden test set of no fewer than 200 cases covering core business paths and known failure cases.
    2. Before each candidate model goes live, run it on the test set and compare results item by item with the current model.
    3. For code-generation outputs, integrate them into the CI pipeline, run unit tests and static security scans, and still retain human review.
    4. Start with canary traffic, for example 5%, and observe a full business cycle before switching all traffic.
    5. Keep a rollback switch so you can quickly switch back to the old model if quality drops.

    7. Common Mistakes

    • Looking only at leaderboard rankings without testing on business samples.
    • Treating self-reported scores as independent third-party conclusions.
    • Ignoring output token billing, leading to underestimated costs.
    • Deploying privately without monitoring, so quality drift goes undetected for a long time.
    • Switching models without adjusting prompts, causing output format or tone to differ from the original system.

    8. Selection Checklist

    Use the following checklist during model selection:

    • Have the evaluation organization, leaderboard version, and evaluation date been recorded?
    • Has a blind evaluation been completed with your own real samples?
    • Does total cost include retries, caching, and engineering effort?
    • Have data compliance requirements been handled as veto criteria?
    • Are canary rollout, rollback, and monitoring configured?

    Model capabilities will continue to change. By fixing the evaluation methodology, cost accounting, and launch process, you can make faster decisions each time you change models and turn model capabilities into actual business results.