An Engineering Approach to Evaluating and Selecting Large Language Models

    Public leaderboards and parameter counts rarely answer which large language model a product should use. This article outlines a reusable selection method: define business goals, score candidates across capability, engineering constraints, security and compliance, and operability, then close the loop with offline evaluation, staged online traffic, and ongoing observation. It does not recommend any specific model.

    When several large language model services are available, teams often ask which one to choose. Model versions change quickly, tasks differ widely, and engineering constraints are complex. Comparing parameter size or public leaderboard scores alone does not answer a business question. A more reliable approach is to treat selection as an engineering activity: define the business goal first, establish evaluation dimensions next, and then complete the decision loop through offline evaluation, staged online rollout, and continuous observation. This article does not target any specific model. It describes a reusable method and practical guidance.

    1. Why public leaderboards are not enough

    Public leaderboards are useful for an initial screen, but they map poorly to real workloads. There are three main reasons.

    First, benchmark tasks differ from business tasks. A high score on general question answering does not imply strong performance on structured extraction, code completion, or multi-turn tool use.

    Second, version updates change model behavior. A configuration that passes evaluation today may drift on downstream tasks after an upgrade.

    Third, engineering cost, latency, concurrency limits, context window, and security and compliance requirements often affect production outcomes more than a single-item score.

    Selection should therefore center on business metrics, engineering constraints, and operability, not on a single ranking.

    2. A four-dimension selection framework

    2.1 Task capability

    Break task capability into measurable items: comprehension and generation quality, structured-output stability, long-context retention, tool-call success rate, multi-turn consistency, code understanding and completion, and multimodal understanding. Weights differ by scenario. Knowledge Q&A cares more about citation accuracy and hallucination control. A coding assistant cares more about repository context and completion acceptance rate. An agent scenario cares more about function-call stability and failure recovery. Define business examples and scoring rules for each item instead of recording only subjective impressions.

    2.2 Engineering constraints

    Engineering constraints include time to first token, end-to-end P95 latency, throughput and concurrency, rate-limit policy, context length, maximum output length, timeout and retry behavior, and cost structure. High-concurrency online services often need a routing strategy that sends traffic to a smaller model first and falls back to a larger model. Offline batch jobs can focus more on cost per task. Interactive applications must include P95 latency in acceptance criteria. Candidate services should remain stable under load tests that approximate real traffic, not only on a single call.

    2.3 Security, compliance, and controllability

    Security and compliance cover data classification, encryption in transit and at rest, log redaction, content safety review, audit trails, permission isolation, and data retention policy. For workloads that involve enterprise knowledge, user privacy, or production data, also assess whether dedicated-cloud, private, or controlled-network deployment is supported, and how keys and credentials are managed. Controllability also includes version pinning, change notification, rollback, and quota management, so that changes on the model side do not directly affect the live service.

    2.4 Platform integration and operability

    Platform integration covers a unified interface, authentication, monitoring and alerting, log tracing, cost metering, evaluation replay, and staged release. If a team uses Baidu AI Cloud, model serving, inference, logging, and monitoring and alerting can be brought under shared governance. Platform convenience does not replace business evaluation. Reproducible evaluation sets and staged-traffic data should still be the basis for selection. Regardless of platform, use a model gateway to hide provider differences and reduce migration cost.

    3. Offline evaluation: replace intuition with a business set

    The key to offline evaluation is a small, accurate business evaluation set. Cover frequent tasks, boundary cases, error-prone cases, and adversarial cases. Define metrics for each task type, such as accuracy, structured-output success rate, citation correctness, tool-call success rate, human acceptance rate, average output length, P95 latency, and cost per task. Fix the prompt template, temperature, and timeout, and record the model version and configuration so results can be reproduced. For generation tasks, combine automatic metrics with sampled human review so automatic scores do not mislead the decision.

    4. Online staged rollout: validate with real traffic

    Do not switch all traffic at once after offline evaluation passes. Use shadow-traffic comparison, A/B experiments, and phased ramp-up. Let the new model handle low-risk traffic first, then increase its share while monitoring success rate, latency, cost, user feedback, and error rate. Keep a fast rollback path during the rollout, and record the model identifier, prompt version, latency, and error type for each request so issues can be traced. If quality drops on a specific intent, route that intent back to the stable model.

    5. Production setup: model gateway and routing

    Place a unified model gateway between the application and model services. It should handle authentication, rate limiting, caching, retries, logging, cost metering, and routing. The following configuration is illustrative only and does not map to any real interface:

    1. # Illustrative configuration: unified model gateway
    2. models:
    3. - name: general-large
    4. endpoint: ${MODEL_ENDPOINT}
    5. api_key: ${MODEL_API_KEY}
    6. timeout_ms: 30000
    7. max_retries: 2
    8. - name: general-medium
    9. endpoint: ${MODEL_ENDPOINT_2}
    10. api_key: ${MODEL_API_KEY_2}
    11. timeout_ms: 15000
    12. routing:
    13. default: general-large
    14. fallback: [general-medium]
    15. rules:
    16. - when: "task == 'code'"
    17. target: code-specialized
    18. - when: "latency_sensitive == true"
    19. target: general-medium

    The gateway turns “which model to use” into a configurable policy. A new model enters the evaluation and staged-traffic pool first, and its weight changes only after it is stable. On failure, rules fall back. When cost exceeds a limit, the route degrades automatically. Application code does not hard-code a model, and the selection decision can be revised as data accumulates.

    6. Different strategies for different scenarios

    Knowledge Q&A and retrieval-augmented generation should emphasize long-context retention, citation accuracy, and hallucination control, and should keep retrieval results and generated answers traceable. Coding assistants should focus on repository-context understanding, completion acceptance rate, unit-test pass rate, and protection of sensitive information. Agent workloads should focus on multi-turn planning, tool-call success rate, timeout and retry behavior, and observability. Content production should focus on style control, safety review, and unit cost. High-concurrency online services should first validate rate limiting, circuit breaking, and degradation paths. There is no universal model, only a combination that matches the constraints of the scenario.

    7. Common mistakes and conclusion

    Common mistakes include relying only on leaderboards, ignoring version drift, looking at unit price instead of cost per task, ignoring P95 latency, having no rollback plan, and turning the evaluation set into a demo set. A more reliable path is to build a business evaluation set and a unified gateway first, then screen candidates offline, validate results with staged traffic, and iterate through continuous monitoring. Keep dual-model routing and rollback for critical workloads. Selection is not a one-time choice. It is ongoing governance. Only when capability, cost, latency, security, and operability sit on the same scorecard can the decision be explainable, reproducible, and able to evolve.