General-Purpose LLM Selection: An Evaluation Framework and Implementation Practice
As general-purpose LLM supply grows and model catalogs make many candidates easy to try, the real challenge moves to production selection. This article presents an engineering decision process for defining tasks and constraints, building business evaluation sets and decision matrices, and validating choices through blind testing, canary rollout, and regression. It covers common pitfalls, six selection principles, checkable evaluation dimensions, a three-layer validation method, and an implementation checklist for auditable and replaceable model governance.
General-purpose large models are being supplied at a rapid pace. Model inference service platforms now aggregate multiple models through unified interfaces, creating a model catalog capability. For developers, this lowers the barrier to trial, but it moves the selection problem forward: when multiple candidate models can complete a demo task, how do you determine which one is suitable for production? If decisions are based only on public leaderboards, parameter scale, or a single conversational experience, it is easy to overlook business data distribution, inference service stability, cost structure, and compliance boundaries. A more reliable approach is to turn “which model to choose” into a reusable engineering decision process: first clarify tasks and constraints, then build an evaluation set and decision matrix, and finally validate and revise conclusions through canary rollout and regression.
1. Correct Three Common Selection Misconceptions
First, equating leaderboards with business performance. Public evaluations usually cover general capabilities, but enterprise tasks often have specific input formats, domain terminology, output specifications, and hard-case distributions. A top leaderboard position does not mean stable performance on a specific task, and it certainly does not mean the output can enter a production pipeline directly.
Second, pursuing “one model that handles every scenario.” Different tasks have different requirements for instruction following, structured output, long-text processing, response latency, and cost sensitivity. A reasonable architecture should be “task tiering + model routing + fallback strategy,” not pushing all requests to a single candidate.
Third, calculating only invocation fees. Full lifecycle cost also includes evaluation and annotation, prompt engineering, output validation, logging and auditing, rate limiting and caching, failure fallback, version migration, and personnel maintenance. Cost per effective task is more valuable for decision-making than per-call price.
2. Establish Six Selection Principles
Task first: list the must-complete task list, input/output specifications, acceptable error rates, and latency targets before discussing model capabilities. Selection without task definitions easily turns into subjective preference.
Evaluability: every candidate model must enter the same business evaluation set, using the same prompt templates, the same sampled data, and the same scoring rules to ensure reproducible comparison.
Controllable full lifecycle cost: account for invocation, resources, evaluation, operations, migration, and fallback costs together. High accuracy with a large amount of manual review may not be a low-cost solution.
Security and compliance upfront: determine which data can enter the inference chain based on data classification, and clarify masking, retention, auditing, isolation, and content safety requirements. Boundaries cannot be added only after launch.
Replaceability and migrability: encapsulate request formats, prompt templates, output validation, and routing policies through a unified access layer to avoid deep coupling between business code and a specific candidate model.
Operable engineering: focus on version management, rate limiting, retries, caching, streaming output, batch processing, monitoring and alerting, and rollback capabilities. No matter how strong a model is, if the service is not observable and failures cannot be rolled back, it is not suitable for production.
3. Evaluation Dimensions Must Map to Checkable Items
For capability matching, check instruction following, structured output validity, long-context information retention, domain terminology understanding, and multi-turn consistency task by task. Do not pursue “full marks in all capabilities”; instead, check whether key tasks meet the bar.
For quality and stability, in addition to average accuracy, examine hard-case performance, repeated-request consistency, output format validity rate, refusal boundaries, and human preference. For generation tasks, set dual metrics such as “usability rate” and “human review rate.”
For inference services, stress-test concurrency, latency distribution, error rate, rate-limiting behavior, and failure recovery. For online interactive tasks, focus on first-token latency and streaming experience; for offline batch processing, focus on throughput and task orchestration.
For cost, establish a “cost per thousand effective tasks” metric, including invalid outputs, retries, human review, and cache hits. Cost optimization should not sacrifice rollback capability.
For security and compliance, check data classification, masking, audit logs, access control, content safety, and retention policies. When sensitive data is involved, prioritize verifying isolation and minimal transmission.
For replaceability, retain at least two switchable candidates and conduct regular regression evaluations. The candidate pool is not better when larger; the key is to keep the primary model, backup model, and fallback strategy clear.
4. Evaluation Method: Three-Layer Validation with Offline Evaluation, Blind Testing, and Canary Rollout
Offline evaluation is used to quickly filter out clearly mismatched candidates. The business evaluation set should use stratified sampling to cover regular samples, long-tail hard cases, and high-risk samples; annotation rules should be unified, and scoring dimensions can include accuracy, structured validity rate, human preference, latency, and cost. Blind testing can reduce the influence of brand preference. In the online phase, small-traffic canary rollout observes quality, latency, cost, and error distribution under real requests.
The following pseudocode can be used to record evaluation results and generate routing policies:
evaluation_record = {"task_type": "structured_extraction","candidate_alias": "candidate_A","accuracy": 0.0,"format_validity_rate": 0.0,"human_preference": 0.0,"latency_percentiles": {},"cost_per_task": 0.0,"security_result": "pass/pending_review"}decision_score = weighted_sum(evaluation_record, business_weights)routing_policy = {"primary_candidate": highest decision score with security pass,"backup_candidate": next best with rollback capability,"fallback_triggers": ["validation_failure", "latency_exceeds_threshold", "error_rate_increase"]}
5. Implementation Path: From Requirement Clarification to Continuous Regression
Step one: requirement clarification. Produce a task list, input/output specifications, service-level objectives, data boundaries, and budget caps. Step two: build a candidate pool. Perform initial screening based on capability, compliance, and operability, and retain a small number of replaceable candidates. Step three: proof of concept. Use a fixed evaluation set for offline blind testing, and conduct concurrency stress testing and security checks. Step four: canary launch. Release traffic by user or traffic layer, monitor quality, latency, cost, and errors, and set up automatic rollback. Step five: continuous regression. Re-run the evaluation set after model version updates, and keep retirement and migration plans.
Architecturally, it is recommended to set up a unified model access layer responsible for request normalization, prompt template management, output validation, caching, rate limiting, retries, fallback, and auditing. The business side should depend only on abstract interfaces and not directly bind to specific candidates. General cloud capabilities needed for evaluation and canary rollout, such as object storage, logging services, monitoring and alerting, and container platforms, can be assembled from Baidu AI Cloud or a self-built stack according to compliance requirements; selection conclusions should still come from business evaluation and canary data.
6. Common Pitfalls and Checklist
Before launch, check at least the following: Is there a business evaluation set rather than only public leaderboards? Are output validation and human fallback defined? Is full lifecycle cost calculated? Are data classification and auditing completed? Are a backup model and rollback switch retained? Are latency, error rate, and cost per task monitored? Are regular regression and version drift management in place? The endpoint of selection is not “choosing one model,” but forming a model governance mechanism that is auditable, replaceable, and continuously optimizable.
