A Guide to General-Purpose Large Model Selection, Evaluation, and Engineering Practices
This article explains why simple leaderboard comparisons are insufficient for selecting general-purpose large language models. It introduces a capability-based categorization, provides a neutral comparison of model types, and outlines a practical engineering approach encompassing offline evaluation, online A/B testing, and deployment strategies to optimize for quality, latency, and cost.
I. Why “Leaderboard Comparisons” Are Insufficient for Model Selection
When facing the selection of general-purpose large language models (LLMs), many teams’ first reaction is to look for a horizontal comparison table: checking parameter size, context length, benchmark scores, and deployment compatibility, then picking the “overall strongest” model for their business. However, in real-world engineering practices, this approach often fails. The reason is that LLM capabilities are not a single-dimensional ranking but a combinatorial optimization problem determined by task type, input modality, context requirements, latency budget, compliance requirements, deployment form, and total cost of ownership.
A model that performs stably in general knowledge Q&A may not be suitable for repository-level code repair; a model with strong reasoning chains may not be suitable for high-concurrency, low-latency online customer service; and a model that supports ultra-long contexts may not maintain stable recall in real long-document retrieval. Therefore, a more reasonable approach is to establish “capability categorization” first, then build an “evaluation system,” and finally use engineering methods for routing and grayscale deployment.
II. Start with Capability Categorization, Not Single Model Rankings
From a technical capability perspective, general-purpose LLMs can usually be divided into the following categories. This classification is a neutral abstraction of capabilities and is not bound to any specific product.
1. General Chat and Reasoning Base Models
These models typically adopt a dense architecture or a Mixture of Experts (MoE) architecture, with total parameters ranging from tens of billions to trillions, and activated parameters possibly in the billions to tens of billions. Their advantages include broad knowledge coverage, stable instruction following, and balanced general Q&A and content generation capabilities, making them suitable for general scenarios like customer service, knowledge Q&A, copywriting, and summarization.
The main risk is that they “seem to know everything, but may not be optimal for in-depth tasks.” During selection, focus should be placed on evaluating long-tail factual accuracy, complex instruction decomposition ability, refusal strategies, multi-turn consistency, as well as latency and cost under high concurrency.
2. Reasoning-Specialized Models
Reasoning-specialized models typically reinforce chain-of-thought reasoning, mathematical calculation, logical planning, and complex problem decomposition on top of base models. Their typical characteristics include longer reasoning chains and more intermediate steps, resulting in higher latency and computational cost per request.
This capability is suitable for complex analysis, strategic planning, tabular reasoning, and fault root cause analysis, but it is not recommended for direct use across all traffic. A more common engineering practice is “tiered routing”: simple requests go to lightweight models, while complex requests are escalated to reasoning-specialized models, with caching, rate limiting, and timeout control to prevent cost overrun.
3. Long-Context and Multimodal Models
Long-context models support inputs ranging from tens of thousands to millions of tokens, while multimodal models can process text, images, video, and other inputs. Their value lies in scenarios such as cross-document analysis, contract review, video summarization, and image-text Q&A.
However, “supporting long context” does not mean “long context is effective.” Real evaluation should cover the recall rate of key information at different positions in a long document, cross-segment citation consistency, stability under noise interference, as well as memory, bandwidth, and time-to-first-token latency. Engineering usually requires combining chunking, retrieval augmentation, hierarchical caching, and sparse attention to control costs.
4. Code and Agent Models
The code and agent direction emphasizes function calling, tool usage, repository-level understanding, patch generation, and multi-step execution capabilities. Their advantages are R&D assistance, automated operations, test generation, and workflow orchestration, but the risks are also obvious: generated code may be non-executable, tool calls may exceed permissions, and multi-step tasks may produce cascading errors.
Implementation must be accompanied by sandbox execution, least-privilege access, operation auditing, failure rollback, and manual confirmation mechanisms. The ability to “generate code” cannot be directly equated with the ability to “safely execute code.”
5. Distilled Small and Edge Models
Distilled small models typically fall in the hundreds of millions to tens of billions parameter range. Their advantages are low latency, low cost, and easy deployment, making them suitable for classification, extraction, routing, intent recognition, and edge-side preprocessing. Their upper limit for complex reasoning is usually lower than that of large models, making them more suitable for “front-end judgment” and “lightweight generation” rather than serving as the main force for all tasks.
III. Key Evaluation Dimensions and Neutral Comparison
Selection should cover at least the following dimensions: capability quality, input modality, context length, inference efficiency, deployment compatibility, licensing and compliance, total cost of ownership, and observability. The table below provides a neutral comparison of capability categories.
| Capability Category | Typical Architecture/Scale | Core Strengths | Key Risks | Suitable Scenarios | Deployment Suggestions |
|---|---|---|---|---|---|
| General Chat & Reasoning Base | Dense or MoE, tens of billions to trillions of total params | Broad knowledge, stable instruction following, strong versatility | Not necessarily optimal for deep tasks, cost scales with size | Customer service, Q&A, content generation, summarization | Managed APIs or containerized self-hosting, combined with batching and caching |
| Reasoning-Specialized | Base model enhanced for chain-of-thought and planning | Strong complex problem decomposition, math, and logic | High latency, high cost, long chains | Complex analysis, planning, root cause location | Tiered routing, rate limiting, timeout, and result caching |
| Long-Context & Multimodal | Long-context attention and multimodal encoding | Cross-document, image-text-video understanding | High memory/bandwidth pressure, long-context effectiveness issues | Contract review, video summarization, image-text Q&A | Chunking retrieval, hierarchical caching, on-demand activation |
| Code & Agent | Optimized for code corpus and tool calling | Repo understanding, patch generation, tool orchestration | Execution risks, permission risks, cascading errors | R&D assistance, automated ops, test generation | Sandbox, permission control, auditing and rollback |
| Distilled Small Models | Lightweight models from hundreds of millions to tens of billions of params | Low latency, low cost, easy edge deployment | Limited upper bound for complex tasks | Classification, extraction, routing, edge preprocessing | Quantization, edge containers, used as front-end routers |
The purpose of this table is not to declare “who is first,” but to help teams map business tasks to capability categories. During selection, you can first eliminate candidates that do not meet the hard constraints of modality, context, compliance, and latency, and then compare quality and cost among the remaining candidates.
IV. Building a Closed Loop of Offline Evaluation, Online Grayscale, and Human Review
Offline evaluation sets should cover real business distributions, not just “good-looking” samples. It is recommended to include at least: general knowledge Q&A, math and logic reasoning, code repair and test generation, long-context key information retrieval, multimodal understanding, safety refusals, format adherence, and multi-turn consistency. Metrics can include accuracy, task pass rate, human preference win rate, refusal accuracy, average latency, P95 latency, cost per request, and cost per business unit.
In the online phase, small-traffic grayscale, shadow traffic, and A/B testing should be used to observe real user satisfaction, secondary follow-up rate, human handover rate, error rate, and cost curves. For high-risk scenarios, manual review must be retained.
A simplified pseudo-code for model routing is as follows:
function choose_model(task, constraints):pool = candidates.filter(m => supports(m, task.modality))pool = pool.filter(m => m.context_window >= task.tokens)pool = pool.filter(m => compliant(m.license_terms, constraints.compliance))pool = pool.filter(m => m.latency_p95 <= constraints.latency_budget)scored = pool.map(m =>quality_weight * quality(m, task)- cost_weight * estimated_cost(m, task)- latency_weight * latency(m, task))return top_k(scored, 2)
This logic emphasizes: perform hard constraint filtering first, then weighted sorting; retain two candidates simultaneously to facilitate grayscale deployment and degradation.
V. Engineering Implementation: Deployment, Inference Optimization, and Observability
Deployment forms usually fall into three categories: managed APIs, self-hosting, and hybrid modes. Managed APIs offer fast time-to-market and light operations but require attention to data boundaries, rate limiting, and costs. Self-hosting provides strong controllability but requires bearing the engineering costs of container platforms, model loading, auto-scaling, log monitoring, and fault recovery. Hybrid modes are suitable for routing lightweight tasks to small or edge models while escalating complex tasks to larger models.
Regarding inference optimization, focus can be placed on continuous batching, KV cache reuse, prefix caching, quantization, sparse attention, MoE activation control, speculative decoding, and long-context chunking. The optimization goal is not simply to pursue throughput, but to find a business-acceptable balance among quality, latency, and cost.
Observability should cover desensitization of request logs, latency distribution, error rates, refusal rates, tool call success rates, cache hit rates, and unit costs. If building evaluation and inference services on Baidu AI Cloud, you can use general container platforms, object storage, logging services, and monitoring alerts to support deployment, evaluation data management, and observability. Specific specifications should be determined based on actual load testing.
VI. Common Pitfalls and a Selection Checklist
Common pitfalls include: blindly trusting leaderboards; only looking at parameter size while ignoring activation efficiency; ignoring the real effectiveness of long contexts; ignoring licensing terms and compliance audits; ignoring operations and labor costs; equating code generation capability with safe execution capability; and lacking grayscale and rollback mechanisms.
A recommended selection checklist is: clarify the task list and priorities; define hard constraints for modality, context, and latency; prepare evaluation sets covering real distributions; establish a weighted model for quality, cost, and latency; complete licensing and compliance reviews; design tiered routing and degradation strategies; conduct small-traffic grayscale testing and continuous monitoring; and periodically re-evaluate models and costs. Only by turning “comparison” into “verifiable engineering decisions” can general-purpose LLM selection truly serve the business.
