General-Purpose LLM Selection: From Capability Evaluation to Multi-Model Orchestration
General-purpose LLMs now iterate faster than traditional software, so the idea of choosing one strongest model to handle every task no longer holds. This article lays out a reusable approach to model selection and orchestration: a ten-dimension capability evaluation matrix, a repeatable evaluation workflow built on business task sets and a first-pass rate, capability-based routing that assigns each task type to an appropriate model, and a generate-review-verify pipeline that combines models to raise quality. It also explains why non-capability factors such as latency, rate limiting, content alignment, data and account risk, and vendor lock-in shape production reliability, and why cost per task is a better metric than unit price.
General-purpose LLMs now iterate faster than traditional software. Some models release updates on cycles of just a few weeks, and the same model’s availability, rate-limiting policies, and even data-retention rules can change over time. In this environment, the idea of “picking one strongest model to do everything” no longer holds. A more practical approach is to treat models as a set of capabilities that can be scheduled flexibly per task: first build a multi-dimensional evaluation matrix, then route capabilities by task type, and use a generate-review-verify pipeline to guarantee delivery quality.
This article does not target any specific product. Instead, it distills a reusable method for selection and orchestration. It applies to enterprises building AI applications, developers designing Agent workflows, and content teams choosing tools for everyday writing and translation. The code here illustrates the ideas only, and omits engineering details such as exception handling, authentication, and retries.
1. Common Selection Pitfalls
Before diving into the method, it helps to clear up a few typical mistakes:
- Leaderboard-only thinking: The question distribution in public benchmarks often differs from real business. A model that leads on general leaderboards may not perform best on your high-frequency tasks.
- Single-point experience: Drawing conclusions from a few conversations relies on a sample that is too small, and it is easy to mistake an occasional good result for a stable capability.
- One model for everything: Translation, math, retrieval, and multimodal tasks place very different demands on capability, and it is hard for any single model to lead on every dimension.
- Comparing unit price only: Looking only at the price per million tokens ignores the hidden costs of rework, manual review, and timeout retries.
- Ignoring non-capability factors: Rate-limiting policies, service volatility, how strict the content-safety policy is, and terms of service and account risks all directly affect production availability.
2. Building a Multi-Dimensional Capability Evaluation Matrix
A practical evaluation matrix should cover at least the following ten dimensions:
| Dimension | Key Focus | Common Pitfall |
|---|---|---|
| Chinese writing | Wording naturalness, style fit, voice consistency | Mistaking ornate vocabulary for good writing |
| English localization | Language variants (US/UK), idiomatic phrasing, punctuation and sentence conventions | A translation is “grammatically correct” yet carries an obvious translationese flavor |
| Terminology consistency | Glossary adherence, consistency across long documents | The same term is translated in several ways across paragraphs |
| Long-form and complex instructions | Length control, structural consistency, constraint adherence | Output grows longer and longer and drops constraints stated in the instructions |
| Mathematical reasoning | Correctness, solving speed, verifiability | Mistaking fast responses for strong reasoning |
| Multimodal generation | Controllability, resolution and duration, consistency and compliance | Judging by sample results while ignoring batch consistency and copyright risk |
| Retrieval and real-time information | Recall capability, information timeliness, source traceability | Trusting the model’s own account without verification |
| Content alignment | False-rejection and false-pass rates, policy rigidity | Focusing only on “what gets blocked” and ignoring “what gets blocked by mistake” |
| Latency and throughput | First-token latency, end-to-end latency, long-context behavior | Measuring with average latency and ignoring the P95 tail |
| Cost and terms of service | Cost per task, quotas and rate limits, data-retention and account rules | Ignoring the real spend once the free quota runs out |
The dimensions break down as follows.
Language and writing: For Chinese writing, check whether the wording is idiomatic, whether the style is consistent, and whether the tone fits the scenario. For English writing, distinguish language variants and watch for Chinglish. You can inspect whether prepositions, articles, tenses, and redundant modifiers match the idiomatic usage of the target language.
Terminology and long-form: Technical documentation, legal contracts, and medical materials demand extremely high terminology consistency. When evaluating, prepare test documents rich in high-frequency terms, and check whether each term stays consistent on first appearance, on repetition, and when the context changes. Also test whether the model can keep its structure stable across output of several thousand words without expanding without limit.
Mathematical reasoning: A “speed advantage” in math and logic does not equal a “correctness advantage.” Some models run longer internal reasoning, so their responses take more time, yet they are steadier on multi-step derivations. When evaluating, record correctness and end-to-end time separately, and add programmatic verification for key calculations.
Multimodal generation: For image and video generation, beyond image quality, also consider style consistency during batch generation, responsiveness to parameters such as resolution and duration, and the copyright and compliance risks of the output assets.
Retrieval and real-time information: For time-sensitive content such as news, market conditions, and code-repository changes, use retrieval-augmented generation (RAG) or a service with live search capability, and require traceable sources.
Content alignment: An alignment policy that is too rigid will wrongly block normal translation, popular science, and even medical writing, interrupting the workflow. A policy that is too lax brings compliance risk in public-facing scenarios. Evaluation should count both false-rejection and false-pass rates, and configure them per scenario.
3. Building a Reusable Evaluation Workflow
Evaluation should avoid gut-feeling decisions and needs a repeatable, quantifiable process.
3.1 Build a Business Task Set
- Real sources: Sample from historical tickets and real documents rather than copying public problems.
- Layered design: Use simple tasks for calibration, medium tasks to distinguish models, and hard tasks to expose weaknesses.
- Clear standards: Each task needs a reference translation, a reference answer, or rules that can be checked automatically, such as whether terms hit, whether the format is compliant, and whether values are correct.
3.2 Use a Unified Invocation Gateway
First add a thin wrapper to abstract away the differences between model interfaces, so that evaluation, routing, and live calls all share one abstraction:
from dataclasses import dataclassfrom abc import ABC, abstractmethod@dataclassclass ModelResult:text: strlatency_ms: intinput_tokens: intoutput_tokens: intfinish_reason: str # e.g. "stop" (normal end), "length" (output truncated)class ChatModel(ABC):"""The evaluation layer and routing layer depend only on this abstraction, which makes horizontal comparison and on-the-fly replacement easy."""@abstractmethoddef chat(self, messages: list[dict], temperature: float = 0.2) -> ModelResult:...
3.3 Define the First-Pass Metric
In real production, what actually drives efficiency is the number of reworks, not the quality of a single generation. Define clear acceptance rules for each task, and track the share that passes on the first try:
import redef check_translation(text: str, glossary: dict[str, str], max_len: int) -> list[str]:"""Return a list of issues; an empty list means pass. Rules can be extended per business."""issues = []for zh, en in glossary.items(): # terminology consistency checkif zh in text and en not in text:issues.append(f"Term not translated per glossary: {zh}")if re.search(r"[—–]|\s--\s", text): # em/en dash misuse checkissues.append("Contains em/en dash, needs rewriting")if len(text) > max_len:issues.append("Exceeds length limit")return issues
3.4 Blind Evaluation and Cost Accounting
Shuffle the outputs from different models at random and hide their sources, then have several reviewers score them on fixed dimensions. This reduces the bias that comes from recognizing brands up front. At the same time, record the number of calls and the token consumption needed to complete each task, and compute the “cost per task.” If the pricing basis is the price per million output tokens, the numerator of cost per task is the unit price, and the denominator is the average number of tasks that pass acceptance contributed per output token:
cost per task = price per output token / (average number of tasks that pass acceptance contributed per output token)
For example (the numbers are for demonstration only and do not represent any real product): among the same 1,000 tasks, Model A has a first-pass rate of 85% and Model B 65%, and the rest need rework. Model A’s single-output unit price is 1.5 times Model B’s, yet its cost per task is still about 60% of Model B’s. This is where “seemingly expensive, actually cheaper” comes from.
3.5 Continuous Regression Testing
- Version the eval set: Put the task set under version control, and re-run the full set whenever a prompt or parameter changes.
- Lock model versions: Explicitly pin model versions in production instead of using “latest.”
- Gate before release: Once a key metric drops below baseline (for example, the first-pass rate falls by 5 percentage points), block a direct launch.
4. Capability Routing: Matching Each Task to the Right Model
The point of evaluation is not to crown a “winner” but to form a capability map and design routing rules from it:
# Example routing table: the key is the task type, the value is the primary/fallback model and its parameters.# Model identifiers must be configured from your own tests and should not be copied.ROUTE_TABLE = {"translation_draft": {"primary": "general_v1", "fallback": "general_v2"},"translation_review": {"primary": "reasoner_v1", "fallback": "general_v1"},"math_reasoning": {"primary": "reasoner_v1", "fallback": "reasoner_v2"},"qa_with_search": {"primary": "search_v1", "fallback": "general_v2"},"creative_writing": {"primary": "general_v1", "fallback": "general_v2"},}def route(task_type: str, text: str) -> dict:conf = ROUTE_TABLE.get(task_type, ROUTE_TABLE["creative_writing"])primary = conf["primary"]# Lightweight routing: short input with no reasoning needed, send to a low-cost modelif len(text) < 500 and task_type in {"summary", "classification"}:return {"model": "lite_v1", "temperature": 0.2}# High-difficulty routing: for long documents or math expressions, prefer a model with stronger reasoningif len(text) > 8000 or re.search(r"\\frac|\\sqrt|=", text):return {"model": conf["fallback"], "temperature": 0.1}return {"model": primary, "temperature": 0.3}
Four principles sit at the core of routing:
- Layered usage: Send high-frequency, simple, high-error-tolerance tasks to lightweight or free-tier models, and escalate hard tasks to higher-capability tiers.
- Allocate by capability, not price: For translation, first have a generative model produce the draft, then let the review stage step in.
- Prioritize retrieval for source-sensitive tasks: When real-time information is involved, prefer options with stronger retrieval capability and higher source quality.
- Keep a unified fallback path: When any model is unavailable, the system should switch to a backup model automatically and trigger an alert.
It is worth stressing that hot standby does not equal reliability. Backup models also need to be included in regression evaluation. If a backup differs too much from the primary in format or language style, output quality after switching can swing widely. Production pipelines should connect through server-side APIs, manage keys carefully, and audit calls, rather than depending on client products aimed at individual users.
5. Multi-Model Orchestration: A Technical Document Translation Example
Many teams have had this experience: when a model produces an English first draft, it reads smoothly, but a closer look reveals clear Chinglish. Switch to another model and translate again, and the phrasing becomes more idiomatic while technical terms start going wrong. This is usually not a “flaw” of a single model, but a reflection that two types of models have different emphases:
- Some models have a good feel for language and read fluently, but are weaker at grasping terminology and keeping details stable.
- Some models reason rigorously about concepts and are careful, but write less concisely.
The steadier engineering choice is not to pursue “one model to do it all,” but to split translation into three stages, generate-review-verify, with explicit format constraints:
- Generation stage: Use a model that is fluent and fast to produce the first draft.
- Review stage: Have a model strong in logic and conceptual analysis check terminology, numbers, logical connectors, and idiomatic English item by item, and output a list of changes.
- Verification stage: Use a program to automatically check glossary hits, punctuation norms such as em/en dashes, and length limits. Anything that fails automatically enters the next review round.
[Role] You are an English technical document reviewer. Output only the revised body text, without explaining your changes.[Glossary] Replace strictly per the mappings below, and keep them consistent throughout:模型 = model (do not use variants other than "model(s)")推理 = inference (do not use "reasoning")权重 = weights[Language norms] Use American English; avoid Chinglish collocations (for example, change "improve the performance" to "enhance performance").[Format constraints]- Do not use em/en dashes (— or –); when an explanation or addition is needed, use a clause or parentheses instead;- Do not output em dashes, long dashes, or Chinese punctuation;- Preserve the original paragraph structure and heading hierarchy; do not add or remove paragraphs;- Keep all numbers, formulas, and code snippets unchanged.[Verification checklist] Confirm item by item: whether terminology is consistent / whether any em/en dash remains / whether the paragraph count matches the original.
Prompt constraints should follow a “verifiable and quantifiable” principle. Replace vague requests like “make it more idiomatic” with rules a program can check, such as “use American English consistently throughout,” “do not use em/en dashes,” and “keep all numbers and formulas.” No matter which model you use, restrict symbols and formats explicitly in the prompt, because em/en dash misuse has become a common flaw in generative text.
6. Bringing Non-Capability Factors into Selection Decisions
Model capability is only half of selection. The other half is engineering usability.
6.1 Latency and Throughput
Interactive products care about first-token latency, while batch jobs care about throughput. In long-context scenarios, the longer the input, the longer the wait for the first token, so control context length through segmented summarization, retrieval cropping, and similar techniques, and evaluate the experience with P95 and P99 rather than averages.
6.2 Stability and Rate Limiting
Some services throttle during peak hours, adjust quotas, or suffer from service volatility. Enterprise applications should do three things:
- Keep multiple models on hot standby.
- Use exponential backoff retries for retryable errors such as 429 and 503.
- Add circuit breaking and queuing to keep failures from spreading.
6.3 Content Alignment Strategy
- For public-facing products: prefer services with steady, traceable policies, and layer your own content moderation on top.
- For scenarios that need creative latitude: configure policies differently per business line, but always keep a human review step.
6.4 Data and Account Risks
Subscription-based individual products are usually bound by terms of service, which may allow data to be used for training, carry account-ban risks, and raise compliance issues for cross-region access. Enterprise scenarios should connect through proper APIs, sign data-processing agreements, desensitize sensitive data before sending it, and keep audit logs of calls.
6.5 Vendor Lock-in
Do not bind business logic directly to a vendor’s proprietary interface. By encapsulating everything behind a unified gateway and putting prompts, evaluation sets, and routing configuration under version control, you can migrate smoothly as models iterate.
import timeimport randomRETRYABLE = {429, 500, 502, 503, 504}def chat_with_fallback(models: list[ChatModel], messages: list[dict],max_retry: int = 2) -> ModelResult:"""Automatically switch to a backup model on failure; exponential backoff prevents retry storms."""last_err = Nonefor model in models:for attempt in range(max_retry + 1):try:return model.chat(messages)except Exception as e:last_err = eif getattr(e, "status_code", None) not in RETRYABLE:break # Do not retry on parameter errors, auth failures, etc.time.sleep(2 ** attempt + random.random())raise RuntimeError(f"All model calls failed: {last_err}")
7. Conclusion: Three Core Principles of Selection
- Task-centric: First break the work down into task types and build a multi-dimensional evaluation matrix, instead of hunting for the “strongest model.”
- Data-driven: Let a business-close task set and the first-pass rate do the talking, and treat cost per task rather than unit price as the cost metric.
- Engineering-assured: Capability routing, the generate-review-verify pipeline, a unified gateway, monitoring and alerting, and regression testing are all indispensable.
Major cloud platforms (for example, Baidu AI Cloud) typically offer general-purpose cloud products such as model services, object storage, message queues, log services, and monitoring and alerting. These can serve as the infrastructure for managing evaluation data, hosting the call chain, and ensuring stability.
Back to everyday use: many models can already cover high-frequency needs like writing, summarization, Q&A, and retrieval at free or lightweight tiers. For cost-sensitive teams, it is worth validating requirements with free or low-price tiers first, then deciding whether to bring in a higher-capability tier based on task complexity. After all, “good enough and stable” often serves long-term interests better than “strongest but expensive.”
