LLM Selection and Evaluation: A Practical Guide to Building a Multi-Model Architecture

    Choosing which single LLM to pay for is the wrong question. This article reframes model selection as an engineering problem: how to run multiple models together in a way that is stable, cost-effective, and replaceable. It shows how to build a measurable evaluation methodology using your own task sets and a multi-dimensional metric system, how to design an aggregation gateway with unified access, intelligent routing, and fault-tolerant fallback, and how to understand the risk boundaries of reverse-engineered APIs and tool-calling protocols. It also explains why LLMs are weak at precise calculation and how to solve it by externalizing computation to deterministic components, and closes with a landing path built on general cloud capabilities. By the end, which specific model you pick becomes just one parameter in your routing configuration, adjustable as your business and evaluation results evolve.

    1. The Real Question: It’s Not Just “Which One to Pick”

    In real projects, developers often face the same dilemma: there are plenty of available models, but their strengths are unevenly distributed across writing and expression, code generation, multimodal understanding, long-context processing, tool calling, and reasoning. Some are robust at structured output and instruction following, some excel at creative text, some stand out on coding tasks, and some are better at retrieving and summarizing very long documents. So which one should you choose?

    If you pay for only one model, you run into another set of problems: a single channel’s call quota, rate-limiting policies, service fluctuations, and behavioral changes from version updates can all affect business continuity.

    So the question worth answering is not “which one should I pay for,” but rather: how can I make multiple models work together in my system in a stable, economical, and replaceable way? This article is organized around that thread:

    • Building a measurable selection and evaluation method;
    • Designing an aggregation gateway with unified access, intelligent routing, and fault-tolerant degradation;
    • Clarifying the risk boundaries of reverse-engineered interfaces and tool integration protocols;
    • Solving the engineering challenge of weak precise calculation in large models;
    • Laying out an implementation path based on general cloud capabilities.

    2. Evaluation First: Turning “Which Is Better” into a Measurable Problem

    2.1 Replace Generic Rankings with Your Own Task Set

    Public leaderboards have limited reference value. Their questions often differ greatly from your real business distribution, and different evaluation methods (such as answer-matching automatic scoring versus preference-comparison evaluation) do not necessarily agree. So the first step is to collect real, de-identified samples from your business and build your own evaluation set:

    • Scenario coverage: Stratify by task type, covering at least question answering, summarization, extraction, classification, rewriting, code generation, multi-turn dialogue, document understanding, and visual understanding, with a comparable number of samples per category.
    • Difficulty distribution: Include both regular samples and boundary/adversarial samples, such as long inputs, ambiguous instructions, implicit constraints, distractor information, and leading questions.
    • Reference answers: For tasks that can be scored automatically, provide reference answers or rules; for open-ended questions, provide scoring criteria for human or model-assisted review.
    • Continuous updates: Re-run scoring after version updates, and periodically add failure cases discovered in production.

    2.2 A Multi-Dimensional Metric System

    Dimension Metrics to Observe Notes
    Correctness Answer accuracy, match with reference answers, citation consistency For factual questions, focus on whether the model fabricates content
    Instruction following Format compliance rate, constraint satisfaction rate e.g., output specified as JSON, word-count limits, prohibited expressions
    Expression quality Readability, terminology accuracy, tone consistency For open-ended questions, recommend multi-person blind review
    Code & structured output Runnable rate, test pass rate, Schema validation pass rate Auto-determined via compile, execute, and validation scripts
    Stability Variance across repeated samplings of the same input, performance degradation under long context Examines how reproducible the output is
    Engineering metrics First-token latency, end-to-end P95 latency, cost per task Weighed together with quality and stability
    Security & compliance Blocking of unauthorized and harmful content, spot checks for privacy leakage Combined with content safety capabilities and manual spot checks

    2.3 Recommendations for Running Evaluations

    • Blind cross-review: Hide model identities and have different people score in a cross fashion to reduce bias from brand and ordering effects.
    • Combine automation with human review: Fully automate tasks that rules can score, and sample open-ended questions for manual review.
    • Control sampling parameters: Interface parameters do not fully correspond across models (for example, the valid range of sampling temperature), so unifying them is neither feasible nor necessary. Fix each model’s recommended parameter configuration and average over multiple samples to avoid a single coincidental result misleading your decision.
    • Integrate into the regression pipeline: Automatically re-run when models or prompt versions change, to prevent “fixing one problem and introducing another.”

    3. Aggregation Gateway: A Reference Architecture for Unified Multi-Model Access

    3.1 Design Goals

    The essence of an aggregation gateway is to treat models as replaceable infrastructure, so that business code depends only on a unified abstraction. Its core goals include:

    • Decoupling: The business layer is not directly bound to any specific model service;
    • Routing: Select models by capability, cost, latency, and data sensitivity;
    • Fault tolerance: Retries, circuit breaking, and degradation ensure availability;
    • Observability: The call chain, quality, and cost are fully traceable;
    • Security: Credentials are centrally managed, and ingress/egress are controlled uniformly.

    3.2 Layered Architecture

    • Access layer: Provides unified chat, completion, embedding, and similar interfaces to the business;
    • Routing layer: Parses task intent and selects target models and backup models by rule;
    • Adapter layer: Hides differences among model services in request fields, message formats, streaming protocols, and usage statistics, and exposes a unified request/response structure;
    • Governance layer: Handles authentication and quotas, rate limiting, circuit breaking and degradation, caching, and logging and metric reporting;
    • Model layer: Connects to official open APIs, cloud-vendor-hosted model services, or privately deployed open-source models.

    3.3 A Sample Routing Strategy

    The following pseudocode illustrates a combination of “capability routing + cost preference + multi-candidate fault tolerance”:

    1. # Sample model registry (capacity is a capability profile obtained from internal evaluation)
    2. # profile is the business-configured preference strategy, defaulting to balanced
    3. REGISTRY = {
    4. "chat_general": {
    5. "provider": "...", "model_id": "...",
    6. "capacity": ["general QA", "long text", "function calling", "visual understanding"],
    7. "base_cost": "middle", "latency": "middle",
    8. },
    9. "reasoner": {
    10. "provider": "...", "model_id": "...",
    11. "capacity": ["math reasoning", "complex logic", "code"],
    12. "base_cost": "high", "latency": "high",
    13. },
    14. }
    15. PREFERENCE = {
    16. "quality_first": ["capacity_match", "quality_score"],
    17. "cost_first": ["base_cost", "capacity_match", "quality_score"],
    18. "speed_first": ["latency", "base_cost", "capacity_match"],
    19. }
    20. def select_candidates(task, profile="balanced"):
    21. # 1) Coarse filter: must have the capabilities the task requires
    22. candidates = [m for m in REGISTRY.values()
    23. if set(task.required_capabilities) <= set(m["capacity"])]
    24. # 2) Sort: compare by strategy in order, adding the evaluation score as a discriminator
    25. ordered = sorted(candidates,
    26. key=lambda m: tuple(rank(m, key) for key in PREFERENCE[profile])
    27. + [-quality_score(m, task.type)])
    28. # 3) Configure primary, backup, and fallback candidates in a 5:3:2 split
    29. head = max(1, int(len(ordered) * 0.5))
    30. second = max(1, int(len(ordered) * 0.3))
    31. return ordered[:head], ordered[head:head + second], ordered[-1:]

    This works together with a “primary model — backup model — fallback model” invocation strategy: when the primary model fails, switch to the backup; when both fail, use the fallback, ensuring the core path stays available.

    3.4 Fault Tolerance and Degradation

    1. Error classification for retries: Only retry recoverable errors such as rate limiting, timeouts, and network jitter, using exponential backoff with added random jitter. For non-recoverable errors like parameter errors or content violations, return immediately to avoid invalid retries inflating cost and latency.
    2. Circuit breaking and degradation: When failures accumulate, temporarily pause a model channel. The degradation chain can be configured in an order of “similar capability, lower cost,” and degradation events should be logged for retrospectives.
    3. Tiered timeouts: Distinguish connection timeouts from read timeouts. In streaming scenarios, set a “first-token timeout” so users do not stare at a blank screen for too long.
    4. A fallback that responds: The fallback message should clearly tell the user that the service is temporarily unavailable and how it will be handled, rather than fabricating an answer.

    3.5 Cost Control

    • Usage attribution: For each call, record the caller, business scenario, input/output token counts, the chosen model, and the cost, aggregated by tenant or business line.
    • Budgets and quotas: Set daily/monthly call limits for different businesses, with alerts as they approach the threshold, rather than discovering anomalies only after the bill arrives.
    • Caching strategy: Exact-match requests can use short-term caching; semantic caching must be used cautiously and disabled for sensitive information such as amounts, dates, and IDs.
    • Context optimization: During retrieval augmentation, inject only relevant snippets; for multi-turn dialogue, compress historical summaries to reduce wasted token consumption.

    3.6 Observability and Governance

    • Metric system: QPS, P95 latency, error rate, degradation rate, cache hit rate, and token and cost trends;
    • Structured logs: Record request IDs, model identifiers, duration, usage, and result summaries, with sensitive fields de-identified;
    • Tracing: Chain the gateway, retrieval, tool calls, and model calls into a single trace, making it easy to locate “where it’s slow and where it’s wrong”;
    • Version management: Prompt templates, routing rules, and model versions are all placed under version control, so changes are traceable and rollback-able.

    4. Reverse-Engineered APIs and Tool Integration Protocols: Defining the Risk Boundary

    4.1 Typical Risks of Reverse-Engineered Interfaces

    Connecting to model services via reverse engineering or unofficial channels may lower costs or bypass restrictions in the short term, but the risks are significant in production:

    • Poor stability: A slight server-side change can break the integration, requiring continuous reverse-engineering maintenance;
    • Policy risk: Unexpected calls easily trigger risk-control mechanisms and may even get an account restricted;
    • Security and trust risk: When credentials pass through an uncontrolled channel, they may be intercepted or recorded; and content returned by the server cannot be treated as trusted data, since it may carry injection content that lures the model into performing unauthorized actions;
    • Compliance risk: It may violate the terms of service, and in enterprise settings it also raises audit and liability questions.

    The more prudent engineering approach is to prefer official open APIs or cloud-vendor-hosted model services. When a third-party service is genuinely needed, connect through a compliant aggregation channel and perform security and compliance assessment.

    4.2 Why Tool Integration Protocols Demand Extra Caution Here

    Tool integration protocols (in common implementations, the descriptions of callable tools are handed to the model, which then outputs structured tool-call requests) are themselves a valuable capability extension. They let models call external tools such as calculators, databases, search engines, and internal business APIs, compensating for their own weaknesses in precise calculation and real-time information.

    The problem is that tool calling means model output can directly trigger real system operations. If the upstream channel itself is untrusted, the attack surface expands from “generating a piece of erroneous text” to “executing an unauthorized operation.” Typical risks include:

    • Credential exposure: Reverse-engineered channels often require holding account credentials on the client; once leaked, an attacker can call tools directly;
    • Prompt injection: Tool return content such as external documents, web pages, and emails may hide instructions that lure the model into performing unintended operations;
    • Unauthorized actions and misoperations: When the model misunderstands intent, it may execute irreversible operations such as deletion, transfers, or sending messages;
    • Missing audit trail: When calls do not go through a unified gateway, operation logs are incomplete and hard to trace afterward.

    4.3 A Security Baseline for Tool Integration

    1. Trusted channel: Only connect to model services through official or compliant channels.
    2. Least privilege: Grant each tool only the minimum permissions needed to complete the task, with write operations requiring user confirmation by default.
    3. Sandbox isolation: Run code execution, file operations, and network access in an isolated environment, with resource and time limits.
    4. Input/output validation: Validate tool arguments against a Schema, and treat tool return results as untrusted data rather than executing them as new system instructions.
    5. Credential management: Store keys centrally in a key management service; never hard-code them or write them into logs.
    6. Full auditing: Completely retain the initiator, arguments, results, and approval records of each tool call.

    5. Weak Precise Calculation: Root-Cause Analysis and an Externalization Approach

    5.1 Why Calculations Go Wrong

    A large model’s “calculation” is essentially probabilistic prediction of numeric patterns in text, not mathematical computation. There are three main reasons:

    1. No explicit arithmetic unit: There is no computing unit equivalent to a CPU or calculator inside the model’s parameters;
    2. Tokenization causes error accumulation: Multi-digit numbers are often split into several subword fragments, so carries and borrows across fragments are error-prone, and errors grow with more steps;
    3. Information dilution in long context: In long text, attention to key numbers is easily scattered, causing incorrect citations or omissions.

    This is not a defect unique to any one class of models, but a shared characteristic of this technical approach. The most reliable idea is not to expect the model to “calculate more accurately,” but to hand computation to deterministic components.

    5.2 The Externalization Approach

    • Function calling (tool calling): The model is only responsible for understanding intent, extracting parameters, and organizing the call, while a deterministic function performs the computation and returns the result;
    • Code execution: Have the model generate code, execute it in a sandbox, and read the result; suitable for multi-step calculations and data processing;
    • Retrieval augmentation: For factual and numeric questions, retrieve from authoritative data sources first rather than relying on model memory;
    • Result validation: Perform range, magnitude, and consistency checks on key values, and where necessary introduce a rule engine or cross-validate against a second source.

    A sample flow for calculation tasks:

    1. User: What's the tax-inclusive total for this batch of orders? Please calculate it.
    2. Step 1 Model analyzes and outputs a tool call (only responsible for understanding and planning):
    3. { "tool": "calc",
    4. "arguments": { "expression": "123.45 * 6 * 1.13" } }
    5. Step 2 Executor evaluates deterministically:
    6. Use decimal high-precision arithmetic (not binary floating point) with unified rounding rules
    7. 123.45 * 6 = 740.70
    8. 740.70 * 1.13 = 836.991
    9. Rounded to the nearest "cent" = 836.99
    10. Step 3 Model composes a natural-language reply and includes the calculation process:
    11. The tax-inclusive total is 836.99 (excluding tax 740.70, tax rate 13%).

    A sample implementation of a high-precision calculator (note: never directly execute arbitrary code coming from the model):

    1. from decimal import Decimal, ROUND_HALF_UP, localcontext
    2. def calculate(expression: str, precision: int = 28) -> Decimal:
    3. """
    4. Evaluation is done by a restricted expression parser (e.g., AST-based parsing
    5. that allows only arithmetic operations).
    6. Using eval or exec to directly run model-generated code is forbidden.
    7. """
    8. with localcontext() as ctx:
    9. ctx.prec = precision
    10. value = _safe_eval_decimal(expression) # placeholder: a constrained evaluation implementation
    11. return value
    12. def to_currency(value: Decimal) -> Decimal:
    13. # Round to the nearest "cent" to avoid a mismatch between display and actual billing
    14. return value.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

    5.3 Engineering Recommendations

    1. A calculation regression set: Assemble cases covering multi-digit multiplication and division, consecutive addition and subtraction, percentages, date and duration conversions, unit conversions, and currency and quantity totals, to serve as a regression gate when models or prompts change.
    2. Clear rounding and precision rules: For scenarios such as amounts, taxes, and interest rates, unify precision and rounding methods, and write them into both the prompt and validation logic.
    3. Verifiable output: Attach the formula and intermediate results to numeric answers, so users and systems can check them.

    6. An Implementation Path Based on General Cloud Capabilities

    Model capabilities are obtained through API calls, but a production-grade multi-model system also needs support from several general cloud capabilities, which can be combined flexibly by scenario:

    • Managed model services: Connect through a cloud vendor’s managed model service, avoiding the burden of building your own inference cluster and maintaining compute;
    • Container platform: Deploy the gateway and routing services, scaling elastically with traffic;
    • Object storage: Store evaluation datasets, call-log archives, and cold data;
    • Message queue: Handle asynchronous tasks such as batch evaluation and offline report generation, smoothing traffic peaks;
    • Logging, monitoring, and alerting: Provide call search, a cost-accounting dashboard, and threshold alerts for error rate, latency, and cost;
    • Content safety capabilities: Perform compliance checks on inputs and outputs;
    • Key management: Centrally manage and periodically rotate model service credentials.

    For example, on Baidu AI Cloud you can combine cloud products of the categories above: use managed model services for model inference, a container platform to deploy the aggregation gateway, a message queue to handle offline evaluation tasks, and logging services plus monitoring and alerting to ensure observability. Specific product types, activation methods, and configuration parameters are subject to the official documentation and actual circumstances.

    7. Conclusion: From “Which Model to Choose” to “How to Design the Model Layer”

    Back to the original question: instead of agonizing over which one to pay for, it’s better to invest your energy in designing the model access layer. This can be summarized in five practical principles:

    1. Evaluation first: Replace vague impressions of rankings with your own task set and multi-dimensional metrics;
    2. Unified access: Use an adapter layer to hide differences, making models pluggable and replaceable;
    3. Intelligent routing: Dynamically choose by capability, cost, and latency, with multiple candidates to ensure availability;
    4. Security boundaries: Reject reverse-engineered channels, and follow least privilege and full auditing for tool integration;
    5. Externalized calculation: Hand precise calculation to deterministic components, letting the model focus on understanding, planning, and expression.

    This way, which specific model you choose becomes just one parameter in your routing configuration, adjustable at any time as your business stage and evaluation results change. If you want to validate results quickly, you can start with offline evaluation and a lightweight prototype, then progressively build out the gateway’s capabilities to ultimately form a sustainable, evolving multi-model collaboration system.