LLM Selection in Practice: Building a Multi-Dimensional Evaluation System and Multi-Model Collaboration Strategies

15 hr ago0 views

As the development of large language models enters deeper waters, no single model excels at every task. This article explores why public leaderboards are no longer sufficient and outlines a practical guide for building a custom multi-dimensional evaluation system. It covers defining evaluation dimensions, creating private test sets, designing scoring mechanisms, and establishing regression testing. Furthermore, it delves into multi-model collaboration architectures like routing and pipeline patterns, prompt engineering techniques to maximize model capabilities, and common pitfalls to avoid in engineering practices.

1. Why Public Leaderboards Are No Longer Enough

As the development of large language models (LLMs) enters deeper waters, a clear industry consensus has emerged: no single model can simultaneously be the best at all tasks. While public leaderboards provide a reference baseline, they come with three inherent limitations:

Data Contamination Risk: Leaderboard questions may have been included in the training corpus, leading to artificially inflated scores.
Scenario Mismatch: The general capabilities evaluated by leaderboards do not necessarily align with real-world, high-frequency business tasks such as long-document review, code refactoring, or domain-specific Q&A.
Version Lag: Models iterate at an extremely rapid pace, and leaderboard update cycles often fail to keep up.

Therefore, rather than asking “which model is the strongest,” a question with more engineering value is: On each specific dimension I care about, which model performs the best respectively, and how can I combine them to work together?

2. Four Steps to Build a Custom Evaluation System

1. Define Evaluation Dimensions

It is recommended to split the dimensions into two tiers:

General Intelligence Tier: Macro planning capabilities, multi-step logical reasoning, instruction-following stability, and anti-hallucination capabilities.
Vertical Capability Tier: Selected based on business needs, such as code generation and debugging, long-context understanding and summarization, structured output, domain-specific knowledge Q&A, and tool-calling/Agent task completion rates.

2. Build a Private Test Set

Sample from real business traffic, stratify by difficulty (easy/medium/hard), and remove publicly available samples to prevent leakage. It is recommended to prepare at least several dozen samples for each dimension to ensure statistical significance.

3. Design Scoring Mechanisms

Blind Testing First: Hide model names to prevent preconceived brand bias from affecting judgment.
Dual-Track Scoring: Use automated assertions for objective questions, and a combination of “LLM-as-a-judge + manual spot-checking” for subjective questions.
Record Costs: Simultaneously record latency and API costs. When capabilities are similar, these are often the deciding factors.

A minimal weighted scoring example:

EVAL_DIMS = {
    "reasoning": 0.30,      # Logical reasoning and macro planning
    "coding": 0.25,         # Code generation and debugging
    "long_context": 0.15,   # Long-context understanding
    "instruction": 0.15,    # Instruction following
    "latency_cost": 0.15,   # Latency and cost
}
def total_score(item_scores: dict) -> float:
    return round(sum(item_scores[d] * w for d, w in EVAL_DIMS.items()), 2)

4. Establish a Regression Mechanism

Model versions are updated continuously. You should consolidate the test set and scoring scripts as team assets. When a new version is released, rerun them with one click to generate a comparison report, rather than relying on subjective impressions.

3. Multi-Model Collaboration: Lego-Style Composite Architecture

The end goal of evaluation is not to “crown a single champion,” but to form a “capability -> best model” mapping table, and then build a collaborative architecture based on it. Two common patterns exist:

1. Routing Pattern: A lightweight classifier or rule engine is placed at the front to identify the task type and dispatch it to the corresponding best model.

User Request -> Intent Recognition -> Routing Decision -> ┌─ Reasoning Model (Complex Planning)
                                                          ├─ Coding Model (Encoding Tasks)
                                                          └─ Long-Context Model (Document Processing)

2. Pipeline Pattern: Break down complex tasks into three stages: “Planning - Execution - Review.” Assign planning to a model with strong macro-thinking, execution to a model with strong vertical capabilities, and review to another model for cross-validation, thereby reducing the systematic bias of any single model.

In terms of engineering implementation, a unified gateway is required to converge the interface differences of various models and centrally handle caching, rate limiting, and monitoring. Cloud platforms like Baidu AI Cloud typically provide unified access, scheduling, and observability capabilities within their LLM services, which can significantly reduce the maintenance costs of operating multiple models concurrently.

4. Prompt Engineering: Maximizing Model Capabilities

Even if you select the right model, prompt quality can still introduce significant variance. Here are two widely validated techniques:

1. Enforce Full-Processing Instructions. In long-context tasks, models might employ “retrieval-based sampling reading” to save computation, focusing only on parts of the material. In such cases, explicit mandatory instructions should be given:

You will receive multiple reference materials. Please follow these rules:
1. Read through each material completely; do not skim, sample, or only search for keywords.
2. After reading, first assess the relevance of each material to the current task.
3. Include all relevant information in your analysis and conclusions without omission.
4. Before outputting, self-check: Have all key points from the materials been covered?

2. Socratic Self-Questioning. Require the model to question and iterate on its own conclusions multiple times before providing an answer: Are the premises valid? Are there counterexamples? Which link in the logical chain is the weakest? This “self-review first, answer later” structure can significantly improve the answer quality for complex reasoning tasks.

5. Common Pitfalls and Engineering Advice

Leaderboard Obsession: Leading leaderboard scores do not equal leading performance in your specific scenario; always rely on your private test set.
One-Time Evaluation: Models iterate every few months; evaluation must be a continuous process.
Ignoring Costs: High-capability models are often more expensive and slower; routing simple tasks to lightweight models is key to cost control.
Neglecting Test Data Governance: The test set itself requires version management to prevent it from gradually becoming obsolete as the business evolves.

6. Conclusion

In the current stage of LLM competition, the focus is no longer just on “who has the highest intelligence ceiling,” but on “who can be evaluated more scientifically and combined more reasonably.” For developers and enterprises, establishing a custom evaluation system, adopting a Lego-style multi-model collaboration, and pairing them with refined prompt engineering is the deterministic path to landing AI capabilities stably.