Evaluating LLMs Beyond Leaderboards: A Dimension-by-Dimension Framework and Practical Testing Methodology

1. Why “Which Model Is Strongest” Is the Wrong Question

The gaps between mainstream LLM assistants on public leaderboards keep shrinking, yet real-world experience still diverges noticeably. There are three main reasons:

1. Benchmark workloads differ from real workloads. Most benchmark questions have a single correct answer, whereas everyday tasks—long-form rewriting, code refactoring, technical translation—are judged on a continuous spectrum rather than a binary right-or-wrong basis. Leaderboard rankings do not map directly to experience differences.
2. Capability distributions are typically skewed. Some models are fast and accurate on math and symbolic reasoning, and are also solid at multimodal generation; some stand out in code generation, sentiment understanding, and conceptual discussion; others may not be the strongest at any single point, but are rigorous on details and rarely make small mistakes, making them more balanced in practice.
3. Cost differences are enormous. Subscription tiers range from tens of dollars to over a thousand per month, and higher-priced tiers do not necessarily maintain a generational advantage across all tasks.

The reasonable question, therefore, is not “which model is strongest” but: for my task mix, which model (or combination of models) offers the best overall cost-performance and stability? Answering that requires a reproducible evaluation methodology.

2. Six Core Evaluation Dimensions

2.1 Math and Symbolic Reasoning

Prepare 10–20 questions of mixed difficulty (engineering calculations, logical reasoning, unit conversions), send the same question concurrently to candidate models, and record two metrics: accuracy and time to solution.

A common observation in practice: computation speed differs visibly between models—on the same question, some are clearly faster. But comparison must be conditioned on accuracy first: compare speed only within the set of correctly answered questions, otherwise a speed advantage can mask process errors. It is also advisable to sample and recompute the model’s intermediate steps to verify that its reasoning chain is explainable and reproducible, rather than judging only the final answer.

2.2 Code Generation and First-Pass Success Rate

Code capability should be tested in layers:

• L1: implement standalone small functions (string processing, date calculation, etc.);
• L2: locate and fix defects in a given multi-file codebase;
• L3: complete a small refactor based on vague requirements and add test cases.

The core metrics are first-pass success rate and number of rework rounds. A rule of thumb: mid-tier and flagship models differ little on L1; but when tasks get slightly more complex (L2/L3 or complex text processing), with the same prompt the flagship often completes in one shot while the mid-tier model requires multiple back-and-forth revisions. If you fold the time and tokens consumed by rework into the cost, “flagship is more expensive” can actually be cheaper on complex tasks. The principle is straightforward: leave simple tasks to mid-tier models, go straight to flagship for complex ones, and don’t waste time on rework.

2.3 Text Creation and Conceptual Discrimination

• Creative writing: have models continue a passage or rewrite copy, and observe tone control, emotional nuance, and structural discipline;
• Conceptual discrimination: prepare pairs of similar concepts (concurrency vs. parallelism, latency vs. throughput, authentication vs. authorization) and require the model to distinguish them precisely with examples.

Notably, some models produce well-formed language but occasionally err on conceptual discrimination questions and are not concise enough in their output; such weaknesses do not appear in official marketing and can only be exposed through your own test sets.

2.4 Translation Quality and Terminology Accuracy

Translation is the most efficient test for exposing a model’s language weaknesses. Two common patterns: one is fluent output that carries traces of source-language expression, with terminology details frequently getting it wrong and insufficient target-language convention; the other is conventionally correct target language but verbose expression and occasional conceptual drift. A directly reusable prompt template:

You are a senior technical translator. Translate the following Chinese passage into English, observing these requirements:
1. Strictly follow the glossary: {terms};
2. Do not use dashes for parenthetical insertions; rewrite the sentence when supplementary information is needed;
3. Preserve the original paragraph structure, lists, and code blocks;
4. After translating, add a separate section listing uncertain terms and alternative translations.

Requirement 2 is no joke: some models have a pronounced tendency to overuse dashes, and constraining this directly in the prompt is better than cleaning it up manually afterward.

2.5 Multimodal Capabilities

If your workflow involves image or video generation, evaluate three elements separately: generation quality, instruction adherence (whether details in the prompt are accurately rendered), and coupling with text capability—whether a “text-to-image + image-to-text” loop can be completed within the same session. Also note: some models with strong text capabilities do not support image generation at all, so the capability matrix should mark this explicitly to avoid mistaking “strong at text” for “all-around.”

2.6 Stability, Detail Rigor, and Service Risk

• Detail rigor: error rates on numbers, units, punctuation, and citation formats. Some models may not produce the most impressive conclusions, but they rarely err on small details—crucial for engineering documents and externally facing materials.
• Token consumption speed: for the same conversation volume, some models “burn through quota fast,” so a nominally cheaper plan may offer less usable capacity in practice.
• Service risk: rate-limiting policies, account suspension rules, regional availability, and data compliance. These factors do not appear on capability leaderboards but directly determine whether a service can be used stably long-term; treat them as deal-breakers in evaluation.

3. Making Evaluation Reproducible: Comparison Experiment Design

Single impressions are unreliable; turn evaluation into a reproducible experiment:

1. Fixed test set: version-control 20–50 prompts covering the dimensions above, and run regression tests after model upgrades;
2. Concurrent same-question comparison: send the same prompt with similar sampling parameters to candidate models simultaneously to eliminate session-state interference;
3. First-pass success statistics: use “meets the bar without human rework” as the pass criterion, not “the result looks decent”;
4. Cross-review: have another model act as judge and score the outputs, then manually spot-check to calibrate judge bias.

Evaluation record template:

Dimension	Test Items	First-Pass Passes	Avg. Time	Rework Rounds	Notes
Math reasoning	10
Code generation	5
Translation	5
Conceptual discrimination	5

4. Combined Workflows: Compensating for Weaknesses with Strengths

Since individual models each have shortcomings, a mature approach is to split tasks by stage and route each to the model that handles it best. Taking English translation of technical documentation as an example, a two-stage pipeline is recommended:

Draft translation ──► Completed by a model with strong target-language convention and stable terminology
     │
     ▼
Polish and review ──► A model with stronger language sense and conceptual grasp revises expression and terminology
     │
     ▼
Mechanical check ──► A script performs a final consistency check against the glossary

Illustrative script for terminology consistency checking:

GLOSSARY = {
    "熔断": "circuit breaking",
    "幂等": "idempotent",
}
def check_translation(text_en):
    issues = []
    for zh, en in GLOSSARY.items():
        if match(zh, text_en):             # untranslated Chinese term残留
            issues.append("Term not translated: " + zh)
        elif not match_word(en, text_en):   # agreed translation not found
            issues.append("Translation needs review: " + zh + " should be " + en)
    return issues

Supporting recommendations:

• Front-load constraints: put all formatting requirements (no dashes, preserve code blocks, glossary) into the prompt rather than fixing afterward;
• Triage routing: simple tasks such as one-sentence rewrites and headline generation go to mid-tier models; complex tasks use flagship throughout to avoid burning time on rework;
• Build a template library: turn validated prompt templates into reusable assets, archive them by task type, and share them across the team.

5. Cost Accounting: From “Monthly Fee” to “Cost per Task”

The monthly fee itself is not a decision basis; convert it into two metrics:

1. Cost per task = subscription fee ÷ number of qualified completed tasks. Models with high rework rates see unit costs rise significantly;
2. Quota consumption speed: for the same conversation volume, token consumption varies greatly between models—a lower nominal price does not mean a lower actual cost.

Practical advice: first record one week of real task distribution (how much coding, translation, and writing each account for), then choose a tier accordingly; also pay attention to how clear the plan rules are—the more complex the naming and entitlement design, the more likely expectations will diverge at renewal and upgrade time. Enterprise users can also consider pay-as-you-go cloud model services, such as the model inference capabilities offered by Baidu AI Cloud, which tie cost directly to actual call volume and make it easier to allocate budgets by project; for sensitive code and data, choosing a compliant and trustworthy service provider is itself part of risk control.

6. Conclusions and Action Checklist

1. There is no all-around champion: fast reasoning, strong coding, nuanced text, and rigorous details are often spread across different models;
2. Evaluation must be task-aligned: clarify your task distribution first, then use a fixed test set for same-question comparison and first-pass success statistics;
3. Combination beats single choice: a two-stage draft-plus-polish pipeline, front-loaded prompt constraints, and triage routing can assemble each model’s strengths into a stable workflow;
4. Judge cost per task: the monthly fee is only surface-level; token consumption speed and rework rate are the real costs;
5. Evaluation is continuous: models iterate quickly, so run a regression with the fixed test set every quarter and dynamically adjust your primary and specialized models.

After one full round with this methodology, you will have a capability matrix you generated yourself, aligned with your real workload—one that is more worthy of being your selection basis than any leaderboard.