How to Objectively Evaluate LLMs: Core Dimensions and Methodologies

I. Why Model Evaluation is Inherently a Technical Challenge

Whenever a new flagship Large Language Model (LLM) is released, public opinion quickly polarizes into hype versus skepticism. However, what truly matters for engineering decisions is a layered, reproducible evaluation framework. There are three main reasons for this:

1. Leaderboard scores contain noise and selection bias. The task distribution across different evaluation sets varies significantly. A high score on a specialized leaderboard (e.g., purely covering code generation) cannot be extrapolated to general capabilities. Furthermore, public evaluation sets may have been ingested into the model’s training data, resulting in inflated scores due to “data contamination.”
2. “Capability” is multi-dimensional. A model might excel at long-context retrieval but perform averagely at complex reasoning; it might be great at code generation but unstable when making Agentic tool calls. Only by breaking down these dimensions can you precisely match a model to your business requirements.
3. Cost and deployment constraints are first-class citizens. A cheaper model with equivalent quality, or a higher-quality model at the same price point, often provides more practical value in real-world projects than the “absolute strongest” model.

Therefore, we will explore evaluation across five layers: Architecture → Capabilities → Engineering → Cost → Openness.

II. Architecture Layer: Parameter Scale and MoE Nuances

2.1 Total Parameters vs. Activated Parameters

Current flagship models widely adopt the Mixture of Experts (MoE) architecture. Taking a typical industry configuration as an example: the total parameter count can reach the trillion level, with hundreds to over a thousand experts. However, for each token, the gating network routes to only a dozen or so experts. The activated parameter ratio is usually in the single-digit percentage range. This leads to two direct conclusions:

• Training side: Total parameters determine the upper limit of knowledge capacity. “Bigger” still means a higher potential ceiling.
• Inference side: Activated parameters determine the computation required per token. Sparse activation allows a trillion-parameter model to achieve inference costs close to a dense model with tens of billions of parameters.

The key points during evaluation are the activation ratio and routing quality: overly sparse activation might harm knowledge sharing among experts, while unstable routing can cause load imbalance. These details are rarely mentioned in press releases but can be indirectly perceived through actual inference throughput and stability during long-running tasks.

2.2 Evolution of Attention Mechanisms

Standard Softmax attention has quadratic complexity regarding sequence length, which is the primary cost driver for long contexts. Recent common industry practices include:

• Hybrid Architecture: Retaining global attention only in a few key layers while replacing others with efficient variants like linear attention or sliding windows, striking a balance between capability and cost.
• Residual and Normalization Design: In deep networks with dozens of layers, whether information “flows smoothly” heavily depends on the details of residual pathways and normalization positions. Such design choices often determine whether a model suffers from “reading the end but forgetting the beginning” in ultra-long contexts.

The takeaway for users is: when vendors announce a “new attention mechanism,” focus on what problem it actually solves (e.g., maintaining long-range dependencies, compressing KV cache, or improving inference speed) rather than the buzzword itself.

2.3 Scaling Efficiency is a Hidden Metric

Press releases often claim “X times efficiency improvement over the previous generation.” You must question the metrics used: Is it the rate of training loss reduction under equal compute? The benchmark score under an equal inference budget? Or the serving cost per token? “Efficiency multiples” only hold engineering significance when the metrics are clearly defined. A simple, viable validation method is to run the same set of private tasks on both the old and new generations of models, simultaneously comparing scores and token consumption.

III. Context Length: Nominal Window ≠ Effective Window

Context windows at the million-token level have become standard for flagship models. Its engineering significance includes:

• Whole-Repository Code Understanding: Medium-to-large projects easily contain hundreds of thousands of lines. Long windows make “putting the entire repository into context” possible, allowing cross-file refactoring and dependency analysis without relying on fragile retrieval stitching.
• Long Documents and Ultra-Long Conversations: Hundreds of pages of documents or hours of meeting transcripts can be processed at once. Multi-turn Agent tasks no longer require frequent history compression.

However, beware of two common traps:

1. Effective Context Decay. A nominal 1-million token window does not guarantee solid retrieval and reasoning capabilities at the 900,000th token. Focus on “over what range the recall rate and reasoning quality remain stable,” rather than the advertised number itself.
2. Cost and Latency of Long Contexts. The longer the input, the higher the time-to-first-token latency and the overall cost. This is exactly why context caching mechanisms (see Section VII) exist.

IV. Multimodality: Distinguishing “Seeing” from “Understanding”

New-generation flagships natively support image and even video inputs (outputs remain text). When evaluating multimodal capabilities, it is recommended to distinguish three levels:

• Basic Perception: Can it accurately read text, tables, and coordinates within an image? This determines the lower bound for “screenshot feeding” tasks.
• Structural Understanding: Can it comprehend the layout intent of a UI screenshot, the data relationships in a chart, or the hierarchical structure of a design mockup? This determines the upper bound for front-end restoration and data analysis tasks.
• Temporal Information: For videos, can it track events and state changes across frames, rather than just describing them frame-by-frame in isolation?

It must be emphasized that native multimodality (where visual signals are jointly modeled with text during the pre-training phase) generally outperforms patched-together solutions like “external visual encoder + post-hoc alignment,” especially for tasks requiring fine-grained spatial understanding, such as “writing code from an image.” The most valuable engineering use case is the visual feedback loop: the model generates a page, takes a screenshot of the rendered result, feeds it back to itself for inspection, and autonomously iterates until visual consistency is achieved. This loop capability is more worth validating in evaluations than “one-shot generation.”

V. Reading Benchmarks Correctly: Cross-Validating Three Information Sources

Post-release information typically comes from three channels, each with its own biases, and should be read cross-referentially:

1. Official Self-Evaluation: Broad task coverage and rich details, but inherently carries selection bias—often only reporting advantageous items, with baselines cherry-picked by the vendor.
2. Independent Lab Comprehensive Indices: Aggregated by third parties using unified metrics across reasoning, code, knowledge, math, etc. Suitable for horizontal comparison, but releases lag, and the weighting might not match your business needs.
3. Blind Test Arenas: Real users vote anonymously between two models, ranked using Elo-style scoring. The advantage is immunity to targeted “benchmark gaming,” reflecting human-perceivable quality (writing style, aesthetics, direct usability). The downside is a preference bias toward what “looks good,” offering limited reference value for scenarios demanding strict correctness.

For the typical pattern of “outstanding on specialized leaderboards but slightly behind on comprehensive indices,” the correct interpretation is: Specialized capabilities might trade blows with the first tier, but comprehensive capabilities are still half a step behind. When selecting a model, first confirm which specialized category your core tasks fall into, rather than just looking at the overall ranking.

Beyond public leaderboards, it is crucial to build a private evaluation set. A minimal viable framework looks like this:

# Minimal framework for custom evaluation
tasks = load_tasks("private_eval_suite.jsonl")  # Each entry contains instruction, context, and rubric
def run_eval(model_id: str) -> dict:
    scores = {}
    for task in tasks:
        output = call_model(model_id, task.prompt, max_tokens=task.budget)
        scores[task.id] = rubric_score(output, task.rubric)  # Rule scoring + manual spot-check
    return aggregate(scores)  # Weighted summary by task type

The evaluation set doesn’t need to be massive: dozens of questions covering real business patterns (e.g., “modify a page based on a screenshot,” “fix a bug across three files,” “extract structured fields from a long document”) will predict production performance far better than any public leaderboard.

VI. Engineering and Agent Capabilities: Long-Horizon Tasks are the Touchstone

Measuring whether a model can be “used as an employee” depends not on single-turn Q&A, but on stability over long-horizon tasks:

• Toolchain Operation: Can it continuously use the terminal, edit files, and run tests? Can it read logs, change strategies, and retry upon failure? This tests error recovery and context management, not single-point intelligence.
• Cross-File Consistency: Do modifications consider related callers, type definitions, and tests? Or does it “fix one thing and break three others”?
• Autonomous Verification: Does it proactively run tests or open a page to self-check, rather than pushing all verification responsibilities onto humans?

The most effective way to evaluate these capabilities is to assign “verifiable” tasks: code repositories where tests can be run, pages with reference designs, or extraction tasks with standard answers. Score them based on objective results, not subjective impressions.

VII. Cost Model: Understanding Per-Token Pricing and Caching

Mainstream APIs price input and output separately per million tokens. Furthermore, the input side generally introduces context caching: when the same prefix (system prompts, long documents, chat history) appears repeatedly, the cached portion is billed at a significantly lower rate, while non-cached portions are billed normally. The cost of a single request can be approximated as:

Cost ≈ (Uncached Input Tokens × Uncached Unit Price
      + Cached Input Tokens × Cache Unit Price
      + Output Tokens × Output Unit Price) / 1,000,000

This yields three general conclusions:

1. Put stable content first. Immutable content like system prompts and knowledge base summaries should be placed at the very beginning of the context to maximize cache hits.
2. Cache-friendliness should be part of architectural design. In scenarios like multi-turn conversations, RAG, or Agent loops, caching can reduce actual costs by an order of magnitude.
3. Compare costs using “unit cost per quality.” A model with a 30% lower unit price but requiring more retries and manual corrections might actually result in a higher true cost.

VIII. Openness: Weights, Licensing, and Hardware Thresholds

Fully open weights provide engineering teams with threefold value: Private deployment (data stays in-house), Domain fine-tuning (strengthening specialized capabilities with proprietary data), and Supply chain autonomy (not relying on the availability of a single service). When assessing openness strategies, consider:

• License terms: Is commercial use allowed? Are there user scale limits?
• Ecosystem completeness: Are inference code, recommended deployment configurations, and quantized versions provided?
• Hardware thresholds: A trillion-parameter MoE, even with sparse activation, still requires a considerable cluster for weight storage and multi-node inference. “Open weights” does not equal “runs on a personal GPU.”
• Delivery certainty: Promised open timelines and final delivery scopes may change. Critical decisions should not be bet on undelivered capabilities.

IX. General Practices for Model Selection and Integration

Flagship models typically offer multiple endpoints: web and client interfaces are suitable for interactive validation and ideation, CLI formats fit into engineering pipelines, and open platform APIs are for product integration. General advice for integration:

• Pin versions in production: Always explicitly specify the model version number to prevent behavioral drift caused by silent server-side upgrades.
• Canary releases and rollbacks: Use shadow traffic to compare before launching a new model, and retain a one-click path to revert to the old version.
• Observability: Log token usage, latency, and error rates for every call, bringing cost and quality into the same monitoring dashboard. Managed cloud services (such as the model service platforms provided by Baidu AI Cloud) typically have built-in authentication, rate limiting, logging, and monitoring, saving you the effort of building peripheral infrastructure.

A generic API call example compatible with mainstream open platforms:

import os, httpx
API_KEY = os.environ["MODEL_API_KEY"]
resp = httpx.post(
    "https://{your-platform-endpoint}/v1/chat/completions",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": "your-model-id",        # Explicitly specify version in production
        "messages": [
            {"role": "system", "content": system_prompt},  # Stable content first for cache hits
            {"role": "user", "content": user_input},
        ],
        "temperature": 0.2,
    },
    timeout=120,
)
print(resp.json()["choices"][0]["message"]["content"])

X. Conclusion: Turning Evaluation into a Reproducible Internal Process

Evaluating a newly released model can entirely be distilled into a fixed team process:

1. Read the release notes and extract architectural facts (parameters, context length, modalities, attention design). Flag all “claims to be verified.”
2. Cross-reference official self-evaluations, independent indices, and blind test arenas to form an initial screening impression.
3. Run tests on real tasks using a private evaluation set, simultaneously recording cost and latency.
4. Integrate via a small-scale canary release, observing error recovery, cache hit rates, and end-to-end quality.
5. Output a one-page conclusion: applicable scenarios, non-applicable scenarios, unit cost per quality, and version recommendations.

Model iterations will continue to happen, but this framework can be reused long-term. For developers, the most important capability is not memorizing leaderboard rankings, but possessing an engineering methodology that translates “new models” into “can we use it, is it worth using, and how should we use it.”