LLM Arena: A Complete Guide to Multi-Dimensional Benchmarking and Evaluation
This guide explains how to build a structured "arena" for evaluating large language models across perception, reasoning, code generation, and agentic tasks. It covers the five core problems that make model selection hard, four design principles for fair comparison, and a seven-stage pipeline that turns test cases into reports and regression alerts. You will find a unified model-adapter layer, layered and versioned test cases, multimodal and video-understanding metrics, reasoning-effort tradeoffs, five-dimension scoring for one-shot frontend generation, sandbox and repository-level code evaluation, agentic scoring metrics, prompt-ordering strategies that improve cache hit rates, a cost-estimation example, and a comparison of open-weight versus hosted models. It closes with practical notes on small parameter models, edge deployment, CI integration, and a pre-launch checklist. The goal is not a single leaderboard, but an engineering system that continuously produces defensible selection decisions.
1. Introduction: Why Build an Arena?
Over the past year or two, foundation models have iterated almost monthly. A new version might break through on one of visual understanding, code generation, or tool calling, or it might simply reason more “carefully,” knowing better when to stop. The more information there is, the harder selection becomes. Five difficulties stand out:
- Fragmented dimensions: Text conversation, multimodal perception, code generation, and agent execution test different capabilities. A high overall score does not mean a model works well in your scenario.
- Aging benchmarks: The longer a task stays public, the more likely it enters training data, inflating scores and eroding discriminative power.
- Non-determinism: Sampling parameters, reasoning-effort levels, context length, concurrency, and rate limiting all influence results. Change one set of parameters and the conclusion can flip.
- Ignored cost: Comparing quality without cost fails to capture real per-task expenses in production. Agentic workloads are especially affected—input-side costs often far exceed expectations.
- Invisible process: Recording only the final answer makes it impossible to analyze how a model “thinks and acts,” and failures are hard to localize.
Organizing evaluation as an “arena” is a direct response to these problems: the same test cases, the same scoring rules, and the same runtime environment let candidate models compete head-to-head, so conclusions come from data rather than impressions.
1.1 Four principles
- Same task, same stage: The only variable allowed between models is the “subject under test.” Sampling parameters, prompts, tools, and environment must be identical.
- Reproducible: Anyone using the same version and configuration can reproduce consistent results.
- Observable process: Every step’s input, output, and tool-call records are fully preserved, so failures can be localized to a specific stage.
- Accountable cost: Accuracy, latency, and cost appear in the same report, making it easy to plot a cost-effectiveness curve.
1.2 What an arena is made of
A functioning arena contains at least five parts:
- A unified access layer that abstracts away protocol differences across model services;
- A layered test-case set that version-controls what is tested and how it is judged;
- Execution and sandboxing that safely runs model-generated code in isolation;
- An automated scorer that turns subjective impressions into repeatable metrics;
- Reporting and regression that turn results into trends and trigger alerts on key changes.
2. Overall Architecture: From Test Cases to Reports
The evaluation pipeline can be split into seven stages:
Case loading → Unified access layer → Sandbox execution / environment interaction → Raw record collection→ Scorer → Metric aggregation → Reports and regression alerts
2.1 Unified access layer: turning “swap the model” into “change the config”
If test code is littered with a specific vendor’s calling conventions, swapping a model means rewriting the evaluation. A safer approach is to define a unified adapter interface: it exposes only four generic capabilities upward—text input, multimodal input, tool calling, and streaming output—while each adapter handles protocol conversion downward.
from dataclasses import dataclass, fieldfrom typing import Any, Protocol@dataclassclass ModelSpec:"""Model capability profile; determines which cases can be sent to this model."""model_id: str # Internal identifier; does not expose vendor information directlysupports_vision: bool = Falsesupports_tools: bool = Falsecontext_limit: int = 128_000 # Context window limit, used for trimming history@dataclassclass CallRecord:"""Raw record of a single call; the basis for cost accounting and regression analysis."""model_id: strprompt_tokens: intcompletion_tokens: intlatency_ms: floatmeta: dict = field(default_factory=dict)class ModelAdapter(Protocol):def chat(self, messages: list[dict], **params: Any) -> str:"""messages use a generic message format; multimodal content isrepresented as typed blocks in content."""REGISTRY: dict[str, ModelAdapter] = {} # model_id -> adapter instancedef run_case(case: dict, spec: ModelSpec) -> tuple[str, CallRecord]:adapter = REGISTRY[spec.model_id]messages = build_messages(case) # Case -> unified message formatparams = sampling_params(case) # Temperature, max output, reasoning effort, etc.text = adapter.chat(messages, **params)return text, adapter.last_usage() # Raw usage is passed to the unified scorer
The access layer needs to enforce three points:
- Centralized parameter management: Sampling parameters live in the case or model config, not scattered as hard-coded values in code.
- Capability declaration first: Each model first declares whether it supports vision, tool calling, and how large a context it accepts. Unsupported cases are skipped automatically with a stated reason, avoiding the misreading of a “blank result” as “lack of ability.”
- Raw metering is mandatory: Every call records prompt tokens, output tokens, latency, and usage details. Do not rely on costs converted after the fact by the platform; the accounting basis is hard to unify and impossible to recompute.
2.2 Layered test cases: declaring “what to test” in configuration
It is best to organize the test set into four layers by capability, with each case testing only one capability:
| Layer | Focus | Example cases |
|---|---|---|
| Perception | Text, image, and video understanding and extraction | Screenshot reconstruction, chart reading, screen-recording replay |
| Reasoning | Depth and robustness of multi-step reasoning | Logic puzzles, math proofs, resolving conflicting requirements |
| Code | Single-turn generation and repo-level fixes | One-shot page generation, defect localization and fixing, test completion |
| Agent | Multi-turn tool calling and long-horizon planning | Gamified environments, database operations, multi-source retrieval |
| Cross-cutting | Spans all layers | Latency, cost, stability, output-format compliance |
Cases themselves should be declarative and versionable:
id: ui-restore-login-pagelayer: codeinput:image: fixtures/ui/login-page.pnginstruction: Reconstruct the page from the screenshot only; keep interaction behavior identical to the originalscoring:weights: {build: 25, console: 15, visual: 30, a11y: 20, deps: 10}threshold: 75budget:steps: 1timeout_s: 600
This yields three benefits:
- Zero-cost onboarding of new models, which only need to be registered in config;
- Traceable case changes, where every adjustment enters the version record and historical scores can be recomputed;
- Calibratable weights: First have senior engineers manually score a batch of samples, then repeatedly adjust the automated scorer’s weights until the two rankings largely agree. Only then does the score carry reference value.
2.3 Reproducibility: the control-variable checklist
Before evaluation begins, the following variables must be frozen and recorded:
- Model version: Record the snapshot or version number to prevent silent server-side upgrades during evaluation;
- Sampling parameters: Temperature, max output length, reasoning-effort level, and so on;
- Prompts and tool definitions: Including the system prompt, tool schemas, and examples;
- Runtime environment: Dependency versions, image versions, resource quotas;
- Scheduling conditions: Concurrency, retry policy, timeout.
Note that “set temperature to 0” does not mean fully deterministic. Batch scheduling, floating-point accumulation order, and other factors can still produce subtle differences for the same input. If a case is sensitive to stability, sample multiple times and report mean and variance rather than concluding from a single run.
3. Perception Layer: Hands-On Multimodal and Video Understanding
3.1 “Can see images” is not the same as “sees accurately”
Once many models support image and even video input, the first use is often “describing a picture,” but this is far from the limit. The valuable question in evaluation is: can the observed content be extracted accurately, completely, and in structured form, and then used reliably in downstream tasks?
It helps to split perception cases into three types:
- Recognition: table reading, formula transcription, on-screen text localization;
- Understanding: judging inflection points in trend charts, mapping relationships in diagrams, reconstructing operation steps from video;
- Generation: producing a runnable frontend page from a UI screenshot.
3.2 Three ways to handle video input
How video input is implemented often affects results more than the model itself:
- Frame extraction into an image sequence: Good compatibility and controllability, but too large an interval misses key actions while too small an interval wastes context.
- Native video-stream input: Better preserves temporal information, but demands more of context length, input cost, and latency.
During evaluation, try 1 FPS, 2 FPS, keyframe extraction, and other strategies on the same video, then compare task completion against input cost to find the inflection point for your scenario.
3.3 Quantifying the perception layer
- Field-level accuracy: The proportion of correctly filled fields in structured extraction;
- Completeness: How many of the expected items were actually recognized;
- Hallucination rate: The share of objects or data in the output that do not exist in the input;
- Localization error: Intersection-over-union of bounding boxes, or pixel-level reconstruction difference;
- Temporal correctness: Whether step order is correct and whether any steps are missing.
3.4 A structured-extraction example
Content such as video tutorials and screen recordings is often used to train “follow-the-flow” ability. You can ask a model to transcribe a recording into structured steps:
System prompt: You are a screen-recording transcriber. Describe only facts observablein the frames; do not infer the operator's intent; set undeterminable fields to null.User input: <video or frame sequence> Output a JSON array where each step contains:step step indextimestamp the second at which the step first appears in the framesaction one of click / type / scroll / page navigationtarget name of the control or region being acted uponscreen the page shown after the action
Score field by field. Giving a correct judgment to fields left as null suppresses the “rather fabricate than leave blank” tendency, which is one of the main sources of hallucination.
4. Reasoning Layer: Deep Thinking and “Reasoning Effort”
4.1 What reasoning effort means
Recent models generally offer distinct modes such as “fast response” and “deep thinking,” with adjustable thinking time or token budget. This creates a new evaluation question: is the high-effort mode worth its extra time and cost?
4.2 A difficulty-stratified question bank
Split reasoning cases into four tiers:
- Common-sense reasoning;
- Multi-step logic and mathematics;
- Complex problems requiring self-verification and backtracking;
- Competition-level hard problems.
By recording pass rate, average thinking tokens, and end-to-end latency at different effort levels for each tier, you can draw a “capability vs. cost” curve. Most teams find that routine business problems nearly saturate at medium effort, and only high-difficulty cases need higher effort.
4.3 Process quality matters more than the answer
For a single reasoning run, beyond the final answer, also check:
- Self-correction: Can it detect and fix contradictions when they arise?
- Avoiding cognitive laziness: Does it overthink simple problems?
- Process readability: Does the reasoning provide verifiable grounds in its summary?
5. Code Layer, Part 1: Frontend One-Shot Generation
5.1 What frontend one-shot generation is
Frontend one-shot generation means producing a runnable page from a single prompt, without relying on multiple rounds of follow-up and debugging. It simulates the real scenario of “a product manager hands a design to a developer and expects it done in one pass,” placing high demands on the model’s UI understanding, layout ability, and code completeness.
5.2 Five-dimension scoring
The automated scorer rates five dimensions, each with a clear, computable criterion:
| Dimension | Judgment method | Suggested weight |
|---|---|---|
| Runnability | Whether the build passes and whether there are runtime errors | 25 |
| Console cleanliness | Number of errors and warnings during load and interaction | 15 |
| Visual fidelity | Difference between the rendered screenshot and the reference image | 30 |
| Accessibility | Semantic tags, image alt text, focus visibility, and so on | 20 |
| Dependency sanity | Whether undeclared external dependencies are introduced, and any security risks | 10 |
5.3 Scoring implementation example
import jsondef score_frontend(result: dict, weights: dict | None = None) -> dict:"""result is produced automatically by the sandbox. Example fields:build_ok bool whether the build succeededconsole_count int number of errors and warnings during load and interactionvisual_score float combined visual difference score between renderedand reference screenshots, in [0, 1]a11y_pass float fraction of accessibility checks passed, in [0, 1]deps_ok bool whether dependencies are fully declared and nosuspicious external resources are introduced"""w = weights or {"build": 25, "console": 15, "visual": 30,"a11y": 20, "deps": 10}parts = {"build": 1.0 if result["build_ok"] else 0.0,"console": max(0.0, 1.0 - result["console_count"] / 5),"visual": max(0.0, min(1.0, result["visual_score"])),"a11y": max(0.0, min(1.0, result["a11y_pass"])),"deps": 1.0 if result["deps_ok"] else 0.0,}detail = {k: round(parts[k] * w[k], 1) for k in w}return {"score": round(sum(detail.values()), 1), "detail": detail}
5.4 Automated rendering and screenshot comparison
The key is “automated rendering”: the sandbox starts a local preview server, uses browser automation to load, screenshot, and traverse interactions, then compares the screenshot against the reference. Pixel-level difference is too sensitive to fonts, rendering engines, and window size. A combined “structural similarity + color histogram” metric is recommended, with a unified viewport size and scale factor.
def visual_score(rendered_path: str, reference_path: str) -> float:"""Weighted average of structural similarity (0~1, higher is closer) andcolor histogram similarity. Rendering and comparison must be done at afixed viewport size to avoid font differences dominating the result."""ssim = 0.0 # structural_similarity(rendered_path, reference_path)color = 0.0 # histogram_similarity(rendered_path, reference_path)return round(0.7 * ssim + 0.3 * color, 4)
If the environment allows network access, also add an “offline usability” check: whether the page still renders and behaves correctly after disconnecting from the network, which directly reflects how much it depends on external resources.
5.5 Interpreting results
- High visual fidelity but poor accessibility indicates the model favors “good-looking” over accessible.
- A high build-failure rate often stems from undeclared dependencies or mismatched environment versions, which can improve by stating the tech stack and version constraints explicitly in the prompt.
- Many console warnings usually relate to framework versions and lifecycle misuse; fixing the framework version in the case reduces noise.
6. Code Layer, Part 2: Backend and Repository-Level Programming
6.1 What matters in backend tasks
Backend code is judged by different standards than the frontend. It cares more about:
- Functional correctness;
- Boundary-condition handling;
- Security, such as input validation and parameterized queries;
- Concurrency and idempotency design;
- Error handling and observability.
6.2 Judging repository-level defect fixes
More challenging are repository-level tasks: given a project with a defect, the model must locate, modify, and submit a patch itself. The standard automated judging flow is:
- Apply the model-generated patch;
- Run the “fix-verification cases,” which must all pass;
- Run the “regression cases,” where no new failures are allowed;
- Check whether static-analysis tools report new issues.
The score can be combined as “verification pass rate × 0.6 + regression retention rate × 0.4.” If the same case is unstable across runs (flaky), isolate or fix it first; otherwise it becomes a source of noise for the whole score set.
6.3 Preventing “teaching-to-the-test” behavior
Guard against the model editing test files, commenting out validation logic, or writing hard-coded values unrelated to the problem just to make tests pass. Countermeasures include:
- Making test files read-only, or bringing them under checksum validation;
- Running a diff check on changes and penalizing clearly expanded scope;
- Keeping a portion of hidden cases, used only during final scoring.
6.4 Sandbox and resource limits
Model-generated code is inherently untrusted, so the execution environment must be isolated:
- Use containers or micro-VMs, blocking access to the host network and sensitive directories;
- Set upper bounds on CPU, memory, and runtime;
- Run tool calls inside the sandbox too, with allow-list restrictions on file read/write scope;
- Write all commands, file changes, and network requests to an audit log.
7. Agent Layer: Tool Calling, Long-Horizon Tasks, and “Gamified” Testing
7.1 Why test agents with games
Agent capability is hard to measure with static problems, because it tests “continuous decision-making in a dynamic environment.” Games happen to have several natural advantages:
- Clear objectives, with wins and losses judgeable automatically;
- Continuously changing state, testing multi-step planning;
- Closed rules, making fair comparison easy;
- The ability to record every action, forming a complete trajectory.
Many evaluation teams use board games, text adventures, or sandbox games for stress testing, observing whether a model keeps its goal consistent over long turns and corrects errors in time. Interestingly, game environments also expose differences in a model’s “personality”: some are aggressive risk-takers, others cautious and conservative. This directly affects its risk appetite in real business.
7.2 Agent scoring metrics
| Metric | Definition | Note |
|---|---|---|
| Task completion rate | Share of turns that reach the goal | Core metric |
| Step efficiency | Minimum viable steps / actual steps | Measures redundant actions |
| Tool-call correctness | Share of calls with valid arguments | Reflects understanding of tool definitions |
| Recovery ability | Share of cases that successfully continue after an error | Shows fault tolerance and retry strategy |
| Trajectory length | Average number of turns to complete the task | Directly affects call cost |
7.3 Common problems in long-horizon tasks
- Goal drift: Gradually deviating from the original goal as turns accumulate;
- Repeated ineffective actions: Executing the same step over and over;
- Context forgetting: Early constraints diluted in a long context;
- Premature stopping: Declaring the task complete without finishing verification.
The corresponding improvements are to introduce external memory, periodic summaries, and explicit verification steps, rather than simply enlarging the context window.
8. Cost and Caching: Getting the Per-Task Expense Right
8.1 Cost is more than unit price
Real cost depends on token usage, cache hit rate, retry count, and failure rate. For agentic applications, the input side is usually much larger than the output side, because every turn carries the system prompt, tool definitions, and history.
An easily overlooked fact: when the cache hit rate is high, the billed cost on the input side can be only a small fraction of the miss scenario. But this has a precondition—the request prefix must stay stable, and cache matching follows the provider’s published rules. So the first step in cost analysis is to separate the “prefix” from the “variable part.”
8.2 Prompt structure: stable content first, volatile content last
┌──────────────────────────────────────────────────┐│ System prompt and tool definitions (stable) │ ← change as little as possible across turns├──────────────────────────────────────────────────┤│ Conversation history (grows by appending) │ ← append new content each turn├──────────────────────────────────────────────────┤│ Current turn's user input and retrieval (volatile) │ ← keep at the very end└──────────────────────────────────────────────────┘
Three engineering practices:
- Do not put volatile information such as timestamps or random IDs in the prefix, or it breaks prefix consistency every turn;
- Adding or removing tool definitions changes the prefix, so agentic apps that frequently adjust their tool set need to re-evaluate their caching strategy;
- Keep order stable when evicting history, truncating from the start or replacing whole segments with a summary, to avoid changing content in the middle.
8.3 A cost-calculation example
def estimate_cost(usage: dict, price: dict, cache_discount: float = 0.25) -> dict:"""usage: raw usage, containing prompt_tokens, completion_tokens, cached_tokensprice: pricing config with prompt and completion entriescache_discount: billing discount for the cached portion; follow theprovider's actual billing rules"""fresh = usage["prompt_tokens"] - usage.get("cached_tokens", 0)input_cost = (fresh + usage.get("cached_tokens", 0) * cache_discount) * price["prompt"]output_cost = usage["completion_tokens"] * price["completion"]return {"input_cost": round(input_cost, 6),"output_cost": round(output_cost, 6),"total": round(input_cost + output_cost, 6),"cache_hit_rate": round(usage.get("cached_tokens", 0) / max(usage["prompt_tokens"], 1), 4),}
8.4 A reporting perspective
Plotting “task completion rate” on the vertical axis and “per-task cost” on the horizontal axis puts all candidate models on one chart. The top-left represents high cost-effectiveness, the top-right represents high capability but high cost, and the bottom-left should be eliminated. Selection is not about finding “the best model,” but about finding the model near the inflection point of your business curve.
9. Open vs. Closed Weights: What Public Weights Change
9.1 The real value of public weights
Some compare releasing model weights to a martial-arts master giving away a secret manual for free: making the manual public does not mean everyone can master it. Similarly, open weights lower the barrier to use, not the barrier to capability. The real gap often lies in data quality, training recipes, inference optimization, and engineering tooling.
For enterprises, weight models that can be deployed in their own environment usually mean:
- Data control: Sensitive data stays within the domain, easing compliance;
- Deep customization: Domain fine-tuning combined with private data;
- Predictable cost: Under stable load, the resource cost of self-hosted inference is more estimable;
- Avoiding lock-in: Not being tied to a single provider’s product cadence.
9.2 Selection comparison dimensions
| Dimension | Open-weight models | Hosted commercial models |
|---|---|---|
| Time to start | Requires self-deployment and tuning | Ready out of the box |
| Customization | High; fine-tuning and distillation | Limited, mainly prompts and parameters |
| Ops burden | Higher; involves compute and scaling | Handled by the provider |
| Capability ceiling | Depends on the specific version | Usually keeps pace quickly |
| Compliance control | Data can stay in your own environment | Depends on the provider’s compliance |
The conclusion is not absolute: sensitive-data, stable-load scenarios suit open-weight approaches; teams chasing fast launch and multimodal capabilities often validate value with hosted services first.
10. Small Parameter Models: Capability Density and Boundary Awareness
10.1 “Small” does not mean “weak”
Parameter count is often used as a proxy for capability, but it is not always reliable on specific tasks. Some vision models in the mere billion-parameter range can already approach larger models on spatial-relation judgment, boundary recognition, and small-object localization.
The reasons are usually threefold:
- Computation for vision tasks can be highly parallel, so inference-time compute does not fully track parameter scale;
- High-quality, fine-grained labeled data is especially critical to shaping small-model capability;
- The clearer the task boundary, the more easily distillation and targeted fine-tuning can “compress” capability into a small model.
10.2 What to focus on when testing small models
- Spatial geometry: Relative position, occlusion, overlap area;
- Boundary awareness: Whether segmentation edges are sharp and whether tiny structures are missed;
- Counting and localization: Number and coordinate precision of small objects;
- Deployment feasibility: Accuracy loss after quantization, memory footprint, per-frame inference latency.
10.3 On-device deployment considerations
The real value of small parameter models lies on-device and at the edge:
- Low GPU or memory footprint, runnable on consumer hardware;
- Controllable inference latency, suitable for real-time interaction;
- Data stays local, with lower privacy risk;
- Usable offline, suitable for network-restricted scenarios.
When evaluating edge models, report “accuracy–memory–latency” as a triple rather than giving a single accuracy number.
11. Engineering the Evaluation Pipeline
11.1 Directory and configuration
arena/├── adapters/ # Unified adapters for each model service├── cases/ # Declarative test case sets (YAML/JSON)├── runners/ # Single-turn, multi-turn, sandbox runners├── judges/ # Automated scorers├── sandbox/ # Container images and resource limit config└── reports/ # Structured results and visual reports
11.2 Integrating with continuous integration
Wire evaluation into CI to trigger regression automatically when prompts, tool definitions, or model versions change:
- Run the full or sampled case set;
- Compare against baseline scores and alert when the drop exceeds a threshold;
- Write results to a database and retain historical trends;
- Auto-archive failing cases with full trajectories for review.
11.3 Building an enterprise evaluation pipeline on Baidu AI Cloud
Enterprises can build their own evaluation pipeline on Baidu AI Cloud:
- Unified model-service access: Package internal models and external services under one interface, so evaluation code targets only that interface;
- Object storage: Store cases, reference assets, and historical reports with built-in versioning;
- Message queues: Decouple case scheduling from scoring execution to support large-scale concurrent evaluation;
- Logging and monitoring alerts: Collect usage and latency for each call and notify automatically when metrics go anomalous.
Combined, these capabilities can evolve the “arena” from a one-off script into infrastructure that can be operated sustainably.
12. Conclusions and Checklist
12.1 Core takeaways
- Evaluation is an engineering problem, not a one-time benchmark run: Cases, environments, and scorers all need versioning and continuous maintenance.
- Layered design beats blanket scoring: Perception, reasoning, code, and agents each have their own metrics; an overall score is only a reference.
- Cost and quality must be viewed together: Per-task cost is the real basis for business decisions.
- Process carries more information than results: Trajectories, tool calls, and failure reasons are key inputs for improving prompts and workflows.
- There is no single right choice: Open vs. closed, small vs. large models, fast vs. slow reasoning—each has its fit. Decide with data, not impressions.
12.2 Pre-launch checklist
- Does the case set cover real business scenarios, not just public question banks?
- Does each case declare a clear scoring rule and weights?
- Are sampling parameters, model versions, and runtime environments frozen and recorded?
- Are there contaminated questions or long-flaky cases?
- Is the sandbox truly isolated, and do tool calls have permission boundaries?
- Does cost accounting include input, output, and cache hits?
- Are full trajectories preserved, and can the root cause of a single failure be located?
- Is there a regression baseline, and can key changes auto-trigger comparison?
Only when these questions are addressed one by one does the “arena” become more than a leaderboard—it becomes an engineering system that continuously produces decision-ready evidence.
