Design and Implementation of a Multi-Model Anonymous Arena Evaluation System
As the number of AI models rapidly increases and iteration cycles shorten, single-dimensional offline evaluations struggle to determine which model is more stable, preferred, and less prone to quality degradation under real user prompt distributions. This article explores the design and practice of a scalable, sustainable multi-model anonymous arena evaluation system. It addresses key engineering challenges, including output non-determinism, prompt distribution shifts, reviewer bias, and reproducibility. By detailing a six-layer architecture, task scheduling, statistical confidence calculation, and data governance, this guide demonstrates how to transform unstable model outputs into explainable, reproducible, and auditable comparative evidence for model selection and quality regression.
Design and Implementation of a Multi-Model Anonymous Arena Evaluation System
1. Background and Problem Definition
When the number of models grows rapidly and iteration cycles continuously shorten, single-dimensional offline evaluation struggles to answer a practical question: under a real user prompt distribution, which model is more stable, better aligns with preferences, and exhibits less quality degradation? The so-called “infinite arena” is essentially a sustainably scalable anonymous dueling evaluation system. Candidate models can join dynamically, evaluation tasks can be generated continuously, and the system outputs explainable comparative conclusions through anonymous pairing, randomized presentation, and statistical aggregation.
Although such a system appears to simply “let two models answer the same question and let a judge pick the winner,” it faces at least five engineering challenges:
- Output Non-Determinism: The same model may produce significantly different outputs under different sampling parameters or prompt versions. A single result is insufficient to support a conclusion.
- Prompt Distribution Shift: If the evaluation set remains fixed long-term, it will quickly overfit. If completely random, it may lack coverage.
- Reviewer Bias: Reviewers may prefer longer, more formatted, or earlier-appearing answers, leading to position bias, length bias, and style bias.
- Repeated Evaluation and Confidence: Repeating an evaluation twice for the same candidate model only serves as a preliminary signal and cannot be directly equated to a stable conclusion. It requires recording sampling conditions and calculating confidence intervals.
- Reproducibility and Auditability: Model versions, prompt templates, sampling parameters, reviewer strategies, and aggregation algorithms must all be versioned; otherwise, conclusions cannot be traced.
Therefore, the system design goals should be clearly defined as: anonymous, randomizable, repeatable, observable, auditable, and scalable. The “infinite” in the title should not be understood as infinitely stacking machines, but rather that evaluation tasks, candidate models, and reviewer strategies can be continuously integrated, maintaining system stability through orchestration and governance.
2. Overall Architecture
It is recommended to split the system into six layers to avoid deep coupling between evaluation logic and infrastructure:
- Access Layer: Receives candidate model registration, evaluation task submission, manual review, and result query requests.
- Task Orchestration Layer: Responsible for pairing strategies, repeated sampling counts, priority, rate limiting, timeouts, and retries.
- Inference Execution Layer: Calls candidate models in an isolated environment, recording inputs, outputs, latency, error codes, and sampling parameters.
- Review and Aggregation Layer: Executes anonymous review, position swapping, win rate statistics, confidence interval calculation, and outlier removal.
- Data and Versioning Layer: Saves prompt sets, model versions, evaluation events, review records, and aggregation results. All objects carry version numbers.
- Observability and Governance Layer: Provides logs, metrics, tracing, alerts, audits, and cost dashboards.
Regarding deployment, a container platform can host the inference execution and orchestration services, object storage can save desensitized evaluation samples, message queues can handle peak shaving and valley filling, and logging services combined with monitoring and alerting can build the observability system. If implemented based on Baidu AI Cloud, general-purpose capabilities like its container platform, object storage, message queue, logging service, and monitoring and alerting can be used, but specific interfaces and quotas should be subject to the actual environment, and infrastructure details should not be hardcoded into the evaluation logic.
3. Anonymous Duels and Task Scheduling
The core of an anonymous duel is not simply “hiding names,” but controlling information leakage and order bias. Each match should generate an independent session identifier, candidate models should appear with anonymous IDs, and the review interface should not display the vendor, version, or parameter count. Pairing should use a randomized strategy and execute position swapping for the same prompt: A/B and B/A are each reviewed once, reducing the bias of “the first-appearing answer is more likely to be selected.”
The scheduling layer must guarantee idempotency. Tasks may be retried due to timeouts; without an idempotency key, duplicate executions will pollute the statistics. It is recommended to use “Prompt ID + Model Version + Sampling Parameter Hash + Round” as the idempotency key. Execution results are written to the event table before triggering aggregation.
function schedule_duel(prompt, modelA, modelB, round):session = new_anonymous_session()order = random_choice([A_FIRST, B_FIRST])for side in order:task = build_task(idempotency_key = hash(prompt.id, side.model.version,side.sampling_hash, round),timeout = policy.timeout,retry = policy.retry_with_backoff)enqueue(task)record_session(session, prompt.id, modelA.version, modelB.version,order, round)
Scheduling should also support tiered queues: routine evaluations use low priority, regression evaluations use high priority, and manual spot checks use a separate queue. Rate limiting needs to be set across three dimensions: candidate model, tenant, and global, to prevent anomalous retries from a single model from dragging down overall throughput. Timeouts and error codes must be classified and recorded (e.g., input exceeding limits, content safety interception, service unavailable, and empty output) and should not simply be categorized as “failure.”
4. Repeated Evaluation and Statistical Confidence
The original concept of “the same candidate being tested twice” highlights a key mechanism: repeated sampling. Two evaluations can reduce chance, but the results still need to be interpreted cautiously. Engineering-wise, the repetition count N should be a configurable parameter, and each sampling’s temperature, top-p, random seed, prompt template version, and evaluation round should be recorded. If N=2, it is only suitable for preliminary screening or smoke regression; for formal conclusions, it is recommended to incorporate more rounds, confidence intervals, and stratified analysis.
Statistically, pairwise comparison can be used: each match produces a win, loss, tie, or invalid result, which is then aggregated into a win rate. For win rates, a confidence interval should be provided rather than just a point estimate. Bootstrap resampling can be used to estimate the interval, or a binomial distribution approximation can be applied. When aggregating, stratify by prompt category, language, and length range to prevent a certain type of prompt from dominating the overall conclusion.
function aggregate(records):valid = filter(records, r -> r.status == VALID)by_model = group_by(valid, r -> r.model_version)for model, rows in by_model:wins = count(rows, r -> r.outcome == WIN)losses = count(rows, r -> r.outcome == LOSS)ties = count(rows, r -> r.outcome == TIE)win_rate = (wins + 0.5 * ties) / (wins + losses + ties)ci = bootstrap_ci(rows, win_rate, iterations=1000)report(model, win_rate, ci, sample_size=len(rows))
It must be emphasized that repeated evaluation twice does not mean “testing twice will definitely be more accurate”. If the sampling conditions for both runs are exactly the same, it only reflects stability under that condition; if the conditions are different, the variables must be clearly defined. The system should prohibit directly mixing results from different prompt versions or different sampling parameters. For anomalous outputs, such as empty responses, truncations, or severe formatting errors, they should be marked as invalid and sent for manual review rather than counted towards the win/loss tally.
5. Debiasing and Quality Control
Debiasing should occur simultaneously at three levels: data, presentation, and review.
- Data Layer: Sample prompt sets stratified by topic, difficulty, language, and length. Rotate periodically, maintaining a fixed regression set and a dynamic exploration set.
- Presentation Layer: Use anonymous IDs, random order, position swapping, and uniform rendering styles to avoid displaying model names, version numbers, and parameter counts.
- Review Layer: Calibrate reviewer models and incorporate manual spot checks. Conduct specific analyses for length bias, format bias, and position bias.
Quality control can implement gates: if a match experiences a timeout, empty output, safety interception, or reviewer conflict, it enters a review queue. If a candidate model’s continuous error rate exceeds a threshold, it is automatically downgraded or suspended. If the confidence interval of an aggregated result is too wide, it is marked as “insufficient evidence,” and no deterministic conclusion is published externally. Manual spot checks should cover highly contested samples and edge cases, with the results fed back as reviewer calibration data.
6. Data Governance and Observability
The credibility of the evaluation system largely depends on data governance. It is recommended to establish four types of versions: prompt set version, model version, sampling strategy version, and aggregation algorithm version. Each evaluation event should include a trace ID, session ID, prompt ID, anonymous side identifier, model version hash, sampling parameters, start and end times, status code, and review result. Sensitive information must be desensitized before storage, original inputs and outputs should be retained following the principle of least privilege, and access audits should be enforced.
Observability metrics should at least include: task queue time, execution success rate, timeout rate, invalid result rate, reviewer conflict rate, repeated evaluation consistency rate, sample size per stratum, and cost consumption. Logs, metrics, and traces should be correlated via the same trace ID, facilitating the identification of whether an “anomalous conclusion” was caused by a shift in prompt distribution, a change in model version, or a change in aggregation logic.
7. Security, Compliance, and Cost Control
The arena processes a large volume of user prompts and model outputs, necessitating robust tenant isolation, key management, content security filtering, and access control. Candidate model call credentials should not appear in the task payload; they should be retrieved by the execution layer through controlled means. When presenting results externally, avoid leaking internal version numbers and non-desensitized samples.
For cost control, general quotas and budget alerts can be established based on call counts, input/output scale, and storage volume. Repeated evaluations linearly increase costs; therefore, dynamic sampling should be supported: increase rounds for highly contested samples and reduce rounds for low-risk samples. Use higher sample sizes for formal conclusions and smaller sample sizes for smoke regressions.
8. Implementation Suggestions
It is recommended to proceed in three phases: Phase 1 establishes anonymous dueling, event recording, and a basic win rate dashboard; Phase 2 adds position swapping, repeated sampling, confidence intervals, and stratified analysis; Phase 3 completes reviewer calibration, anomaly detection, automated regression, and audit capabilities. When presenting conclusions externally, simultaneously provide the sample size, confidence interval, evaluation time window, and version information. Avoid packaging “preliminary results from two evaluations” as an absolute ranking.
In summary, the engineering focus of a multi-model anonymous arena is not about “infinitely increasing matches,” but rather using randomization, repeated sampling, statistical confidence, and governance mechanisms to transform unstable model outputs into explainable, reproducible, and auditable comparative evidence. Only by closing the loop on scheduling, data, review, and observability can the arena continuously scale and genuinely support model selection and quality regression.
