AI Model Infinite Arena Evaluation: Architecture Design and Practice

    This article explores the architecture and engineering practices for an infinite arena that evaluates AI models through anonymous pairwise comparisons. It covers requirements for fairness, reproducibility, repeated sampling, anti-abuse protection, cost control, and data governance; presents a layered asynchronous system design spanning access, battle orchestration, model invocation, vote collection, rating computation, data governance, and observability; and explains idempotent battle creation, explicit repeated evaluation, pairwise rating methods with uncertainty reporting, audit controls, and operational monitoring. The goal is to show how model evaluation can move from noisy voting toward a more trustworthy and transparent measurement system without relying on a specific proprietary interface.

    As model capabilities evolve rapidly, static leaderboards are increasingly insufficient for comparison in real-world scenarios. Static prompts are easy to leak, scoring dimensions are often one-dimensional, and such benchmarks struggle to reflect multi-turn interaction and subjective preference. Anonymous battle-style evaluation, in which users compare two responses without knowing which model produced each one and votes continuously update ratings, has become an important complement to model evaluation. The term “infinite arena” emphasizes a system that can continuously onboard new models, dynamically generate battles, accumulate preference data over time, and remain fair, reproducible, and observable under high concurrency and complex adversarial conditions.

    This article starts from general technical problems and discusses the evaluation architecture, key mechanisms, and engineering practices of a multi-model infinite arena. It does not target any specific platform or product.

    I. Core Challenges: From “It Works” to “Trustworthy”

    An infinite arena is not simply “two models answer the same question and users vote.” To produce trustworthy evaluation, at least the following problems need to be addressed:

    1. Fairness: Model identities must be anonymous, and display order must be randomized to avoid position bias. Different model versions, prompt versions, and sampling parameters must be fixed and recorded.
    2. Reproducibility: The same set of model versions, prompts, sampling parameters, and random seeds should be able to replay the same or explainable battle results. Rating rules, data cleaning rules, and abnormal vote handling rules should also be versioned.
    3. Repeated evaluation and confidence: A single battle contains a large amount of noise. The same model needs to be evaluated multiple times across different prompts, different opponents, and different time windows. The system should support repeated sampling and output confidence intervals or uncertainty indicators rather than only a bare score.
    4. Anti-abuse and adversarial resistance: Votes may be polluted by scripts, bulk accounts, or strategic behavior. Rate limiting, deduplication, behavioral anomaly detection, auditing, and manual review mechanisms are required.
    5. Cost and latency: Model invocation cost, queuing latency, and vote collection speed directly affect system throughput. Cost needs to be controlled through asynchronous orchestration, caching, batching, and elastic scheduling.
    6. Data governance: Battle logs, vote records, model versions, and rating snapshots need to be searchable, auditable, and retainable, while meeting de-identification and access control requirements.

    II. System Architecture: Layered Decoupling and Asynchronous Orchestration

    A scalable infinite arena can generally be divided into an access layer, battle orchestration layer, model invocation layer, vote collection layer, rating computation layer, data governance layer, and observability layer.

    • Access layer: Provides APIs for creating battles, retrieving anonymous responses, submitting votes, and querying leaderboards. The access layer should enforce authentication, rate limiting, idempotency key validation, and request auditing.
    • Battle orchestration layer: Selects prompts, pairs models, randomizes display order, generates battle identifiers, and maintains the battle state machine. A battle can move from “created” to “responses completed,” “waiting for vote,” “settled,” or “closed due to exception.”
    • Model invocation layer: Uses a unified adapter to hide input and output differences across models, and records model versions, prompt versions, sampling parameters, latency, and error codes. Invocation should support timeouts, retries, circuit breaking, and degradation.
    • Vote collection layer: Collects results such as “left better,” “right better,” “tie,” and “cannot determine,” together with session identifiers, prompt identifiers, battle identifiers, and necessary behavioral features. Vote records should be written to a message queue to smooth peaks, and then processed asynchronously by consumers.
    • Rating computation layer: Computes ratings and confidence intervals based on historical votes, supporting incremental updates and offline recomputation. The rating service should retain rating rule versions for replay and comparison.
    • Data governance layer: Uses object storage to save raw battle logs and model response snapshots, log services for retrieval, and monitoring and alerting to track key metrics. Data retention periods, de-identification policies, and access permissions must be clearly defined.
    • Observability layer: Monitors metrics such as battle creation volume, model invocation success rate, vote rate, abnormal vote ratio, rating drift, and queue backlog.

    In a Baidu AI Cloud environment, the layered architecture described above can be built using general capabilities such as object storage, message queues, container platforms, log services, and monitoring and alerting. The key is to combine general capabilities rather than depend on a proprietary interface.

    III. Repeated Evaluation and Idempotent Deduplication

    The idea that “the same model is tested multiple times” should be abstracted in engineering as a repeated evaluation mechanism. Repeated evaluation is not simply repeated invocation; it needs to distinguish “intentional repeated sampling” from “accidental duplicate submission.”

    It is recommended to generate an idempotency key from “model identifier + model version + prompt version + sampling parameters + random seed + prompt identifier.” When creating a battle, first check the idempotency key. If it already exists, return the existing battle directly to avoid duplicate billing and duplicate voting. For evaluation tasks that require multiple samples, a sample index can be added to the idempotency key, making repeated evaluation an explicit and traceable behavior.

    A pseudocode example is shown below:

    1. function createBattle(modelA, modelB, prompt, seed, sampleIndex):
    2. key = hash(
    3. modelA.id, modelA.version,
    4. modelB.id, modelB.version,
    5. prompt.id, prompt.version,
    6. seed, sampleIndex
    7. )
    8. if store.exists(key):
    9. return store.get(key)
    10. order = randomChoice(seed, ["A_FIRST", "B_FIRST"])
    11. responseA = invoke(modelA, prompt, seed)
    12. responseB = invoke(modelB, prompt, seed)
    13. battle = {
    14. key: key,
    15. anonymousOrder: order,
    16. responses: [responseA, responseB],
    17. status: "WAITING_VOTE",
    18. createdAt: now()
    19. }
    20. store.put(key, battle)
    21. return battle

    The value of repeated evaluation lies in estimating stability. Multiple samples from the same model on the same prompt can reveal output variation; multiple battles between the same model and different opponents can help estimate rating confidence intervals. The system should avoid counting repeated votes from the same user on the same battle as multiple votes, and should also avoid settling the same idempotency key more than once.

    IV. Rating Algorithms and Uncertainty Expression

    A common approach is to use pairwise comparison rating models, such as Elo-style incremental ratings or Bradley-Terry-style probabilistic models. Elo is suitable for online incremental updates and is simple to compute; Bradley-Terry is better suited to batch fitting and confidence interval estimation. In engineering practice, a dual-track mechanism of “online incremental updates + offline recomputation” can be adopted: the online side quickly updates displayed scores, while the offline side periodically recomputes based on all data to reduce order bias.

    Rating should account for ties, vote quality, and sample size. For new models, Bayesian smoothing or prior scores can be used to avoid extreme rankings caused by a small number of battles. Leaderboards should display ratings, battle counts, confidence intervals, and the latest update time together. For repeated evaluation data, aggregation can be layered by prompt, user, and time window to avoid a single dimension dominating the results.

    A pseudocode example is shown below:

    1. function updateRating(ratings, winnerId, loserId, kFactor):
    2. expectedWin = 1 / (1 + 10 ^ ((ratings[loserId] - ratings[winnerId]) / 400))
    3. ratings[winnerId] = ratings[winnerId] + kFactor * (1 - expectedWin)
    4. ratings[loserId] = ratings[loserId] - kFactor * (1 - expectedWin)
    5. return ratings

    A real system should also record the input snapshot of each update to support replaying ratings at a specific point in time. If abnormal votes are found, recomputation can be performed while retaining audit records.

    V. Anti-Abuse, Rate Limiting, and Auditing

    The openness of an infinite arena introduces adversarial risks. It is advisable to start from four layers of protection:

    1. Request layer: Apply rate limiting by session, user, and source. The voting API should use idempotency keys to prevent duplicate submissions.
    2. Behavior layer: Detect short-time high-frequency voting, fixed preferences, abnormal device features, and bulk session patterns, and trigger verification or downweighting.
    3. Data layer: Vote records must not be modified in place. Use append-only writes and audit logs; anomaly flags and manual review results need to be traceable.
    4. Rating layer: Isolate or downweight abnormal votes, and retain raw data and cleaning rule versions when recomputing ratings.

    Anti-abuse policies should be configurable and support canary rollout to avoid harming normal users. All rule changes should be versioned so that the causes of rating changes can be analyzed.

    VI. Observability and Continuous Operations

    After launch, the system needs to continuously track battle creation success rate, model invocation latency, vote conversion rate, tie rate, abnormal vote ratio, queue backlog, rating drift, and model version distribution. Monitoring and alerting should cover API error rates, message backlog, and rating computation latency. Log services should support retrieval by battle identifier, model version, and prompt identifier. Object storage should save raw response snapshots for replay and dispute handling.

    For operations, it is recommended to first canary new models with small traffic, fix evaluation windows and prompt distributions, and then gradually expand traffic. Set sampling limits and cost budgets for repeated evaluation tasks, and use caching and asynchronous batching to reduce invocation costs. Publish explanations of rating rule changes regularly and retain historical leaderboard snapshots.

    VII. Practical Recommendations

    First, define the evaluation goal: whether it is general conversation preference, coding ability, or multi-turn task completion. Different goals correspond to different prompt distributions and scoring dimensions. Second, all model versions, prompt versions, sampling parameters, and rating rules must be versioned. Third, repeated evaluation must be designed explicitly; idempotency keys and sample indexes are both indispensable. Fourth, leaderboards must include confidence intervals and sample sizes to avoid overinterpreting small score differences. Fifth, anti-abuse and auditing should be foundational capabilities, not after-the-fact patches. Sixth, general cloud capabilities such as object storage, message queues, container platforms, log services, and monitoring and alerting can be used to build scalable and observable evaluation infrastructure at relatively low cost.

    The value of an infinite arena is not in producing an absolute ranking, but in continuously and transparently accumulating pairwise comparison evidence. Only when fairness, reproducibility, repeated evaluation, anti-abuse protection, and observability are incorporated into the same engineering system can model evaluation move from “lively voting” to “trustworthy measurement.”