Building AI Agent Evaluation Benchmarks for Real-World Work Scenarios

    Most benchmarks score large models and agents on a single dimension—algorithm accuracy, function-level code completion, or standardized Q&A—yet these "too clean" setups fail to capture how agents perform in real engineering environments. This article explains how to build an agent evaluation benchmark grounded in real work. It covers two dataset principles (anti-contamination and broad coverage), a unified scheduling and sandbox framework, and domain-specific scoring strategies for code, web/frontend, office, and security tasks. It also details anti-cheating measures and how to interpret results, followed by practical guidance for teams on sampling real tasks, prioritizing deterministic scoring, adding evidence-grounded model review, and treating evaluation as continuous engineering.

    Why Agent Evaluation Needs Real-World Work Scenarios

    Over the past few years, most industry evaluations of large models and agents have concentrated on a single dimension: the accuracy of algorithm problems, the pass rate of function-level code completion, or the accuracy on a handful of standardized Q&A sets. The biggest problem with these evaluations is that they are “too clean”—tasks are highly structured, contexts are short, and verification methods are narrow. As a result, scores can be high while still failing to reflect an agent’s overall capability in a real engineering environment.

    Real software development and office work have several distinct characteristics: cross-file, cross-module, long context, mixed constraints, and diverse deliverables. A real software requirement may ask an agent to locate related logic across dozens of files, modify a public interface, update all callers accordingly, and finally ensure that existing test cases still pass. Tasks like these don’t test “whether you can write a particular piece of code” but “whether you can get the work done in an unfamiliar codebase like a reliable colleague.”

    A valuable agent evaluation benchmark therefore needs to answer three questions at once: Is the dataset realistic and hard to bypass through “memorizing answers”? Can tasks across different domains be scheduled and run in a unified way? Are the scoring standards for each task type objective, reproducible, and hard to game? The sections below address these three points.

    Two Principles for Dataset Construction

    Anti-Contamination: Preventing Models from Scoring on Memory Rather Than Ability

    As public code repositories and technical Q&A communities have been used at massive scale for pretraining, the risk of evaluation sets being “contaminated” is very high. After seeing large numbers of similar bug-fix samples, a model may achieve high scores through pattern memorization rather than genuine reasoning.

    Effective anti-contamination measures include:

    • Rewriting task descriptions in a conversational style: Rephrase the original neat, template-like requirements into natural language that reads like a real person’s request, complete with background context and colloquial habits. This way, even if the underlying code snippets appeared in training data, the model can’t simply “match them up.”
    • Constructing novel composite tasks: Combine multiple real scenarios into cross-file, cross-module engineering requirements, so the answer can’t be found as a whole in the training corpus.
    • Keeping validation logic hidden: Information the scoring depends on is not exposed through the task description, preventing models from “cramming for the test” by guessing the scoring points.

    The goal of anti-contamination is not to make the questions tricky, but to make the scores genuinely reflect generalization ability.

    Broad Coverage: From “Fixing a Bug” to the “Complete Workflow”

    Testing only “fix one defect” is far from enough. A realistic benchmark should cover the full lifecycle: building from scratch, bug fixing, feature extension, code review, test generation, and documentation conversion. The originators of tasks should also be diverse—developers, algorithm engineers, product managers, test engineers, and operations engineers all have very different expectations for the same system. Having requests initiated by multiple roles makes the evaluation closer to real collaboration scenarios.

    A Unified Evaluation and Scheduling Framework

    The most common pitfall in multi-domain evaluation is writing a separate runtime and scoring logic for each domain, which ends up making cross-comparison impossible and maintenance difficult. A workable approach is to define a unified, standardized task directory structure and sandbox protocol for all tasks:

    • Standardized task directory: Each task contains a fixed structure—inputs (initial environment, requirement description), constraints, validation script mount points, and so on—so the scheduler doesn’t need special cases for different tasks.
    • Unified sandbox protocol: All agents run in the same controlled environment, with clearly defined permission boundaries for file read/write, command execution, and network access. The sandbox is responsible for initializing the environment, injecting the task, collecting outputs, running scoring, and reclaiming resources.

    The value of a unified protocol is that it “measures all tasks with the same ruler,” giving results across domains and models both comparability and reproducibility. In practice, these sandbox and scheduling capabilities can typically be built on general cloud infrastructure—for example, using container platforms to host isolated environments, object storage to manage task datasets and artifacts, and logging and monitoring/alerting services to track the evaluation process and locate failures.

    Differentiated Scoring Strategies Across Domains

    The nature of different tasks means a single scoring method won’t do. The core idea is: use deterministic rules for the parts that can be objectively and programmatically verified; introduce model review for the parts involving semantics, visuals, and interaction quality; and, for high-stakes domains like security, aim for fully deterministic scoring.

    Code Tasks: Hidden Unit Tests as the Primary Criterion

    The most reliable criterion for code tasks is the pass rate of hidden unit tests. The agent’s submitted code is run inside a real repository, and whether the tests pass is an objective fact that’s almost impossible to fake. Scoring centers on the test pass rate, while model review serves only as a diagnostic reference (for example, hinting at “what kind of semantics the change broke”) and does not contribute to the final score. These tasks typically cover a large number of real repositories and multiple engineering categories, with requirements differentiated by role.

    Web/Frontend Tasks: Three-Layer Combined Scoring

    Frontend and GUI tasks are hard to judge with unit tests alone, because “does it look good” and “is the interaction correct” are subjective. A three-layer combination can be used:

    1. Rule checks: Verify deterministic constraints, such as whether elements exist, whether styles comply, and whether data is rendered correctly.
    2. LLM/VLM judge: A review model with language and visual understanding assesses semantic soundness and visual quality.
    3. Agent judge: An agent actually operates the page to verify interaction flows, state transitions, and whether the workflow runs end to end.

    The three layers progress from “structure—semantics—behavior,” covering both static and dynamic dimensions.

    Office Tasks: Deterministic Rules + Evidence-Grounded Model Review

    Office scenarios usually involve mixed-format files (spreadsheets, documents, presentations, structured data, markup text, and so on) and require consistency across files, updated state, and usable deliverables. Scoring uses a dual track:

    • Deterministic rule checks: Verify objective facts, such as whether numerical calculations are correct, whether fields are complete, and whether formatting complies.
    • Evidence-grounded judge: Review conclusions must cite specific locations in the output as evidence, reducing the risk of the review model “assigning scores out of thin air.”

    The two tracks are fused by task weight, balancing both “correctness” and “quality.”

    Security Tasks: Fully Deterministic Scoring

    Security tasks (vulnerability discovery and exploitation, malware analysis, security operations, and evaluation of the agents’ own security) demand very high reliability, so no model review is used—everything is decided by deterministic scoring scripts. Typical criteria include: whether a valid proof of concept is produced or a target marker is hit, whether the correct threat indicators are identified, and whether detection rules and conclusions are accurate. Classification metrics and ranking correlation metrics can also help measure quality. The security domain must further include multi-layered anti-cheating mechanisms to prevent agents from “copying answers” by reading scoring logic or environmental traces.

    Anti-Cheating and Result Interpretation

    The more a benchmark relies on automated scoring, the more it must guard against “test-taking” optimization. Common cheating paths and countermeasures include:

    • Reading the scoring logic: Isolate scoring scripts and validation data from the agent’s visible scope, loading them only during the sandbox’s controlled phase.
    • Hard-coding against scoring points: Use hidden tests and composite tasks so answers can’t be simply enumerated.
    • Exploiting environmental traces: Clean up or block usable metadata, and detect abnormal file-access behavior.

    Results should be interpreted rationally. Evaluation tends to reveal clear capability tiers: a few top-tier models form the first group, general-purpose large models and domestic models follow closely and are catching up quickly, and gaps between tiers often show up in cross-module engineering ability and stability across multi-turn interactions. A single aggregate score doesn’t tell the whole story—it’s better to combine per-subset breakdowns and observe each model’s strengths and weaknesses across code, frontend, office, and security dimensions in order to make an accurate selection decision.

    Practical Guidance for Teams

    1. Define real tasks first, then discuss scoring: Sampling tasks from your team’s actual workflows ensures representativeness better than designing them out of thin air.
    2. Prioritize deterministic scoring: Whenever a part can be programmatically verified, use unit tests or rule checks to reduce subjective bias.
    3. Use “review + evidence citation” for subjective dimensions: When introducing model review, require it to provide supporting evidence, and keep a human spot-check step.
    4. Treat evaluation as continuous engineering: The task set needs to evolve with technology, so the evaluation platform should support dataset versioning, reproducible runs, and result archiving.
    5. Take anti-cheating and reproducibility seriously: These two factors determine whether evaluation results can be trusted over the long term.

    Overall, evaluation oriented toward real-world work scenarios is still fundamentally a test of a code agent’s core capabilities, but by adding real cross-file, cross-module, and cross-format tasks, it substantially enriches the evaluation dimensions and makes scores much closer to the agent’s actual usability. For teams looking to build their own evaluation systems, a unified scheduling framework, differentiated scoring strategies, and rigorous anti-cheating mechanisms are the three infrastructure directions most worth investing in.