Evaluating Open-Source AI Agents: From Benchmarks to Full-Stack Delivery

Over the past two years, the iteration speed of open-source large language models (LLMs) has been remarkably fast. According to public data from various third-party aggregate evaluation leaderboards, the performance gap between the open-source camp and closed-source flagships has generally narrowed. In engineering-oriented benchmarks such as software engineering and terminal task execution, open-source models occasionally even surpass their closed-source counterparts. However, leaderboards answer the question of “how much the model knows” but fail to answer “what the model can deliver.” For enterprise users and front-line developers, the real question remains: if you hand over a real-world task with business constraints, can it run successfully end-to-end in one go? This article addresses this issue by first clarifying the value boundaries of benchmark testing and then providing a reusable evaluation methodology based on three typical web development tasks.

1. Understanding Benchmarks: What Three Types of Tests Measure and Miss

Current mainstream evaluations are roughly divided into three categories, each with clear positioning and inherent blind spots.

1. Comprehensive Knowledge Benchmarks. Represented by extremely difficult expert-level exams covering hundreds of subjects, these test a model’s breadth of knowledge and deep reasoning capabilities. However, relying primarily on static question banks raises concerns about question leakage and data contamination. Furthermore, the test format is “Q&A” rather than “doing things,” meaning high scores do not necessarily correlate with practical application performance.

2. Software Engineering Practical Benchmarks. These require models to complete bug fixes or feature implementations within real code repositories, making them the closest evaluation to daily development. The limitation is that task granularity focuses on “single-point modifications,” measuring surgical patching capabilities rather than the ability to build a system from scratch.

3. Terminal Task Benchmarks. These involve completing multi-step tasks like environment configuration, test execution, and error troubleshooting within a command-line environment to assess tool invocation and error recovery. However, the fixed environment and limited set of steps still fall short of the open-ended nature of real engineering problems.

Summary: Benchmarks are suitable for initial screening to eliminate clearly unqualified candidates, but they are not suitable for final adjudication. Final selection must include supplementary testing with custom tasks structurally isomorphic to your own business. This is precisely the premise of the three web development tasks below, which cover the complete pipeline: frontend visuals, content generation, backend business logic, and data persistence.

2. Defining “Capable of Working”: A Four-Layer Capability Model for Full-Stack Delivery

Inputting a prompt and outputting a runnable, deliverable system—this is the watershed distinguishing an AI agent from a mere code completion tool. This watershed can be broken down into four layers:

• Layer 1: Requirements Understanding and Information Architecture. Given a one-sentence requirement, the model must extract page structures, content modules, and business rules, clearly defining the product’s user roles and the actions each role needs to perform.
• Layer 2: Frontend Visual and Interaction Implementation. Is the design system self-consistent? Do colors, fonts, spacing, and animations follow a unified set of rules, and is the interaction loop closed?
• Layer 3: Backend Business Logic. Are interfaces, parameter validations, and state transitions genuinely usable? The judgment criteria here is straightforward: is the system just “looking functional,” or is it “actually functional”?
• Layer 4: Data and Boundary Conditions. Is data truly persisted, are capacity limits effective, how are concurrency conflicts handled, and are exception paths properly caught?

The current industry reality is that a significant number of models stop at Layer 2. The page looks delightful, but opening the debugging tools reveals hardcoded mock data. Layers 3 and 4 are where true disparities emerge.

3. Task 1: Spaceflight Booking Site — Verifying Real Business Rule Implementation

Task Setup: Build an official website for a near-Earth orbit travel company, requiring space capsule seat selection and flight booking features.

What constitutes an excellent delivery? Let’s start with the atmosphere. Aerospace is serious engineering, and a website’s primary task is establishing trust, which stems from detail consistency: a full-screen video background from a capsule window perspective, the Earth’s curvature slowly rotating, the atmosphere’s edge glowing with a ring of blue light. A flight data panel in the corner uses monospaced fonts—a typographic choice naturally associated with “instruments.” A slowly breathing, blinking fluorescent status dot implies the system is online. These elements are individually simple, but combined, they point to a single semantics: this is an institution that genuinely sends people into space.

Next is narrative. A vertical timeline runs through the page: from donning custom pressure suits at the spaceport sixty hours before launch, to the G-force experience during orbital insertion, the moment of weightlessness, and finally the capsule splashing down into the ocean about ninety-two hours later with recovery divers gathering around. Each stage binds a key parameter with a scene image. The essence of this narrative is translating “process complexity” into “perceptible ritual,” helping users understand why this journey is expensive.

Finally, business logic—this is the real test. The booking system must be genuinely runnable: select dates, choose cabins, fill in passenger information, and write booking data to persistent storage. Each flight has a six-seat limit and automatically locks once full. During evaluation, it is recommended to ask two questions: First, which layer enforces the constraints? The correct answer should be at the database layer (constraints, transactions, row-level locks) or at least uniformly validated at the service layer, not just by hiding a button on the frontend, which easily breaks under direct API calls. Second, how does it handle concurrency? If two people grab the last seat simultaneously, does only one succeed?

Example pseudocode (correct implementation logic for full-seat locking):

function bookSeat(flightId, seatNo, passenger):
    beginTransaction()
    lockRows(flightId)                    # Row-level lock to prevent concurrent overselling
    if bookedCount(flightId) >= 6:        # Capacity constraint
        rollback(); throw SeatSoldOut
    insertBooking(flightId, seatNo, passenger)
    commit()

If the delivery is just an empty shell, this single question will expose it.

4. Task 2: Deep-Sea Research Institution Site — Verifying Content Structuring and Role-Playing

Task Setup: Using the same prompt structure, switch the theme to a deep-sea exploration research institution. This task focuses less on engineering and more on content-side capabilities.

Observation 1: Above-the-Fold Camera Movement. An excellent approach is a camera vertically sinking from the sea surface; the moment it pierces the waterline, the screen splits in two, blue gradients from shallow to deep, and two submarine lights shine in the distant, ink-blue depths. A single camera movement establishes the worldview, letting users understand what this institution does within three seconds.

Observation 2: Parameter Precision. Submarine introduction cards should include quantifiable fields like propulsion methods and observation window dimensions. The source of professionalism is “specifics”: “pressure depth of 4,500 meters” is far more convincing than “excellent performance.” If content generation merely stacks adjectives, the model is being perfunctory. True content capability means actively providing verifiable, quantified details and flagging uncertain items.

Observation 3: Role-Playing. Multiple articles in the research notes section should be signed by “authors” of different identities: the chief scientist writes technical assessments, the marine biologist writes species observations, and the documentary director writes field notes. The three voices should differ in tone while remaining factually consistent. The testing method is straightforward: swap the signatures and ask the model to self-check to see if it can identify the mismatch between writing style and identity. This examines whether the model maintains a consistent world setting rather than randomly generating paragraphs independently.

5. Task 3: Luxury Brand Aesthetics Site — Verifying Visual Syntax Control

Task Setup: An official website for a high-end perfume brand. This is the most aesthetically demanding task among the three.

Typical characteristics of excellent delivery: an amber perfume bottle stands on a marble countertop, the liquid inside slowly rotating, light penetrating the glass to cast caustic textures on the countertop. The physical feel of liquid and light is the soul of this visual. The brand name is overlaid on the bottle in a large serif font, with restrained letter spacing and distinct hierarchy.

The interface language of luxury brands is “less is more”: an animation lasting 200 milliseconds too long appears sluggish; a fraction less whitespace appears cheap. How do you determine if a model is “accidentally good-looking” or “genuinely possesses aesthetic syntax”? A practical technique: require it to attach a design decision document explaining why a serif font was chosen over sans-serif, why easing-out curves were used for animations, and what proportions the grid and whitespace follow. A model that can clearly articulate its decision basis has reproducible, iterable aesthetics; one that cannot is merely drawing lots with every subsequent generation.

6. Crystallizing the Method: A Reusable Evaluation Checklist

To turn the above observations into executable items, score each candidate model item by item:

1. Requirements Fidelity: Does the deliverable correspond one-to-one with the functional points in the prompt, without omissions or unauthorized additions?
2. Business Authenticity: Is data persisted to a database? Are caps, locks, and validations taking effect at the correct layer?
3. Visual Consistency: Do colors, fonts, spacing, and animations form a coherent system, or are they randomly collaged?
4. Content Reliability: What is the factual error rate? Does it proactively flag uncertain information? Are multiple role voices consistent?
5. Delivery Completeness: Does it include startup documentation, a dependency list, and a self-test report?
6. Maintainability: Is the code clearly layered? Are there large amounts of hardcoded magic numbers?

Supporting prompt design recommendations include three tips: Front-load and quantify business rules (“six seats per flight, locks when full” is far more effective than “supports booking”). Write acceptance criteria into the prompt, requiring the deliverable to include a self-test report. Adopt a one-generation-plus-one-iteration approach: look at the upper limit in the first round, and give targeted feedback in the second round to observe correction capabilities. Correction capability often predicts long-term user experience better than first-round performance.

7. Engineering Implementation: Three Mandatory Questions from Demo to Production

Three barriers separate a demo from production. If any is not crossed, a system generated by an agent dares not handle real traffic.

The first barrier is security. Generated code must pass dependency audits. Supply chain poisoning is an attack surface unique to agent scenarios; every third-party package introduced by the model is a potential intrusion entry point. The execution environment should be sandboxed and follow the principle of least privilege, with file system and network access governed by explicit whitelists.

The second barrier is architectural deployment. Models typically deliver monolithic applications suitable for demos but not for directly bearing production traffic. The common practice is to split components according to cloud product categories: stateless services are containerized and handed over to a container platform for scheduling; static resources sink into object storage and integrate with a CDN; transactional data like bookings go to a managed relational database; asynchronous tasks and notifications use message queues for peak shaving; and all uniformly integrate with logging services and monitoring alerts. Taking Baidu AI Cloud as an example, these capabilities all have corresponding general categories available. Developers can quickly migrate demo projects into operable services without rewriting business logic.

The third barrier is evaluation automation. Crystallize the checklist from Section 6 into regression test cases within the CI/CD pipeline. Every time the model upgrades a version, run a standard set of tasks. The fluctuation in delivery quality can then be quantified and traced. This is particularly crucial for teams relying long-term on open-source models—open source means version rhythms are beyond your control. Without regression evaluation, you are leaving stability to chance.

Conclusion

Benchmarks determine a model’s lower limit; delivery determines its upper limit. The true value of open-source models has never been just leaderboard rankings, but rather the three rights they return to users: data sovereignty via private deployment, customization freedom through deep fine-tuning, and freely combinable toolchains. The best way to evaluate them is not staring at leaderboards published by others, but picking the toughest nail from your own business and letting candidate models hammer it in one by one. The one that drives it in is the one worth betting on.