Beyond Benchmarks: Evaluating the Full-Stack Delivery Capabilities of AI Agents

I. Why We Must Look Beyond Benchmarks for Real-World Delivery

Whenever a new large language model is released, it is accompanied by impressive benchmark numbers: high scores on complex knowledge evaluations, leading performance on software engineering tasks, dominance in terminal coding benchmarks, and jumps in third-party leaderboards. While these numbers have reference value, they present three unavoidable blind spots for frontline developers:

1. Data Contamination Risk: Training data may overlap with public evaluation sets. Leaderboard scores might overestimate true capabilities and do not equate to performance on completely unseen tasks.
2. Static Evaluation vs. Dynamic Tasks: Single-turn Q&A tests measure knowledge recall, whereas real-world development is multi-turn. It requires tool calling, log reading, self-correction, and long-range planning—capabilities that traditional leaderboards rarely measure.
3. Disconnect Between Scores and Business Metrics: No matter how high the score, it does not guarantee that a model can successfully run a booking system with hard business constraints (e.g., seat capacity limits and automatic locking when full) in one go.

Therefore, a more pragmatic evaluation method is to change the metaphor: treat the model as a “newly hired full-stack engineer.” Assign it a complete, real-world task with hard constraints, and then verify three things: Can the deliverable run directly? Are constraints strictly followed? Do the visual and interactive elements meet production standards? This is the essence of the “AI Agent full-stack delivery” discussion: setting aside benchmark scores and focusing solely on what the model can actually build.

II. What Happens Behind a One-Shot Full-Stack App Generation

The concept of “one prompt, one generation, full-stack delivery” can be technically deconstructed into a six-stage pipeline, where each stage serves as an independent evaluation point:

1. Requirement Structuring: Parsing natural language requirements into page structures, interaction lists, data models, and business rules. The more complex the requirement, the more critical the information compression capability at this stage.
2. Architectural Decision Making: Choosing the tech stack, directory organization, and persistence solution (in-memory, file, embedded database, or cloud-hosted database), while anticipating concurrency and edge-case issues.
3. Frontend Implementation: Layout systems, animation orchestration, state management, and interactive components (e.g., SVG cabin schematics).
4. Backend Implementation: API design, business validation, and transaction/boundary handling.
5. Self-Validation: The Agent loop of running, reading logs, identifying errors, and fixing retries.
6. Delivery Packaging: A runnable project with clear startup instructions.

Meeting both stages 3 and 4 simultaneously is the watershed between a “good-looking demo” and a “functional system.” Many generative site-building solutions only address the former: the page looks stunning, but clicking a booking button exposes its hollow nature. The true threshold for full-stack delivery lies precisely in the backend logic that users cannot see.

III. A Neutral Review of Three Typical Scenarios

Using three highly discussed task scenarios as examples (abstracted and not tied to any specific product), we can illustrate the expected standard of “full-stack delivery.”

Scenario 1: Commercial Spaceflight Booking Website

Task requirements: Brand official website + space capsule seat selection + flight booking. A qualified deliverable should present three layers of capability:

• Visual Layer: A full-screen video background simulating a porthole view with the Earth’s curvature slowly rotating; a monospaced flight parameter panel in the bottom right corner; a slow breathing-blink status dot in fluorescent cyan in the top left. The page uses a vertical timeline to drive the narrative, from pre-launch suiting up, propulsion overload, and zero gravity, all the way to splashdown recovery, with a key data point for each stage. The overall aesthetic should approach the standard of a mature commercial spaceflight company’s official website.
• Interaction Layer: Clickable SVG cabin schematic, date selection, and passenger information form, forming a complete closed loop.
• Data Layer: Real persistent storage of booking data; a hard limit of 6 seats per flight; automatic locking when fully booked to prevent overselling.

The key lies in the final layer: capacity constraints and status locking are typical backend business logic. If the model only generates static pages, this stage will immediately expose its “empty shell” nature.

Scenario 2: Deep-Sea Research Institute Website

Migrating the same prompt structure to a deep-sea theme tests the model’s generalization and information architecture restructuring capabilities. The opening camera movement sinks vertically from the sea surface, splitting the screen the moment it crosses the waterline. The blue gradient transitions from shallow to deep, with distant highlights simulating submersible lights. The parameter cards for three submersibles detail propulsion methods and observation window sizes. The field notes section features signatures from three roles: Chief Scientist, Marine Biologist, and Documentary Director. This demonstrates that the model doesn’t just reuse an engineering skeleton; it can reorganize content tone and narrative rhythm according to the subject matter.

Scenario 3: High-End Consumer Brand Website

Perfume brand requirements best test “extreme aesthetics”: amber-colored bottles, slowly rotating liquid, caustic patterns formed by light penetrating glass, and a classic serif font brand logo. Such tasks lack clear functional acceptance criteria and test the model’s implicit aesthetics regarding typography, light/shadow, and negative space. This is precisely what traditional template-based site builders struggle to cover.

IV. Engineering Practices: How to Reliably Achieve High-Quality Delivery

1. Replace One-Sentence Requirements with Structured Prompts

Role: Senior Full-Stack Engineer + Brand Visual Designer
Task: Build an official website and booking system for a near-Earth orbit tourism company
Hard Constraints:
- Hero Section: Full-screen video background + monospaced data panel + online status indicator
- Narrative: Vertical timeline, from T-60 minutes to splashdown recovery, one key data point per stage
- Interaction: SVG cabin map, date selection, passenger info form
- Backend: Booking persistence; 6-seat limit per flight; auto-lock when full
Deliverable: Directly runnable project with startup instructions

The key is front-loading constraints, quantifying them, and making them verifiable. Rules like “auto-lock when full” are unlikely to be implemented spontaneously by the model if not explicitly stated. Once written down, they become an automatically verifiable acceptance criterion.

2. Focus on Reviewing Data Models and Concurrency Boundaries

The core of a booking system is the transactionality of “locking a seat.” The schematic logic is as follows:

def book_seat(flight_id, seat_no, passenger):
    with db.transaction():
        flight = lock_flight(flight_id)      # Row-level lock to prevent concurrent overselling
        if flight.locked:
            raise BizError("Flight is fully booked")
        seat = lock_seat(flight_id, seat_no)
        if seat.status != "FREE":
            raise BizError("Seat is already booked")
        seat.update(status="HELD", passenger=passenger)
        if count_occupied(flight_id) >= flight.capacity:
            flight.update(locked=True)       # Auto-lock upon reaching capacity limit
    return {"status": "CONFIRMED"}

When reviewing the Agent’s output, focus on whether it spontaneously handles concurrency races, duplicate bookings, and boundary conditions. This is the litmus test for distinguishing “template memorization” from “engineering competence.”

3. Let the Agent Loop Run It for You

Mature Agent execution frameworks generally support an automated “generate-run-read logs-fix” loop. Practical advice: write the startup commands, health checks, and self-test assertions for core APIs directly into the task description. Let the model validate itself, which will significantly improve the first-pass success rate. Developers only need to perform the final manual acceptance review.

V. Build Your Own Evaluation System

If you need to batch-evaluate the delivery capabilities of different models or versions, it is recommended to construct a fixed set of tasks and quantify scores across five dimensions:

1. Runnability: Out-of-the-box execution capability and the number of startup errors.
2. Functional Completeness: Whether core use cases form a closed loop (e.g., completing a full booking process).
3. Constraint Compliance Rate: The proportion of predefined business rules correctly implemented.
4. Visual and Interaction Quality: Design consistency and animation completeness, which can utilize human blind evaluations.
5. Code Quality: Module separation, error handling, comments, and maintainability.

This evaluation pipeline can be fully automated: use a container platform to isolate the runtime environment for each generation, use a logging service to collect execution errors, use monitoring and alerting to track task success rates, and use object storage to archive deliverables for regression comparison. On Baidu AI Cloud, these general capabilities (container platform, cloud database, object storage, logging service, monitoring and alerting) can be directly combined into an evaluation pipeline without building infrastructure from scratch, turning “scoring” into a sustainable routine engineering process.

VI. Conclusion

Benchmark scores are an “entry ticket,” but delivery capability is the “final exam.” For developers, rather than chasing scores, it is better to build an evaluation and implementation methodology based on “real tasks + structured constraints + automated validation.” On one hand, this objectively filters models; on the other, it reverses the optimization direction for prompt design and engineering pipelines. When “delivering a runnable system from a single prompt” becomes a reproducible daily workflow, large models truly transition from demo toys to productivity tools.