Beyond Benchmarks: Evaluating Full-Stack AI Agent Development Capabilities

I. Why Practical Capability Matters More Than Benchmarks

In recent years, LLM scores across various public evaluations—comprehensive knowledge, software engineering, terminal operations, and third-party leaderboards—have steadily climbed. Developers are often surrounded by these numbers when selecting a model. However, benchmark testing has inherent limitations:

1. Narrow Task Distribution: Most evaluations focus on a closed “given problem - output answer” format, whereas real-world development is an open process of “given vague requirements - produce a complete system.”
2. Potential Data Contamination: Public evaluation sets might have appeared in the training corpus, leading to inflated scores.
3. Lack of Product Perspective: Scores cannot reflect the overall performance of the generated results in terms of visual aesthetics, interaction fluency, and business logic integrity.

Therefore, a more intuitive testing method for developers is to give an Agent a natural language requirement and see if it can deliver a complete, usable application in one go. This article follows this approach to deconstruct the true capability boundaries of Agents in full-stack development.

II. Four Dimensions for Evaluating Agent Coding Capabilities

Before hands-on testing, we can establish an evaluation framework:

• Comprehensive Knowledge Depth: Faced with interdisciplinary and highly difficult open-ended questions, does the model have sufficient knowledge reserves and reasoning capabilities? This determines whether it can correctly understand and present cross-domain concepts like “overload,” “caustics,” and “weightless environments.”
• Software Engineering Practice: Can it locate problems, modify code, and pass tests in codebases approaching real-world repository sizes? This tests long-context understanding and code editing capabilities.
• Terminal and Environment Operation: Can it autonomously execute commands, install dependencies, run tests, and iteratively fix bugs based on error logs? This is a prerequisite for Agent autonomy.
• End-to-End Product Delivery: From frontend visuals to backend logic and data persistence, can it deliver a complete, runnable product?

The first three dimensions have relatively mature evaluation paradigms. The fourth is closest to real-world scenarios and tests the model’s comprehensive capabilities the most.

III. Case Study 1: Generating a Complete Brand Website from a Single Prompt

3.1 Task Definition

The Agent was given a requirement: create a website for a fictional low-Earth orbit travel company, including spacecraft seat selection and flight booking functionalities. This is a typical compound task requiring “high aesthetic threshold + strong functional requirements.”

3.2 Visual System Construction

Notable design details in the generated results include:

• Atmosphere Creation: A full-screen video from the perspective of a spacecraft porthole serves as the background, with the Earth’s arc rotating slowly and the atmosphere’s edge emitting a blue glow. The first screen immediately establishes the “commercial spaceflight” brand tone.
• Data Texture: The flight data panel in the bottom right corner uses a monospace font to list key parameters. The top left corner features a fluorescent status dot with a slow breathing blink to simulate system online status—these details elevate the page from a “template feel” to a “product feel.”
• Narrative-Driven: The page organizes the entire flight process using a vertical timeline, from pre-launch preparations, overload during ascent, weightlessness after entering orbit, to reentry and sea recovery. Each stage is paired with a key data point and scene image, turning “selling a ticket” into “telling a journey.”

This tests three model capabilities: understanding design language (color, font, animation density), cross-file consistency constraint propagation (site-wide unified visual variables), and the ability to translate abstract brand tones into concrete styling rules.

3.3 Interactive Components

Seat selection is not a static image, but an interactive SVG cabin diagram: click a seat, view its status, and enter the booking process. Scroll-driven transition animations, sectional gradients, and data cards are all implemented in code, indicating that the model can generate clearly structured, maintainable frontend engineering rather than a “big ball of mud” with inline styles.

IV. Case Study 2: Backend Logic That Actually Runs

4.1 Requirements and Implementation

Beyond the website, the Agent must also deliver a flight booking backend: select dates, choose seats, fill in passenger information, persistently store booking data, and enforce a seat limit per flight with automatic locking when full. This means the model must autonomously complete:

1. Data Modeling: Entity relationships between flights, seats, passengers, and orders.
2. State Machine Design: The transition of seats between “available / locked / sold”.
3. Concurrency Correctness: Preventing overselling when two users attempt to book the same seat simultaneously.

4.2 Key Technical Point: Concurrency Control for Seat Locking

Locking when full seems simple, but correct implementation requires transactions and row-level locks (or optimistic locking with version numbers). An example is as follows:

BEGIN;
SELECT status FROM seats
 WHERE flight_id = $1 AND seat_no = $2
   AND status = 'available'
 FOR UPDATE;
UPDATE seats
   SET status = 'locked'
 WHERE flight_id = $1 AND seat_no = $2
   AND status = 'available';
COMMIT;

If the UPDATE affects zero rows, it means the seat has been taken by a concurrent request, requiring a failure response and prompting the user to reselect. Handling such edge cases is the watershed that separates “generating a page” from “delivering a system.”

4.3 Persistence and Deployment

A demo-level implementation can use a lightweight file database to run quickly; moving to production requires replacing it with a cloud database service, hosting static resources in object storage, deploying the application on a container platform, and integrating logging services with monitoring alerts. This migration path “from demo to production” should become a standard procedure when using Agents to generate code.

V. Case Study 3: Theme Migration and Aesthetic Control

To verify capability reusability, we can maintain the same prompt structure and only change the topic:

• Deep Sea Exploration Theme: The first screen sinks vertically from the sea surface, splitting the screen in two the moment it crosses the waterline. The blue deepens gradually, and two submarine lights illuminate in the distance. Parameter cards for multiple submarines are precise down to propulsion methods and observation window sizes; various deep-sea creatures each occupy a full screen; the field notes section is signed by authors of different identities. The model demonstrated proficiency in the “immersive scrolling narrative” paradigm.
• High-End Perfume Brand Theme: An amber bottle slowly rotates on a marble countertop, light penetrating the glass forms caustic patterns, and the brand name is overprinted on the bottle in a classic serif font. The minimalist layout, restrained animations, and sophisticated material lighting demonstrate the model’s adjustable control over different “aesthetic levels.”

This leads to an important insight: aesthetics is not an occult concept, but an engineering parameter that can be explicitly constrained by prompts—color systems, font temperament, animation intensity, whitespace density, and lighting style. The more specific the description, the more stable the output.

VI. Reusable Prompt Structure

Synthesizing the above cases, a prompt template suitable for “generating a complete site in one go” is as follows:

Please build a website and booking system for a fictional brand in the [Theme/Industry]:
I. Visual Tone
- Color and Material: ...
- Font Temperament: ...
- Animation Intensity: ...
II. First Screen and Narrative
- First Screen Atmosphere: ...
- Content Sections: ... (organized by timeline / cards / columns)
III. Functional Requirements
- Interactive Components: ... (e.g., cabin selection map)
- Backend Logic: ... (booking, validation, persistence)
- Business Rules: ... (e.g., 6-seat limit per flight, automatic locking when full)
IV. Delivery Boundaries
- Tech Stack and File Organization: ...
- Include startup instructions to ensure local runnability

Four key points: front-load delivery boundaries, specify sensory descriptions, explicitly declare business rules, and clarify state and persistence requirements.

VII. A Rational Perspective: Limitations and Engineering Recommendations

The full-stack delivery capability of Agents is impressive, but it is still far from “unattended production development”:

1. Security Cannot Be Omitted: Common missing elements in generated code include input validation, authentication, and anti-duplicate submission protections. These must be manually completed before going live.
2. Hallucination Risks: Details such as API signatures and dependency versions need to be verified one by one.
3. Architectural Evolvability: One-time generated code performs well at a small scale, but under complex business requirements, it still requires manual refactoring and modular governance.
4. Test Debt: Tests generated by models often only cover the main paths; edge cases need to be supplemented by engineers.

A reasonable positioning is to treat the Agent as a “high-speed draft generator”: it compresses the prototyping cycle from weeks to hours, while the engineer’s value shifts towards requirement decomposition, architectural gating, and quality assurance. When deploying to the cloud, you can leverage general cloud service capabilities (such as those provided by Baidu AI Cloud or similar cloud providers) like object storage, cloud databases, container platforms, and logging and monitoring to quickly set up the runtime environment, focusing your energy on core business logic.

VIII. Conclusion

Benchmark scores answer “how smart the model is,” while practical delivery answers “how useful the model is.” When a single prompt can yield a full-stack application with online visuals, complete logic, and persistent data, the workflow of software development is being redefined. For developers, the real question is no longer “can the Agent do the work,” but “how to design tasks well, constrain boundaries, and ensure quality control,” thereby stably transforming generative capabilities into delivery capabilities.