Evaluating Open-Source AI Agents in Full-Stack Development: Capabilities and Practices

I. Beyond Benchmarks: The Capability Divide of Open-Source Models

For the past two years, the consensus has been that open-source models lag a step behind their closed-source counterparts. However, recent iterations are overturning this narrative. In public evaluations covering comprehensive knowledge benchmarks, real-world software engineering tasks, and terminal environment coding, top-tier open-source models are approaching or even locally surpassing closed-source flagships. On comprehensive leaderboards from various third-party evaluation agencies, the open-source camp has overall entered the first tier.

While documenting this trend is valuable, for frontline developers, there is only one real question: setting aside the leaderboards, what can these models actually accomplish?

The reference value of benchmark tests is undeniable, but they have three unavoidable limitations:

1. Narrow Task Scope: Benchmark questions are mostly isolated, static snapshot tasks—a single call yielding a single answer. Real-world delivery is a long-chain workflow requiring the coordination of frontend, backend, and data layers.
2. Contamination Risk: High-difficulty questions often circulate widely within the community. It is difficult to completely separate genuine reasoning from memorization in their scores.
3. No Assessment of “Product Sense”: Almost no benchmark evaluates a system’s aesthetics, narrative structure, or interaction details, which are often the first thresholds determining whether a product will be accepted by users.

Therefore, a more reliable way to evaluate a coding Agent is through capability-oriented, identical-prompt testing. The tasks must be composite (covering multiple tech stacks), closely aligned with real business scenarios (incorporating business rules), and verifiable by both machine and human evaluation. The following section demonstrates this methodology using three product-level website building tasks.

II. Test Case 1: Orbital Travel Company Website with Booking Functionality

The first test requires the Agent to build a website for a fictional low-Earth orbit travel company. The core requirements are twofold: space capsule seat selection and flight booking. The prompt uses the following structure:

# Product-Level Website Testing Template
Role: Senior Full-Stack Engineer + Brand Designer
Business Background: {Fictional Brand} in {Fictional Industry}
Core Features:
  - {Core Business Flow: Seat Selection, Online Booking}
  - Data Persistence (Survives Service Restart)
Business Rules: {Capacity Limit per Flight, Auto-Lock when Full}
Delivery Constraints: One-Shot Generation, Locally Runnable, Complete Frontend & Backend

The tested Agent generated the result in one shot. First, let’s examine the frontend delivery quality:

• Brand Tone Established: The hero section features a full-screen video background simulating the perspective of a space capsule porthole. The Earth’s curvature slowly rotates, and the edge of the atmosphere glows blue, immediately establishing the “premium feel” expected in commercial aerospace.
• Atmospheric Details: The bottom right displays a flight data panel using a monospaced font. The top left features a small, fluorescent cyan dot that slowly pulses, simulating an online system status. These micro-interactions determine whether a page is a “template” or a “crafted work.”
• Structured Narrative: Scrolling down reveals a vertical timeline-driven flight narrative: from donning pressure suits at the spaceport 60 hours before launch, to enduring G-forces during the propulsion phase, entering the zero-gravity phase, and finally splashdown recovery. Each stage is bound to an orange key data point and scene imagery. Information is not flatly laid out but organized along a “mission timeline.”
• Authentic Interactive Assets: The cabin layout is an interactive SVG component where clicking selects a seat, rather than just being a decorative image.

At this point, many would describe the result as “stunning.” However, as an evaluator, one must ask the second-level question: Does it just look good, or is it functional?

III. The Divide: Backend Logic and Business Rules

The key to distinguishing a “demo” from a “work-in-progress” lies in the backend. In this test, the booking flow was a complete closed loop: selecting a date, choosing a seat, filling in passenger information, and submitting the booking, with data actually persisting to a database. Even more critical are two business details:

1. Booking data is persistently stored, remaining intact after a service restart.
2. Each flight has a 6-seat capacity limit, and seats automatically lock once full.

The second point is particularly worth elaborating on—it means the Agent did not merely generate standard CRUD interfaces but translated “capacity constraints” into executable code. An illustration is as follows:

def book_seat(flight_id: str, seat_no: str, passenger: dict):
    with db.transaction():                    # Transaction ensures concurrency safety
        flight = db.lock_flight(flight_id)   # Row-level lock to prevent concurrent overselling
        if len(flight.booked) >= flight.capacity:
            raise SoldOut(f"Flight {flight_id} is fully booked")  # Capacity rule
        if seat_no in flight.booked:
            raise Conflict("This seat has already been booked")  # Conflict detection
        db.save_booking(flight_id, seat_no, passenger)           # Persistent write
    return receipt(flight_id, seat_no, passenger)

The significance of this code lies not in its syntax, but in its rule awareness. A model that proactively handles overselling, duplicate seat selection, and concurrent race conditions provides an order-of-magnitude greater value in a production environment than a model that only writes bare APIs.

Methodological insight for evaluators: When designing test prompts, be sure to incorporate business constraints like “capacity limits” and “status locks.” This is the cheapest probe to distinguish demo-level output from production-level output—low cost, but immediately revealing upon testing.

IV. Cross-Domain Validation: Capability or Memorization?

A high score in a single domain cannot rule out the suspicion of “memorization”—there are already plenty of aerospace website templates in the training data. A truly effective method is cross-domain testing with identical prompts: keeping the prompt structure unchanged while only replacing the industry and brand.

Test Case 2: Deep-Sea Exploration Research Institute Website. The hero section’s camera movement sinks vertically from the sea surface. The moment it crosses the waterline, the screen splits in two, with blue gradient transitioning from shallow to deep. In the distant ink-blue depths, two warm yellow submarine lights shine. Parameter cards for three submersibles are precise down to propulsion methods and observation window dimensions. Six deep-sea creatures each occupy a full screen. Under the “Field Notes” section, three articles are signed by the Chief Scientist, Marine Biologist, and Documentary Director—roles, writing styles, and perspectives are mutually consistent.

Test Case 3: Luxury Perfume Brand Website. An amber perfume bottle rests on a marble countertop. The liquid inside slowly rotates, and light passing through the glass casts caustic textures. The brand wordmark is overlaid on the bottle in a high-contrast serif font—a minimalist, restrained visual language that fits the luxury grammar.

Three verticals—aerospace, deep sea, luxury—using the same prompt, generated in one shot, with their respective aesthetic systems established. This largely rules out the “memorized template” explanation: what the model has learned is not the style of a specific website, but the abstract concept of a “brand visual system,” and it can re-instantiate it according to industry grammar.

Evaluation insight: Cross-domain testing is the most effective way to expose memorization. A high score in one subject proves nothing. When three unrelated subjects all yield valid results, that is true capability.

V. Engineering Perspective: Required Capabilities Behind Full-Stack Delivery

Deconstructing from a principle level, a successful full-stack delivery requires the synergy of at least four capabilities:

1. Long-Range Planning: Decomposing “build a website” into a dependency sequence of Information Architecture -> Data Model -> API -> Frontend Components -> Persistence Scripts, with the correct order.
2. Tool Calling and Sandbox Execution: Creating files, installing dependencies, starting local services, and reading runtime logs—code is not merely “printed,” it is “executed.”
3. Integrated Frontend and Backend Generation: Static assets, interface layers, and database initialization are produced in the same batch. API fields naturally align with frontend calls, eliminating the manual step of piecing interfaces together.
4. Self-Verification Loop: Reading logs upon runtime errors, locating the issue, fixing it, and re-verifying. The reliability of a single-shot generation never depends on how much it gets right the first time, but on its recovery speed when errors occur.

The fourth point is the most easily underestimated. When evaluating an Agent, deliberately introduce minor obstacles in the environment (missing dependencies, port conflicts) and observe its recovery behavior. This is far more informative than watching it generate smoothly without issue.

VI. Enterprise Implementation: Five Recommendations from “Stunning” to “Usable”

1. Build a Custom Evaluation Set; Use Benchmarks Only for Initial Screening. Conduct identical-prompt tests using your team’s real business requirements. The test cases should cover at least five dimensions: visual system, interactive behavior, data persistence, business rules, and exception handling.
2. Standardize the Evaluation Checklist. For example: Visually, do colors, fonts, and spacing form a cohesive system rather than a pile of elements? Interaction-wise, are buttons bound to real behaviors, and are empty and error states handled? Data-wise, does data persist after restart, and are boundary values intercepted? Code-wise, is the layering clear, transferable, and maintainable?
3. Clarify Human-AI Division of Labor. Agents excel at prototyping, scaffolding, and repetitive coding; architectural decisions, security boundaries, and final quality control should still be handled by engineers. “AI generates, humans review architecture” is currently a more prudent collaboration model.
4. Implement Engineering Guardrails. AI-generated code must go through the exact same CI pipeline as human-written code: static scanning, dependency auditing, and test coverage gating. None can be omitted.
5. Choose the Right Runtime Infrastructure; Calculate Costs Before Scaling. For data-sensitive scenarios, prioritize private deployment. For cloud-based routes, you can rely on general cloud capabilities—such as platforms like Baidu AI Cloud, which provide container runtime environments, object storage, and monitoring/alerting basics—suitable for hosting the Agent’s sandbox execution and artifacts. Regardless of the route, it is recommended to first calculate the “comprehensive cost per task” before scaling up.

Conclusion

The convergence of the gap between open-source and closed-source models will ultimately land on a very pragmatic standard: facing the same business problem, can the model deliver a complete system that “runs, looks good, and is ready for production” within a single conversation. Benchmarks measure a model’s lower limit; real-world delivery determines its upper limit on the production line. Rather than debating leaderboard rankings, it is better to give your team three such test questions today—the answers will be more intuitive than any evaluation report.