From Benchmarks to Delivery: Testing an Open-Source Agent’s Full-Stack Development Capabilities

1. Background: When Model Releases Are No Longer Just About Scores

A recent open-source model release came with impressive benchmark results: 54.0 on a high-difficulty comprehensive evaluation often described as “humanity’s last exam,” 58.6 on a software engineering benchmark, and 66.5 on terminal coding tasks — placing it near the top of the open-source leaderboard.

For developers working in production environments, however, benchmark scores are only one reference point. What actually determines whether a model can enter a real workflow is its ability to turn a requirement into something runnable and deliverable. This article sets leaderboards aside and asks a single question: what can this kind of agent actually build?

We tested the agent on three website projects with widely different requirements — commercial aerospace, a scientific research institution, and luxury consumer goods — evaluating each one across visual design, interaction logic, and backend data persistence to measure delivery completeness.

2. Case Study 1: Low Earth Orbit Travel Company Website (Full-Stack Delivery)

2.1 Requirements

The task was to build an official website for a low Earth orbit travel company. Core features included spacecraft seat selection, flight booking, passenger information entry, persistent storage of reservation data, a six-seat limit per flight, and automatic locking once a flight reaches capacity.

2.2 Visual and Narrative Layer

The first impression upon opening the page defines the brand tone. The agent’s solution began with a full-screen video background showing the view from a spacecraft window: the Earth’s curvature slowly rotating, with a blue glow along the atmosphere’s edge. A flight data panel in the lower right used monospace type to display key parameters. In the upper left, a small fluorescent cyan dot pulsed slowly, simulating a live system status indicator.

Scrolling down revealed a vertical timeline-driven flight narrative: from T-60:00, changing into a custom pressure suit at the spaceport, to T+03:00, enduring g-forces during the boost phase, to T+08:00, the onset of zero gravity, and finally T+92:00, splashdown with recovery divers surrounding the capsule. Each stage included an orange key metric and an AI-generated scene illustration.

Notably, a single prompt and a single generation run produced everything from the brand visual system to an interactive SVG seat map. This suggests the agent maintained consistent control over the design language throughout the generation process rather than assembling the page piece by piece.

2.3 Backend Logic Layer

The more critical validation was on the backend. The flight booking system was genuinely functional: selecting a date, choosing a seat, entering passenger information — all reservation data was persisted. Each flight had a six-seat cap, and full flights were automatically locked.

From an engineering perspective, this means the agent needed to handle several tasks:

• Data modeling: Designing the relationships among three entities — flights, seats, and reservations.
• State management: Coordinating seat occupancy status with flight capacity status.
• Concurrency safety: Handling conflicts when multiple users attempt to book the same seat simultaneously.
• Persistence: Storing and retrieving reservation data.

A common simplified implementation approach looks like this (illustrative pseudocode, not tied to a specific framework):

# Atomic validation for seat reservation
def reserve_seat(flight_id, seat_no, passenger):
    with transaction():
        seat = query_seat(flight_id, seat_no, lock=True)
        if seat.status != "available":
            return {"ok": False, "reason": "seat_taken"}
        if count_reserved(flight_id) >= 6:
            return {"ok": False, "reason": "flight_full"}
        seat.status = "reserved"
        insert_reservation(flight_id, seat_no, passenger)
        return {"ok": True}

This website was not a hollow mockup — it included complete backend logic. This is the most valuable finding of this evaluation: the agent’s delivery boundary has advanced from static pages to runnable full-stack applications.

3. Case Study 2: Deep-Sea Research Institution Website (Information Architecture)

Switching to a different subject matter while using the same prompt structure, the agent produced a website for a deep-sea exploration research institution.

The opening shot began on the surface of the North Atlantic, then plunged vertically underwater. As the camera passed through the waterline, the frame split in two, with blue deepening from light to dark. In the distant inky blue, two warm yellow submersible lights glowed. Specification cards for three submersibles included details down to propulsion method and observation window dimensions. Six deep-sea species each received a full screen. A field notes section featured three articles attributed to a chief scientist, a marine biologist, and a documentary filmmaker.

This case was not about visual spectacle but about information architecture and organizational capability. The core requirements for a research institution website are credibility and hierarchy: equipment specs must be precise, species introductions must be systematic, and bylines must reflect professional roles. Without additional instructions, the agent automatically produced a three-section structure — “Equipment — Species — Field Notes” — and matched each content type with an appropriate presentation format.

For developers, this means that when building content-focused sites, part of the “content categorization and presentation format” decision-making can be delegated to the agent, allowing humans to focus on fact-checking and brand review.

4. Case Study 3: Luxury Perfume Brand Website (Aesthetic Control)

The third case was a luxury perfume brand website — the agent’s strongest demonstration of refined aesthetics.

The scene showed an amber perfume bottle on a marble surface, the liquid inside slowly swirling, with light refracting through the glass to cast caustic patterns. The brand name appeared in a serif typeface overlaid on the bottle. The page had no unnecessary elements; the combination of materials, lighting, and typography produced a unified sense of luxury.

Aesthetic tasks challenge the agent because it must understand that restraint is itself a design strategy. Excessive animation or an overcrowded layout would destroy the sense of exclusivity that luxury goods require. The result showed strong consistency in font selection, material rendering, and use of negative space, without the multi-style inconsistency that often plagues generated designs.

5. Capability Breakdown: Three Layers of Full-Stack Delivery

Looking across the three cases, the agent’s capabilities can be broken down into three layers:

Layer 1: Visual system generation. Starting from brand positioning, the agent produces color schemes, typography, motion design, and layout specifications, maintaining consistency across pages. This is a baseline capability, but consistency control is the hard part.

Layer 2: Interaction and frontend engineering. This includes scroll-driven narratives, interactive SVG components, and responsive layouts. The interactive seat map in Case 1 demonstrates that the agent can handle stateful frontend components rather than purely static rendering.

Layer 3: Backend and data persistence. The booking system in Case 1 is the dividing line. The ability to generate runnable backend logic means the agent can take on prototype validation or even the initial build of small-scale production systems.

A simplified layered view:

Requirement → Brand visual system → Page structure & interaction → Business logic & data layer → Runnable deliverable
              (Design specs)          (Frontend engineering)           (Backend / persistence)

6. Practical Recommendations for Engineering Teams

Based on this evaluation, here are actionable suggestions for developers:

1. Use structured prompts. All three cases used the same prompt structure, indicating that structured requirement descriptions (brand positioning + core features + visual keywords + delivery criteria) are more stable than free-form descriptions. Build a reusable template and stick to it.

2. Front-load consistency checks. After generation, prioritize checking cross-page consistency in color, typography, and spacing — this is where the agent is most likely to drift.

3. Always manually review backend logic. Concurrency control, edge conditions (such as capacity locking), and data consistency still require developer verification. Never assume generated code is correct by default.

4. Distinguish between prototype and production. What the agent delivers is well-suited as a high-fidelity prototype or MVP. Before production, you still need to add authentication, logging, monitoring, and load testing.

5. Minimize intervention on aesthetic tasks. When brand positioning is clear, piling on additional instructions tends to break stylistic unity. Filter and select outputs rather than making line-by-line modification requests.

7. Conclusion

Setting aside benchmark scores, the conclusion from this evaluation is clear: the open-source agent’s capability boundary has advanced from “generating code snippets” to “delivering runnable full-stack websites,” with usable quality in visual consistency and information architecture. For developers, it shortens the path from idea to prototype. For organizations, it lowers the cost of validation-stage experimentation. The real value is not in how many points it scored, but in its ability to turn a prompt into something that actually runs — and that has already happened.