Evaluating Open-Source Coding Agents: From Benchmark Scores to Real Full-Stack Delivery

In the past two years, major model vendors have accelerated their release cycles, with late-night launches and immediate open-sourcing becoming increasingly common. Public benchmarks show that new open-source models now perform very close to leading closed-source models on several mainstream evaluation categories, and in some cases even surpass them:

• General knowledge benchmarks: interdisciplinary, high-difficulty question sets that probe knowledge depth and complex reasoning.
• Software engineering benchmarks: locating defects, fixing issues, and completing feature development in real repositories.
• Terminal environment coding benchmarks: completing multi-step tasks in constrained command-line environments.
• Third-party aggregate leaderboards: cross-comparing code, math, reasoning, and agent performance.

This trend reflects a simple fact: on leaderboards alone, the gap between open-source and closed-source models is narrowing rapidly. But for developers, the real question has never been “What score does this model get?” It is “How much work can this model actually do for me?” Benchmark scores are useful for initial screening, but task-based evaluation is what drives real decisions.

What benchmark scores can and cannot tell you

The value of benchmarks is not in question: they provide horizontal comparability and reproducibility, and they offer a rough picture of capability boundaries. But they have at least three limitations:

1. Static distribution. Benchmark questions are fixed. As training data coverage improves, the correlation between leaderboard scores and real-world capability weakens, and there is a risk of overfitting to specific test sets.
2. Disconnected workflows. Real engineering is not a multiple-choice problem. Requirements analysis, technology selection, project structure, dependency management, frontend-backend integration, data persistence, and edge-case handling all matter. A break in any link makes the output unusable. Benchmarks rarely cover these long-horizon tasks.
3. Missing experience quality. There is a large distance between code that “runs” and a product that is “usable, polished, and believable” — and the latter is exactly what business stakeholders care about most.

A more pragmatic evaluation approach is therefore to give an agent a single natural language requirement and ask it to deliver a complete product with a backend in one pass, then assess the output across multiple dimensions.

What “full-stack delivery” means

To judge a coding agent’s maturity, ask three progressively deeper questions:

1. Does it run? The frontend renders correctly, the backend starts successfully, and the APIs are connected.
2. Does it feel real? The visual system is complete, the information architecture is sound, and the interaction details are polished enough to create the impression of a genuine product.
3. Is the business logic closed-loop? Data is actually persisted, and business rules such as capacity limits, state transitions, and exception paths actually work.

Most models can only reach the first level. A smaller number can reach the second. Only those that can consistently achieve the third level demonstrate true full-stack delivery capability.

Experiment 1: Generating a commercial website from a single prompt

The task was deliberately straightforward: create a website for a fictional low Earth orbit travel company, with seat selection for a space capsule and flight booking functionality. The evaluation focused on four dimensions.

1. Visual system: where a premium brand feel comes from

A strong output establishes the commercial spaceflight atmosphere immediately: a full-screen video background from the perspective of a capsule window, with Earth’s curved horizon rotating slowly and a thin blue glow at the edge of the atmosphere. In the lower right, a flight data panel rendered in monospace font lists key parameters such as orbital altitude and velocity. In the upper left, a small fluorescent cyan dot pulses in a breathing rhythm, suggesting the system is live.

None of these details are technically complex on their own, but together they communicate three things: brand tone, a sense of systemness, and a feeling of life. When evaluating, ask: Is the color palette restrained? Do animations have rhythm rather than flashing everywhere? Does the typography establish a clear hierarchy?

2. Narrative structure: telling a complete product story through a timeline

The main body of the page is a vertical timeline that drives the flight narrative: from T-60:00, when passengers change into custom pressure suits at the spaceport, to T+03:00, when the propulsion phase imposes g-forces, to T+08:00, when zero gravity begins, and finally T+92:00, when the capsule splashes down and recovery divers approach. Each phase includes a highlighted key data point and a scene image.

This reflects product design thinking rather than template stitching. It shows that the agent understands that a website tells a story for a brand and can translate an abstract requirement into information architecture.

3. Component interaction: SVG seat map with booking integration

The seat selection is rendered as an interactive SVG. Clicking a seat updates the booking form. The completeness of interactive components directly determines how well the output scores on the “does it feel real” test.

4. Backend logic: is the booking system actually closed-loop?

This is the most critical layer. The way to verify it is not by counting lines of generated code but by walking through the actual flow — selecting a date, choosing a seat, filling in passenger information, and submitting the reservation — then checking three things:

• Persistence: After restarting the service, do the reservations still exist?
• Capacity control: Is each flight strictly limited to 6 seats?
• Full-flight locking: On the 7th submission, does the system correctly reject the booking and lock the flight?

A simplified booking lock logic might look like this:

class BookingService:
    SEATS_PER_FLIGHT = 6
    def book(self, flight_id, seat_no, passenger):
        flight = self.repo.get_or_create(flight_id)
        if flight.locked:
            return BookingResult(False, reason="flight is fully booked")
        if seat_no in flight.occupied:
            return BookingResult(False, reason="seat is already taken")
        flight.occupied.add(seat_no)
        if len(flight.occupied) >= self.SEATS_PER_FLIGHT:
            flight.locked = True          # lock automatically when full
        self.repo.save(flight)             # persist to disk so data survives restart
        return BookingResult(True, seat=seat_no)

You can go further with concurrency tests: when multiple requests compete for the last seat, does overselling occur? Edge cases like this are the fastest way to expose how well an agent handles engineering details. A hollow demo site will fail this test immediately.

Experiment 2: Theme transfer as a test of style generalization

A single impressive case might be luck. A stricter test is to keep the prompt structure unchanged, change the subject matter, and see whether the agent can transfer the same methodology to a different context.

• Deep-sea research institute website: The hero shot descends vertically from the ocean surface. At the moment of crossing the waterline, the screen splits in two, with the water color shifting from light to dark. In the deep blue distance, two warm yellow lights from a submersible glow faintly. The submersible parameter cards include details as specific as propulsion method and observation window dimensions. In the “Field Notes” section, three articles are attributed to a chief scientist, a marine biologist, and a documentary director — character-driven content makes the site feel like a real institution.
• Luxury perfume brand website: An amber perfume bottle stands on a marble surface. The liquid inside rotates slowly, and light passing through the glass casts caustic textures. The brand name is overlaid on the bottle in a serif typeface. The sense of luxury comes from restraint: there are almost no extraneous elements, with everything carried by light, shadow, and typography.

Three subject areas, three completely different visual languages and technical implementations, yet the underlying narrative structure, component design, and data-layer approach remain consistent. This indicates that the agent has learned not a specific template but a transferable style grammar. For teams, this means the prompt structure can become an asset: to enter a new business domain, you only need to replace domain-specific terms and assets while reusing the same generation workflow.

A reusable evaluation checklist

Based on the experiments above, here is a practical checklist for evaluating open-source coding agents:

Dimension	What to check
Visual completeness	Restrained color palette, rhythmic animations, coherent typography
Information architecture	Does the page tell a story? Is the structure self-consistent?
Component interaction	Do interactive elements (SVG, forms, linked updates) actually work?
Data layer	Is persistence real? Does data survive a restart?
Business rules	Are capacity limits, locking, and exception paths closed-loop?
Stability	Does repeated generation with the same prompt produce consistent quality?
Transferability	Does the same prompt structure maintain quality across different subjects?

A suggested workflow: first use public leaderboards to shortlist two or three candidate models, then run each through this checklist with two to three subject areas, sampling multiple times per subject, and finally score them holistically. With a small sample size, a single impressive output has no statistical significance — be wary of being misled by cherry-picked demos.

From demo to production: the last mile of engineering

Agent-generated applications still need work before they are deployable. These are useful observations for evaluation as well as tasks teams must handle when moving toward production:

1. Environment and dependencies: Generated output is usually runnable on a local machine. Production needs dependency locking, configuration management, and secrets management.
2. Data and storage: Replace demo-grade local persistence with production-grade solutions such as relational databases, object storage, caches, and message queues. For example, Baidu AI Cloud provides general-purpose services in categories such as container platforms, object storage, databases, and log monitoring that can be integrated directly, allowing developers to focus on application logic.
3. Observability: Integrate unified logging and monitoring alerts so that anomalies in write operations such as bookings can be located and traced promptly.
4. Code review and testing: Agent-generated code must go through the same review and test supplementation standards as human-written code, especially for logic involving inventory, funds, and concurrency.
5. Prompt asset management: Turn validated prompt structures into team templates, organize them by business domain, and continuously run regression checks as models evolve.

Conclusion

Leaderboard scores for open-source large language models are no longer scarce. What is scarce is the certainty that a single prompt can deliver a product that runs, looks polished, and closes the loop on business logic. To evaluate this capability, benchmark scores are only the entry ticket. Task-based testing is the main event: examine visuals, narrative, and interaction, but more importantly, examine whether the data layer and business rules form a closed loop, and whether the same methodology remains stable across different subjects. By turning the evaluation process into checklists and prompt templates, teams can answer the most important question — “What can it actually build?” — at minimal cost throughout rapid model iteration cycles.