Evaluating Open-Source LLMs for Full-Stack Delivery: From UI Generation to Backend Logic

Over the past year, the capability gap between open-source and closed-source models has continued to narrow. In several comprehensive benchmarks, the open-source camp has begun to compete directly with top-tier closed-source models, even surpassing them in certain software engineering and terminal operation tasks. However, for developers, benchmark scores are merely a reference. The real determinant of viability is another question: What can it actually build?

Rather than focusing on leaderboards, this article dissects the capability structure, technical key points, and evaluation methods of current open-source models in practical delivery. We will explore a specific engineering scenario: using a prompt to drive an AI Agent to complete a runnable full-stack application. The model capabilities discussed are based on public evaluations and general practices, without referring to specific commercial products.

1. From “Writing Code” to “Delivering Systems”: A Capability Shift

The evaluation criteria for traditional code completion models rely on whether the generated code snippet is correct. However, when a model operates as an Agent, the evaluation criteria fundamentally change:

1. Task Decomposition: Can it break down a natural language requirement into sub-tasks such as frontend, backend, data layer, and deployment configuration?
2. Cross-File Consistency: Are the interfaces, field names, and routing conventions across multiple files self-consistent?
3. Runnability: Can the output actually start, interact, and persist data, rather than just “looking like” it works?
4. Visuals and UX: Without design mockups, can it independently establish a self-consistent visual language?
5. Self-Verification: Can it validate and fix the output using terminal commands, test scripts, or browser behaviors?

Among these five dimensions, the first three determine if the output is usable, while the latter two dictate the quality of the experience. Let’s break them down using a typical practice scenario.

2. Practical Scenario: Generating a Business Website from a Single Prompt

Consider the following requirement: Build an official website for a near-Earth orbit travel company. It must include cabin selection and flight booking features, complete with visual storytelling and practically usable booking logic.

The difficulty of this requirement lies in covering both brand expression and business logic. The former tests the model’s aesthetic and typographic capabilities, while the latter tests its data modeling and interface design skills.

2.1 Frontend: Autonomous Establishment of Visual Systems

High-quality frontend generation typically exhibits the following characteristics:

• Unified Color and Typography Systems: Primary colors, secondary colors, monospace fonts for data panels, and serif fonts for brand headings to create hierarchy.
• Animations and State Expression: Breathing status indicators, scroll-driven narrative timelines, rather than static arrangements.
• Interactive Graphic Elements: Cabin selection presented as SVG graphics, complete with click, selected, and disabled states.

A common implementation path is to use scroll position to drive stage transitions. The core logic can be abstracted as follows:

// Scroll-driven narrative section activation
const stages = document.querySelectorAll('[data-stage]');
const observer = new IntersectionObserver((entries) => {
  entries.forEach(e => {
    if (e.isIntersecting) {
      e.target.classList.add('active');
      // Synchronously update the right-side flight data panel
      updateTelemetry(e.target.dataset.stage);
    }
  });
}, { threshold: 0.6 });
stages.forEach(s => observer.observe(s));

Whether a model can autonomously write this kind of structure depends on its abstraction level of the “scroll-driven narrative” pattern, rather than its memorization of a specific framework’s API.

2.2 Backend: Turning a “Shell” into a “Real System”

The most direct way to determine if a generated output is an empty shell is to check whether it contains persistence and constraint logic. Taking flight booking as an example, it requires at least:

• Data Models: Flights, cabins, passengers, orders.
• Capacity Constraints: Seat limits per flight, auto-locking when full.
• Concurrency Safety: Preventing the same seat from being booked multiple times.
• Interfaces: Querying available seats, submitting a booking, querying order status.

A simplified seat locking logic example:

async function reserveSeat(flightId, seatId, passenger) {
  return db.transaction(async (tx) => {
    const seat = await tx.seats.findOne({
      where: { flightId, seatId },
      lock: tx.LOCK.UPDATE,          // Row-level lock to prevent concurrent overselling
    });
    if (!seat || seat.status !== 'available') {
      throw new Error('SEAT_UNAVAILABLE');
    }
    seat.status = 'reserved';
    await seat.save({ transaction: tx });
    return tx.orders.create({ flightId, seatId, ...passenger });
  });
}

The key to this code is not syntax, but whether the model understands the business common sense that “booking systems must have concurrency control.” This is the watershed between “code generation” and “engineering delivery.”

2.3 Cross-Domain Reusability: Prompt Structure Stability

It is worth noting that when the same prompt structure is applied to different domains (such as deep-sea research institutions or luxury perfume brands), whether the output maintains structural consistency and logical integrity is crucial. Stable performance indicates that the model has learned transferable construction paradigms, rather than overfitting to a single template. This is particularly important for enterprise users—they need a reproducible production process, not just a one-time impressive demo.

3. Terminal and Engineering Automation: An Underestimated Component

Performance on terminal operation tasks often reflects a model’s engineering literacy better than UI generation. Such tasks include:

• Initializing project skeletons, installing dependencies, and configuring build scripts.
• Running local services and verifying API responses with scripts.
• Reading error logs and locating problematic files.
• Executing tests, fixing failed cases, and re-verifying.

A typical self-verification loop can be written as:

# Start service and perform a smoke test
npm run build && npm start &
sleep 5
curl -s -X POST localhost:3000/api/reserve \
  -d '{"flightId":"F1","seatId":"A1","name":"test"}' | jq '.status'

If a model can autonomously construct and execute such verifications, it can significantly reduce the rework of “code that compiles but doesn’t run.” This explains why software engineering and terminal coding benchmarks have gained increasing attention recently—they align more closely with real-world development pipelines.

4. Practical Metrics for Evaluating These Capabilities

When selecting or evaluating model capabilities for production, teams should focus on the following dimensions rather than single benchmark scores:

1. First-Pass Success Rate: Given a requirement, can the initial output run directly?
2. Fix Iterations: On average, how many interaction rounds are needed to fix an error?
3. Structural Completeness: Does it include data, interface, and validation layers, rather than just UI?
4. Maintainability: Do file partitioning, naming, and comments follow engineering conventions?
5. Reproducibility: Is the output quality stable across multiple runs with the same prompt?

These metrics align more closely with a team’s actual cost structure.

5. Considerations for Production

• Be Specific with Requirements: Clearly defining data entities, constraints, and interaction paths can significantly improve output quality.
• Establish a Verification Loop: Make “runnability” a hard acceptance criterion, rather than relying on manual visual inspection.
• Focus on Security and Compliance: Generated backend logic requires code auditing, especially for authentication, input validation, and database operations.
• Retain Human Review: The subjective quality of visuals and interactions still requires oversight from designers.

Conclusion

While the performance of open-source models in comprehensive benchmarks is noteworthy, their true value lies in their ability to turn a requirement into a runnable, maintainable, and reproducible system. From UI generation to backend persistence and terminal self-verification, every step in this pipeline tests the model’s engineering comprehension. For developers and enterprises, evaluating and utilizing these capabilities based on “delivery quality” rather than “leaderboard ranking” is the more prudent path forward.