From Benchmarks to Full-Stack Delivery: A Practical Analysis of LLM Agent Capabilities

1. Background: What Developers Should Really Care About Beyond Benchmarks

Open-source large models have been iterating at a notably rapid pace. A new generation of models keeps setting records on public benchmarks: scores on high-difficulty comprehensive knowledge evaluations are approaching those of leading closed-source models, results on software engineering and terminal operation benchmarks are climbing steadily, and open-source models have secured prominent positions on third-party aggregated leaderboards.

For developers, these numbers send a positive signal, but making model selection decisions based directly on benchmark scores can easily lead to misjudgment:

• Benchmark scores reflect “average performance on a specific task distribution” and do not have a linear relationship with delivery quality in real business scenarios;
• Leaderboard rankings are affected by factors such as data contamination, task coverage, and scoring script differences;
• What determines “whether it can be used” is often the failure modes in long-tail scenarios, not the average score.

Therefore, this article deliberately bypasses benchmark scores themselves and focuses on two more practical questions: What exactly are these benchmarks testing? And to what extent can models with Agent capabilities actually get the job done? On this basis, it provides a reusable evaluation and implementation methodology.

2. Understanding Coding Benchmarks: A Three-Layer Evaluation Framework

Mainstream coding evaluations can be roughly divided into three layers. Understanding what each layer examines is essential for correctly interpreting scores.

1. Comprehensive Knowledge Layer: Reasoning Reserves Under Extreme Difficulty

These benchmarks focus on highly difficult, cross-disciplinary questions with broad coverage, examining the model’s long-tail knowledge reserves and complex reasoning abilities. A high score means the model is more reliable at problem decomposition and knowledge retrieval in niche domains, but it has only an indirect relationship with “writing code”—it is more like the intellectual foundation of the entire capability system.

2. Software Engineering Layer: Task Execution in Real Repositories

This layer is closest to daily development: the model is given a bug or requirement in a real code repository and must locate the problem, modify the code, and pass regression tests. Unlike “function-level completion,” it examines:

• Context retrieval: finding relevant modules in hundreds of thousands of lines of code;
• Multi-file coordinated modification: changing one place often affects multiple others, requiring a global perspective;
• Self-verification: the closed-loop ability to run tests, read errors, and iteratively fix.

When scores on this type of benchmark move from the 50s to the 60s, it means the proportion of real engineering problems the model can solve in one shot without human intervention has increased significantly; but there is still a gap before it can “independently hold the position”—failure cases are concentrated in scenarios such as large-scale refactoring, cross-repository dependencies, and implicit business rules.

3. Environment Interaction Layer: Multi-Step Operations in the Terminal

Terminal benchmarks require the model to complete multi-step tasks in a real Shell environment, such as installing dependencies, executing scripts, diagnosing errors, and configuring environments. What is examined is the Agent’s “hand-eye coordination”: issuing commands, observing output, and correcting strategy. A high score at this layer is a prerequisite for an Agent to take on automated operations and environment setup tasks.

4. Aggregated Leaderboards: Reference Value and Limitations

Third-party aggregated leaderboards weight and combine scores from various models. They are suitable for initial screening, but note: the weighting criteria are not transparent, some sub-items have weak relevance to business needs, and they cannot reflect performance in private domains. The correct approach is: use leaderboards to narrow down candidates, and use your own tasks for final judgment.

3. Agent-Style Full-Stack Delivery: Breaking Down Four Core Capabilities

Benchmark scores are “exam-taking ability,” while Agent delivery is “work ability.” From a large number of hands-on test cases, the delivery capability of strong Agent models can be broken down into four aspects:

1. Task Planning: From a Single Sentence to an Engineering Blueprint

Give the Agent a single sentence like “build a website for a low Earth orbit travel company, including cabin selection and flight booking,” and it needs to independently complete:

• Information architecture design: homepage, flight timeline narrative, cabin page, booking flow;
• Visual system definition: primary colors, fonts, grid, motion tone;
• Data model design: entities such as flights, seats, passengers, and orders, along with their relationships;
• Technology selection: frontend structure, backend APIs, persistence solution.

Planning quality directly determines the completeness of subsequent generation, and this is also the stage where the gap between different models is largest.

2. Completeness of Single-Shot Generation: No Patching Holes Through Multiple Rounds of Follow-Up Questions

Weak models need repeated follow-up questions to fill in pages and logic, while strong models should deliver within a single generation: a complete, runnable project structure, connected frontend and backend, and complete empty states and exception handling. The criterion is simple—can the code be pulled down and run directly.

3. Constraint Modeling: Business Rules Translated into Code

“6 seats per flight, automatically locked when fully booked” sounds like a product description, but translating it into code requires considering concurrency and consistency, as shown below:

def book_seat(flight_id, seat_no, passenger):
    with db.transaction():
        flight = db.query_flight(flight_id, for_update=True)
        if flight.locked or seat_taken(flight_id, seat_no):
            raise SeatUnavailable(flight_id, seat_no)
        db.insert_booking(flight_id, seat_no, passenger)
        if booked_count(flight_id) >= flight.capacity:
            db.lock_flight(flight_id)   # auto-lock when full

Whether the model can proactively think of row-level locks, the timing of full-capacity determination, and duplicate submission protection is the dividing line between “can write pages” and “can write systems.”

4. Persistence and State Management: Saying Goodbye to “Lost on Refresh”

Booking data must be persisted to the database, seat status must be consistent across sessions, and full-capacity locks must remain effective persistently. This means the Agent generates not a static shell, but an application with a complete data pipeline.

4. Case Review: Different Themes, Same Capability Patterns

Output from the same Prompt structure across different themes can clearly reveal the stability of model capabilities:

Case Theme	Key Evaluation Points	Capability Demonstrated
Commercial spaceflight website	Narrative + interaction + booking system	Full-screen video background, timeline-driven mission narrative, SVG cabin map, runnable booking backend
Deep-sea research website	Camera movement + information density	First-screen descent animation, equipment specification cards, multi-author section structure
Luxury perfume brand	Extreme visual aesthetics	Light and shadow caustics texture, serif typography, minimalist narrative rhythm

The commonalities across the three cases are worth noting:

1. Narrative-driven: Pages are not a pile of components, but information organized along a “timeline/spatial flow”;
2. Complete visual system: Fonts, colors, and motion form a unified language rather than a template collage;
3. Integrated frontend and backend: Even demo-level sites come with real business constraints and data persistence.

This shows that strong Agent models have evolved from “generating code snippets” to “generating runnable product prototypes.”

5. Engineering Implementation: Prompt Design and Collaboration Boundaries

1. Structured Prompt Template

To get stable output from an Agent, the Prompt itself must be engineered. A five-part structure is recommended:

[Role] Senior full-stack engineer responsible for delivering a runnable site from scratch
[Theme] Low Earth orbit travel company website, brand tone: premium, restrained, tech-forward
[Visual] Full-screen video background (Earth's curvature from a porthole view), monospace data panels, fluorescent cyan status indicators
[Features] SVG cabin selection, flight booking (date/seat/passenger), 6-seat limit per flight, auto-lock when full
[Acceptance] One-click local startup; complete form validation; booking data persistently queryable; no further bookings after full capacity

“Acceptance criteria” is the most easily overlooked yet most effective section—it translates the vague “make it look good” into determinable conditions.

2. Phased Acceptance

• Phase 1: Static structure and visual walkthrough;
• Phase 2: Interaction flow (seat selection → form filling → submission → query);
• Phase 3: Boundary conditions (concurrent bookings, invalid input, full-capacity lock).

3. Human-Machine Collaboration Boundaries

The model handles generation, while humans handle three fallback responsibilities: security (authentication, injection protection, key management), compliance (asset copyright, privacy statements), and performance and cost (static asset size, API stress testing). These are precisely the areas where current Agents are weakest.

4. Deployment and Integration

When deliverables are deployed to production environments, general cloud capabilities can be reused directly: static assets hosted on object storage, backend services deployed on container platforms, data layer using cloud database services, and integration with logging services and monitoring alerts. Taking Baidu AI Cloud as an example, its general product categories such as object storage, container platform, and cloud database can cover the complete deployment pipeline for such sites, without the need to build a self-managed operations system.

6. Building Your Own Evaluation Checklist

Based on the above analysis, developers are advised to use a four-step approach instead of “picking models by looking at leaderboards”:

1. Leaderboard initial screening: Narrow down to 3–5 candidate models;
2. Benchmark re-testing: Select publicly available benchmark sub-items relevant to your business for validation;
3. Business hands-on testing: Run end-to-end delivery with real requirements (including boundary conditions);
4. Failure mode analysis: Focus on recording which scenarios the model fails in and whether errors are fixable, rather than only looking at success rates.

7. Conclusion

The continuous rise in benchmark scores proves that the engineering capabilities of open-source models are rapidly approaching the level of leading closed-source models. But what is truly valuable for developers is a judgment framework that translates “scores” into “delivery capability”: the comprehensive knowledge layer shows the foundation, the software engineering layer shows real-world fixes, and the terminal interaction layer shows the Agent closed loop; then, through structured Prompts, phased acceptance, and clear human-machine division of labor, model capabilities can be stably converted into business output. Benchmark scores will eventually become outdated, but this methodology will remain effective for a long time.