Evaluating Open-Source LLMs on Requirement-Driven Code Generation: A Repeatable Benchmark and Lessons from Production
Leaderboard positions between flagship models shift almost monthly, yet engineering teams keep reporting the same gap: models that chat impressively but deliver weakly. This article explains why, and presents a repeatable evaluation method built around requirement fidelity rather than single-turn accuracy — fixed input documents, a graded page sample set, and double-blind scoring. It walks through the failure modes observed in practice (missing list filters, partial cross-field validation, style drift), shows how to convert implicit requirements into machine-checkable contracts and an automated acceptance pipeline, and covers what open weights actually buy you: data boundary control, targeted fine-tuning, predictable cost, and reproducible evaluation.
Requirement-Driven Generation in Open-Source LLMs: Evaluation and Production Practice
1. Reframing the Problem: Leaderboards Measure the Surface, Requirement Fidelity Decides the Outcome
Over the past two years, flagship models from different vendors have traded wins on public benchmarks, and a lead on any single metric tends to disappear within a month. What frontline teams actually experience is different: models that are strong in conversation but weak in delivery. That is not a perception bias — it is a mismatch between benchmark criteria and engineering criteria.
Leaderboards measure the accuracy of single-turn question answering: complete prompt, single objective, automatically scorable. Engineering measures requirement fidelity under long-document constraints: the input is a document containing field definitions, interaction rules, boundary conditions, and visual specifications; the output is a multi-page application that can be built, integrated, and deployed. The gap between the two breaks down into three dimensions:
- Constraint extraction: whether implicit rules in natural language (pagination resets after a filter change, cross-field conditional validation) can be translated into executable code contracts.
- Cross-page consistency: whether the model drifts when several pages share the same data model and state semantics.
- End-to-end runnability: not “a pasted code snippet”, but a complete, accessible, buildable, deployable project.
So the right question when deciding whether a new model deserves a place in your stack is not “how does it rank”, but: at my level of requirement-document quality, what is its requirement fidelity, and are its failure modes predictable and interceptable? The evaluation method below is a reusable pattern distilled from repeated use on real projects.
2. Evaluation Design: Fixed Inputs, a Fixed Page Set, Double-Blind Scoring
Reproducibility requires locking down three variables.
2.1 Input: a Requirement Document of “Moderate Clarity”
Write the document deliberately at the level your team actually works at — structured and itemized, but with a small number of implicit constraints that require inference. If the document reads like a formal specification, you are measuring long-context recall. If it is too casual, failures cannot be attributed to a cause. Moderate clarity is what best exposes how models differ in inferring intent.
2.2 Sample: A Difficulty Ladder Across Three Page Types
| Page type | What it tests | Typical failure mode |
|---|---|---|
| Login / registration | Basic form validation, routing, state persistence | Rarely fails — this is the comfort zone |
| List / query | Filter conditions, pagination, empty state, mapping between URL and request parameters | Filters systematically missing |
| Complex form | Field linkage, cross-field validation, submit state and failure rollback | Validation logic and field relationships only partially restored |
2.3 Scoring: Machine-Readable Plus Human Double-Blind
Let the automated portion answer “does it run, is it implemented”, and let the human portion answer “does it look like a product”. Freeze the rubric into a checklist so that conclusions are not based on a vague sense that “it looks fine”:
- Build success (high weight, veto item): dependencies complete, type checks pass, starts locally.
- Feature coverage: ratio of implemented items to required items.
- Constraint adherence: proportion of explicitly stated rules correctly implemented.
- Interaction state completeness: loading, empty, error, and disabled states all present.
- Data contract consistency: whether field naming and semantics for the same entity are consistent across pages.
- Visual specification consistency: whether spacing, hierarchy, and color follow a shared design token set rather than arbitrary values.
3. What Testing Showed: Failures Are Attributable, Not Random
Several conclusions from this round of evaluation held up consistently.
3.1 Simple Pages Are Already in “Tweak and Ship” Territory
Login-type pages have mature structure and highly convergent patterns, and the model reliably produces complete implementations: carousels, form validation, redirects, and prompts. Such tasks no longer discriminate between models. Treat them as a pass-line test — a check on whether a model respects basic engineering conventions at all.
3.2 Missing Filters on List Pages Is a Systematic Weakness, Not an Accident
This is the signal most worth watching. The page skeleton is correct and data rendering is correct, but filter controls that are explicitly written in the document are either not implemented or implemented as static shells with no state wiring. The root cause is that filtering is inherently a cross-layer chain:
UI control -> page state -> URL / query param sync -> request param assembly -> pagination reset -> result reflow
Models tend to produce code that is visually complete but logically hollow, because a missing link in this chain breaks neither rendering nor static checks. That is why a list page “looks right and behaves wrong” — and why the same problem repeats across a multi-page requirement set: the model does not treat filtering as a shared state contract, but as a set of isolated UI components.
3.3 Form Pages Expose the Boundary of Validation and Field-Relationship Reasoning
Forms run, but dependencies between fields, when validation triggers, where error messages appear, and idempotency plus failure rollback during submission are often only half implemented. Cross-field validation (for example, “the end time must not precede the start time”) requires the model to hold both rules simultaneously while generating and to maintain their coupling — considerably harder than emitting a single required rule. Similarly, cascading selectors and dynamically required fields are frequently reduced to their first level only.
3.4 Latency and Queueing: Separating Engineering Factors from Model Capability
End-to-end generation of a multi-page project takes noticeably longer, and the timing correlates strongly with free-tier queueing policy. An evaluation must separate inference throughput (affected by service load, concurrent queueing, context length) from generation quality (affected by model capability). Put both in one table and the conclusion is guaranteed to be distorted. For the same model, elapsed time can differ several-fold across time slots and concurrency levels, so latency should be recorded only as an availability metric, never used to rank capability.
3.5 Visual Style: Engineering-Usable Is Not Product-Ready
Generated UIs usually beat wireframe prototypes, but they still fall short of a mature design system — a unified spacing grid, a component library’s design tokens, a responsive breakpoint strategy. The reason is that aesthetics and consistency are global constraints, and a model’s attention to global constraints decays over a long context. This is the same problem as missing rules in the later part of a long document, seen from another angle.
4. Turning the Gap into a Capability Model: Requirement Contracting Is the Fix
A conclusion like “insufficient depth of understanding” has no action value. The actionable statement is: the model degrades when it must maintain implicit constraints that span pages and layers. The corresponding engineering answer is not to expect the model to read an entire document correctly in one pass, but to make implicit constraints explicit as machine-verifiable contracts.
4.1 Replace Prose with Contracts
Alongside the requirement document, add a minimal executable contract snippet for each page, and require the model to fill in the contract before writing code:
page: order-listinputs:filters:- { name: status, type: enum, values: [pending, paid, shipped, closed], default: all }- { name: keyword, type: string, maxLength: 64 }- { name: createTimeRange, type: dateRange, default: last7days }behaviors:- on_filter_change: reset(page, 1) -> merge(query) -> refetch()- on_page_change: keep(filters) -> set(page) -> refetch()- url_sync: filters <-> query params (bidirectional)states: [loading, empty(filters_active -> reset_entry), error(retry)]acceptance:- AC-01: Any filter change must reset pagination to page 1- AC-02: Filter conditions must be restorable from the URL after a refresh- AC-03: An empty result must be displayed distinctly from "no data"
The payoff is that missing implementations turn from random failures into deterministic ones: contract items can be mapped automatically to test cases.
4.2 Generation Is Not the Finish Line — the Pipeline Is
Put generated artifacts through a fixed verification chain, and let humans handle only what machines cannot judge:
for case in eval_set:spec = load(case.spec)repo = agent.generate(spec, framework="team_standard_stack")r.build = lint_and_typecheck(repo) # syntax, types, buildabilityr.contract = run_acceptance_tests(repo, spec) # requirement contract casesr.visual = visual_regression(repo, case.base) # layout and style driftr.review = blind_score(r, rubric) # double-blind human scoringreport.append(case, r)
Contract tests are the critical link. A rule like AC-01, “reset pagination on any filter change”, takes a few minutes to verify with one automated case, versus ten minutes of manual clicking that still tends to miss things. Once a team has this pipeline in place, subjective impressions of a model give way to a pass-rate curve, and model-selection discussions shift from argument to reading numbers.
4.3 Generate Incrementally, Not All at Once
Against long-context attention decay, the more reliable practice is to generate entity definitions and data contracts first, then generate page by page in batches, and finally run a cross-page consistency pass — unifying field naming, reusing validation rules, and extracting shared state management. This produces markedly less rework than generating an entire project at once and then fixing it.
5. What “Open Source” Actually Means: From “Usable” to “Controllable”
Back to the original question: the value of open source is not leaderboard strength. It is four concrete engineering benefits.
Control over data boundaries. Requirement documents, internal API definitions, and business fields often contain sensitive information. Self-hosted open weights keep that content inside your own network, so you never have to assess “is this document suitable to send to an external service” — which is frequently a veto item in a compliance review.
Targeted capability improvement. Open weights permit incremental fine-tuning. The common payoff for teams is not “making the model smarter” but “making it follow our conventions”: a standard stack, a fixed directory structure, mandatory validation rules, a unified set of design tokens. Taught through prompts alone, models drift; fine-tuned on a small set of high-quality internal samples, the difference in stability is substantial.
A predictable cost structure. Self-hosting shifts cost from usage-based billing to “resource depreciation plus headcount”, which is easier to plan capacity around when call volume is stable and concurrency is predictable. If you do not self-host, a general-purpose combination of managed services can carry the same workload — model serving, container platform, object storage, logging, and monitoring with alerting are standard building blocks available as general-purpose offerings on cloud platforms such as Baidu AI Cloud, so you do not have to build the entire infrastructure stack yourself.
An auditable process and reproducible results. Open models let you pin weight versions, fix inference parameters, and version the evaluation set. That means when you look back in six months and ask “why did we choose this one”, you can reproduce the same conclusion. If a closed service undergoes a silent upgrade, historical evaluation data becomes invalid immediately.
One caveat: open source does not mean free capability. Deployment, tuning, inference performance load testing, canary rollout, and rollback are all real engineering investments. Until a small team has costed that headcount, the net benefit of “open source” may be negative.
6. Conclusion: Your Evaluation Criteria Are the Moat, Not a Single Leaderboard
Three takeaways from this round of testing.
First, page types sit on a difficulty ladder. Login pages are already in “tweak and ship” territory and provide no discrimination. The filter chain on list pages and cross-field validation on form pages are where current mainstream models separate. Test the latter when selecting a model.
Second, predictable failure modes are interceptable ones. A model omitting filters is not bad luck; once contracted into test cases, the omission surfaces at commit time. What is genuinely scarce in engineering is not a stronger model, but a team’s ability to translate requirements into verifiable contracts.
Third, the meaning of open source lies in the word “controllable”. Data boundaries, convention alignment, predictable cost, and a reproducible process are the real reasons to put it in your stack — and that controllability has to be delivered through weight pinning, evaluation-set versioning, and an automated pipeline. A license text alone cannot deliver it.
As for “who to benchmark against”, it matters little. What decides a project’s success is whether you have an evaluation system that can quantify requirement fidelity. With one, any new model can be judged within a week. Without one, you are left trying models round after round on gut feel.
