# LLM Version Evaluation and Intent-Driven Programming: A Practical Framework for Benchmarking New Model Releases

    Every new LLM release triggers debate about whether real capability improved and what changed for natural-language-driven coding. This article defines intent-driven programming, lays out a four-dimension, five-principle evaluation framework, describes a tiered task set (L1–L4) with automated pass criteria, and shows how to report results so that gains in benchmark averages translate into measurable engineering productivity rather than unverifiable claims. Includes reference code for the change-delivery loop and for structured evaluation reporting.

    1. Background: Why Every Release Comes With Debate

    Whenever a new LLM version ships, or whenever a coding scenario delivers that “make a wish and it comes true” experience, the discussion tends to narrow to two questions:

    1. Did the new version actually improve real capability — coding, long context, instruction following, tool use — or did something shift in reasoning efficiency or cost structure?
    2. What was optimized for natural-language-driven programming, where users simply describe the result they want and the model grants the wish?

    These discussions have value, but they usually suffer from three problems: small sample sizes, tests improvised on the spot by whoever is doing the evaluating, and no baseline for comparison. More importantly, people tend to equate “looks stronger in one scenario” with “overall capability improved.” From an engineering perspective, a conclusion requires three things: a clearly defined evaluation question, a fixed baseline, and a repeatable methodology. Below, we first define intent-driven programming, then present a complete framework for version evaluation, and finally explain how to convert evaluation results into measurable engineering productivity.

    2. Defining “Make-a-Wish” Programming: The Technical Essence of Intent-Driven Programming

    “Make-a-wish programming” is more precisely called Intent-Driven Programming. Traditional completion answers “what should the next line be”; intent-driven programming answers “what result do I want.” It must satisfy at least three conditions:

    • Intent expression: describe the target behavior, constraints, and acceptance criteria in natural language;
    • Complete deliverables: output executable artifacts — code, interface definitions, configuration, tests — not just fragments;
    • Verifiable feedback loop: confirm through compilation, unit tests, static analysis, and similar means that the artifacts satisfy the intent.

    Figure: The delivery pipeline of intent-driven programming

    1. User intent (natural language)
    2. |
    3. +-- Intent clarification: inputs/outputs, edge cases, constraints, acceptance criteria
    4. v
    5. Context assembly: relevant files, interface definitions, dependency versions, coding conventions
    6. |
    7. v
    8. Implementation: design -> code and configuration -> test cases
    9. |
    10. v
    11. Automated verification: build / unit tests / static scanning / sandbox execution
    12. |
    13. +-- Failure --> Targeted repair based on errors and failing cases (bounded rounds)
    14. |
    15. v Pass
    16. Change delivery: change notes, risk flags, human review, release

    This pipeline imposes several engineering requirements, and these are exactly where “does it really improve productivity” is decided:

    1. Context scale: understanding a single business feature often spans multiple files, which requires long-context comprehension or efficient retrieval augmentation;
    2. Instruction following: constraints may be buried in the conversation — “do not modify shared configuration,” “preserve the existing logging format” — and the model must honor them consistently;
    3. Autonomous execution and self-correction: the model must be able to call tools to read code and run tests, then fix itself based on failure output;
    4. Consistency: the same or similar inputs should produce stable outputs, otherwise a team cannot form expectations.

    Sample Code 1: Skeleton for handling an intent-driven change

    1. def deliver_intent(intent, repo_ctx, max_rounds=3):
    2. """
    3. Converge a vague intent step by step into a verifiable, reviewable code change.
    4. Any failed round must leave readable evidence behind so a human can step in.
    5. """
    6. spec = clarify(intent, ctx=repo_ctx) # Clarify intent: inputs/outputs/constraints/acceptance criteria
    7. spec = spec.with_constraints(["do not modify shared configuration", "preserve the existing logging format"])
    8. plan = split_into_tasks(spec) # Split into small tasks that can be verified independently
    9. for task in plan:
    10. patch = generate(task, ctx=repo_ctx(task)) # Retrieve related files and interface definitions
    11. for round_idx in range(max_rounds):
    12. report = verify(patch) # Build, unit tests, static scanning
    13. if report.passed:
    14. break
    15. patch = repair(patch, errors=report.errors) # Targeted repair driven only by failure evidence
    16. if not report.passed:
    17. return Change(blocked=True, task=task, reason=report.summary) # Hand off to a human
    18. repo_ctx = apply_and_refresh(patch, repo_ctx) # Refresh context after applying
    19. return Change(blocked=False, files=touched_files(), report=report)

    Key design points

    • spec must be clarified before generation, to avoid a “one big change” on top of vague requirements;
    • max_rounds caps automated repair rounds and prevents loops of endless modification;
    • on failure, return a readable blocking reason so that “hand it to a human” is an explicit state rather than a stray error log.

    3. The Overall Evaluation Framework: Four Dimensions and Five Principles

    3.1 Four dimensions

    For a version iteration, establish a baseline across four dimensions:

    Dimension Focus Typical metrics
    Core language ability General reasoning, instruction following, code generation Task pass rate, instruction violation rate
    Long context and retrieval Multi-file understanding, long-document QA Key-information localization accuracy, first-passage answer relevance
    Agentic ability Tool use, multi-step planning, error recovery Task completion rate, average interaction steps, infinite-loop rate
    Engineering cost Latency, throughput, token consumption, stability P50/P95 latency, tokens per task, retry failure rate

    3.2 Five principles

    1. Define the question first: write down what you are answering — for example, “does the new version outperform the baseline on code generation” — and then choose metrics, rather than hunting for an explanation after seeing results.
    2. Fix the baseline: old and new versions must run the same tasks, the same context, and the same grading scripts. Record the baseline version number alongside the dataset version.
    3. Repeat and report variance: sample each task multiple times and report mean and variance. If the gap between two versions falls inside the noise band, you cannot call it an improvement.
    4. Weight by business scenario: replace generic leaderboards with your own task set. Leaderboards reflect industry-average capability, not performance against your codebase and conventions.
    5. Make it attributable: trace significant changes to specific sub-capabilities (long context, tool use, instruction following) instead of settling for the unactionable conclusion that “it got better overall.”

    4. Evaluation Methodology for Coding Scenarios

    4.1 Designing the evaluation task set

    Build a tiered task set organized by “difficulty × decidability”:

    • L1, unit level: given a function signature and semantic requirements, generate the implementation. Graded by unit test pass.
    • L2, module level: add a feature or fix a defect within one module. Graded by regression tests plus functional tests.
    • L3, system level: cross-file changes, dependency replacement, interface migration. Graded by integration tests plus compatibility checks.
    • L4, vague-intent level: the requirement description itself is incomplete and must be clarified before implementation. Graded by clarification quality plus whether the delivered result matches the clarified spec.

    L4 is the core of the “make-a-wish” scenario and the part most prone to overstatement. When evaluating it, explicitly record the number of “clarification rounds” and the share of cases where work started “without clarification.”

    Sample Code 2: Field design for the evaluation report

    1. @dataclass
    2. class CaseResult:
    3. case_id: str
    4. level: str # L1-L4
    5. passed: bool
    6. auto_pass: bool # Passed automated verification (no human review needed)
    7. clarify_rounds: int
    8. patch_lines: int
    9. tokens: int
    10. latency_ms: int
    11. first_failure: str # First failure type: build/unit test/static check/convention
    12. def summarize(results: list[CaseResult]) -> dict:
    13. agg = defaultdict(list)
    14. for r in results:
    15. agg[r.level].append(r)
    16. out = {}
    17. for level, rs in agg.items():
    18. out[level] = {
    19. "pass_rate": round(sum(r.passed for r in rs) / len(rs), 3),
    20. "auto_pass_rate": round(sum(r.auto_pass for r in rs) / len(rs), 3),
    21. "avg_clarify_rounds": round(statistics.mean(r.clarify_rounds for r in rs), 2),
    22. "p95_latency_ms": percentile([r.latency_ms for r in rs], 95),
    23. "failure_breakdown": Counter(r.first_failure for r in rs if not r.passed),
    24. }
    25. return out

    auto_pass_rate and pass_rate must be reported separately. The former measures the share of output that is usable without anyone reading it, which directly determines how much review time a team saves; the latter is the ceiling on final quality.

    4.2 The six stages of the evaluation workflow

    1. Requirement collection: extract tasks from real issues, code review comments, and production defects, then anonymize and store them.
    2. Task standardization: unify input format, context scope, constraint lists, and acceptance scripts.
    3. Automated grading: build, unit tests, static scanning, and convention checks are all scripted, so verdicts do not depend on subjective human scoring.
    4. Two-person review: sample and cross-check to estimate the misgrading rate of automated judging. If the misgrading rate exceeds the threshold, conclusions should be downgraded to “pending confirmation.”
    5. Report output: produce a structured report organized by dimension, level, and failure type.
    6. Archive conclusions: store results together with the version number, dataset version, and sampling parameters so the work remains traceable six months later.

    5. Reading the Results: Avoiding Three Common Misjudgments

    1 “Average score went up” ≠ “critical scenarios are usable.” If the gains are concentrated in L1/L2 while L3/L4 are unchanged, the impact on engineering productivity is limited. Weight by the actual task distribution of your business.

    1. “It runs” ≠ “it is maintainable.” You also need to check coding conventions, duplication rate, comment and documentation completeness, and the growth in changed lines and complexity. Maintainability can be measured with automatically collectible metrics such as convention-check pass rate and duplicated-code share.

    2. “Looks smarter” ≠ “is more stable.” Result consistency across repeated runs of the same task (output stability, whether success can be reproduced) should be measured separately. An unstable model significantly raises the cost of human fallback.

    6. Rollout Checklist

    • Establish a baseline: keep a frozen evaluation set and grading scripts; on version upgrades, swap only the model and leave the tasks untouched.
    • Evaluate in tiers: grade across L1–L4, and use only the levels relevant to your scenario for business metrics.
    • Set thresholds: define the minimum pass rate and the maximum P95 latency allowed for production; anything below the bar does not enter the canary phase.
    • Canary and rollback: start with a small number of internal projects, track the human-takeover rate weekly, and roll back on anomalies.
    • Record cost: count token consumption and retry counts into total cost of ownership, not just per-call price.
    • Institutionalize the practice: maintain the evaluation set as a team asset so you avoid “evaluate once, conclude, and let it expire.”

    7. Conclusion

    New releases come with a lot of noise, but the method can stay clean: define the question, fix the baseline, evaluate in tiers, report in a structured way, and interpret results per scenario. For intent-driven programming, what really determines the experience has never been how impressive a single generation looks, but whether the full chain — clarification, context assembly, automated verification, and human fallback — is intact. Measure first, then migrate; tier first, then conclude — this is the only reliable path from version capability to engineering productivity.