Three-Tier Gradient Testing: A Practical Framework for Evaluating LLM Code Generation
Choosing a code-generation model based on a single impressive demo is unreliable. This article introduces a three-tier gradient testing approach for evaluating large language models on real-world coding tasks. It first outlines four capability dimensions to assess up front — agentic coding, context length and codebase understanding, reasoning enhancement, and cost/throughput — then defines three evaluation tiers: L1 for single-task correctness with automated assertions, L2 for page-level delivery and quantifiable UI quality, and L3 for agentic closed-loop execution and self-correction. It also covers how to make evaluations reproducible, common pitfalls to avoid, and a recommended path for rolling out model selection in production. You'll learn how to turn the vague question "is it usable?" into measurable, repeatable metrics.
1. Why “It Feels Good” Is Not a Selection Criterion
Over the past year, the ability of large models to write code has been discussed repeatedly: generating a runnable webpage from a single prompt, automatically fixing production errors, refactoring across multiple files. The marketing claims of different models keep converging, but the developer’s most basic question has not changed — when I switch this into my real business, will it actually work?
Reaching a conclusion from a single demo hits at least three traps:
- Benchmark scores are disconnected from real experience. Public leaderboard problems have fixed distributions and uniform answer styles, so models are easy to “overfeed” during training and tuning. Real requirements are often vaguely specified and depend on business context — a classic out-of-distribution input.
- Demo paths are carefully curated. A model that can produce a beautiful page won’t necessarily change one piece of logic safely inside an existing codebase.
- Long-term cost gets ignored. No matter how high the success rate of a single generation, if every attempt needs three rounds of manual rework, overall efficiency actually drops.
A more reliable approach is to cut “capability” along a difficulty axis into several tiers, each with a clear passing criterion and an automated way to judge it. This article organizes that into three tiers: L1 looks at correctness, L2 at delivery quality (including aesthetics), and L3 at closed-loop capability.
2. Capability Profile: Four Foundations to Check Before Testing
Before you start testing, you need a mental model of how model capability is structured. Otherwise it’s easy to mistake “strong in one dimension” for “usable overall.”
2.1 Agentic Coding
Traditional AI coding is one-shot text generation: you paste an error, it returns a patch, and you verify whether it’s correct. Agentic coding instead places the model inside an executable environment and gives it a feedback loop close to that of a human programmer:
- Active retrieval: when it needs external information, it goes and reads docs or searches the web instead of fabricating interfaces from memory;
- Execution and verification: it compiles, runs unit tests, and runs static checks before handing code over, rather than dumping it on you directly;
- Self-correction: after reading failure output, it revises its approach and repeats until tests pass or a round limit is reached.
The value here is that it partially converts the “accuracy of a single generation” problem into a “speed of convergence across multiple rounds” problem — and the latter can be improved significantly through engineering means (better error summaries, less context noise).
2.2 Context Length and Codebase Understanding
Long context windows let a model swallow an entire module, API docs, or even a design spec in one shot. But two real constraints apply: first, attention decay — when key information sits in the middle of a very long input, the probability of it being ignored rises noticeably; second, cost scales linearly — mindlessly stuffing the context is neither economical nor necessary. The right practice is to structurally trim the relevant fragments (relevant files, call chains, type definitions) rather than feed in the whole repository.
2.3 The Value of Reasoning Enhancement for Algorithms and Refactoring
For tasks involving math, edge cases, or complexity optimization, chain-of-thought (CoT) enhancement delivers the most obvious gains. Conversely, on highly templated boilerplate (CRUD endpoints, form validation), the marginal benefit of reasoning enhancement is limited, and throughput and unit price become the primary considerations.
2.4 Cost and Throughput
For teams making high-frequency calls, unit price, rate-limiting policy, and concurrency capacity directly determine whether something “can scale into production.” Chinese-hosted model services often stand out here for predictable pricing and stable response times, and multiple models can be managed uniformly through a platform’s model-service interface, making it easy to route by task difficulty and to fall back when needed.
3. Tier 1 (L1): Single-Task Correctness
Goal: verify that the model can produce semantically correct, edge-complete code on isolated tasks with no business context.
Test-case design principles:
- Every case must have an automatically assertable expected value; no eyeballing allowed;
- Cover three classes of traps: boundary values (empty collections, single elements, extreme values), error paths (timeouts, insufficient permissions, missing dependencies), and type contracts (nulls, union types, precision loss).
# Simplified skeleton for automated evaluationCASES = [{"name": "chunk_empty", "fn": chunk, "args": ([], 3), "expect": []},{"name": "chunk_remainder", "fn": chunk, "args": ([1,2,3,4,5], 2), "expect": [[1,2],[3,4],[5]]},{"name": "chunk_invalid", "fn": chunk, "args": ([1,2], 0), "expect": "ValueError"},]def evaluate(model, cases, temperature=0.0):passed, records = 0, []for c in cases:src = call_model(model, build_prompt(c), temperature=temperature)try:got = exec_in_sandbox(src, c["fn"], c["args"])ok = (got == c["expect"]) or (c["expect"] == "ValueError" and isinstance(got, Exception))except Exception as e:ok, got = False, repr(e)passed += int(ok)records.append({"case": c["name"], "ok": ok, "output": got})return passed / len(cases), records
What L1 tells you: first-pass rate, whether failures cluster around a specific class of trap, and whether the generated code includes the necessary input validation. A model scoring below 80% is not recommended to advance to the next tier.
4. Tier 2 (L2): Page-Level Delivery and Quantifying “Aesthetics”
Goal: given a one-line requirement, evaluate whether the model can directly produce a complete page that is well-structured, smooth to interact with, and visually restrained.
“Aesthetics” sounds subjective, but it can be broken down into several observable, scorable engineering metrics:
| Dimension | How to judge it |
|---|---|
| Visual hierarchy | Is there a clear scale of heading sizes (3 levels or fewer recommended) instead of uniform weight throughout? |
| Color restraint | No more than 2 primary colors; neutral colors cover 70%+ of the area; contrast meets WCAG AA (body text ≥ 4.5:1) |
| Spacing and density | Does block spacing follow a consistent rhythm (e.g., a 4/8 scale system) rather than random values? |
| Motion quality | Durations clustered in 150–300ms, natural easing, and disable-able via prefers-reduced-motion |
| Responsiveness | Do layouts genuinely reflow at breakpoints, rather than simply scaling or scrolling horizontally? |
| Semantics and accessibility | Are semantic tags like button/label/alt used, is focus visible, and is it keyboard-operable? |
Typical degradation patterns (also where some models have historically been criticized): piling on colors, overusing gradients and shadows, chaotic type hierarchy, and a rigid “template feel.” The test is simple — does the generated CSS introduce a design system (variables, a grid, a spacing scale), or just a scatter of magic numbers?
/* A design-system approach: tokenized and rhythmically consistent */:root {--space: 8px;--text-xs: 0.75rem;--text-base: 1rem;--text-lg: 1.25rem;--brand: #2563eb;--fg: #0f172a;--muted: #64748b;}.card { padding: calc(var(--space) * 3); border-radius: 12px;box-shadow: 0 1px 2px rgb(15 23 42 / 8%); }@media (prefers-reduced-motion: reduce) {* { animation: none !important; transition: none !important; }}
L2 evaluation recommendations: generate each requirement 3 times, have 2+ reviewers score them blind, and take the median; also run automated tools to check hard metrics like contrast, tag closure, and build artifact size. Gaps on the aesthetic dimension often separate models more sharply than the correctness dimension does.
5. Tier 3 (L3): Agentic Closed Loop and Self-Correction
Goal: in a realistic engineering environment, evaluate whether the model can autonomously complete the loop of “read code → modify code → run verification → fix.”
Typical tasks:
- Given a failing test case, locate the root cause and fix it (modifying unrelated files is not allowed);
- Add a validated, logged endpoint to an existing service and fill in the tests;
- Perform a small-scope refactor within a specified module while keeping behavior equivalent.
The host environment for the loop must provide: a writable filesystem, a restricted terminal (build/test/static checks), truncatable error output, and logged traces of operations.
MAX_ROUNDS = 5def agentic_fix(task, runner, ctx_window=64_000):history = []for rnd in range(1, MAX_ROUNDS + 1):prompt = build_prompt(task, history)[:ctx_window] # keep the context budget in checkpatch = model.generate(prompt)result = runner.verify(patch) # compile + unit tests + lintif result.passed:return {"patch": patch, "rounds": rnd, "ok": True}history.append({"patch": patch,# keep only the error tail and failing case names to avoid polluting the context"stderr": result.stderr[-4000:],"failed": result.failed_cases[:10],})return {"ok": False, "rounds": MAX_ROUNDS}
L3’s four key metrics:
- First-pass rate: the proportion that succeeds in one round, directly reflecting base capability;
- Average rounds to convergence: reflects self-correction efficiency; more than 3 rounds usually means the error summary wasn’t good enough;
- Regression introduction rate: whether the change introduces new failures in other tests — the most easily overlooked and most dangerous metric;
- Token and time cost per task: determines the unit economics of each task.
6. Making Evaluations Reproducible: Four Engineering Disciplines
- Fix randomness: set generation temperature to 0, and keep a small amount of resampling to observe stability;
- Fix the budget: standardize max rounds, max context length, and per-task timeout;
- Fix the case set and maintain it in tiers: version L1/L2/L3 independently, and avoid leaking cases into public corpora;
- Triple verification: automated assertions (correctness) + static and visual metrics (delivery quality) + human blind review (experience). None can substitute for another.
Conclusions should be written as a single actionable statement, for example: “92% pass rate on the L1 case set, blind-review median of 4.1/5 on L2, and L3 averaging 1.8 rounds to convergence with a 3% regression introduction rate — therefore suitable as the default coding model, with human review retained for high-complexity refactoring tasks.”
7. Common Pitfalls and Implementation Advice
- Extrapolating overall capability from a single case: one pretty demo proves nothing about stability; the sample must cover at least three classes of tasks, with multiple generations each.
- Treating “correct on the first try” as the only goal: in engineering practice, verifiability and regression-safety matter more than one-shot correctness.
- Ignoring context engineering: stuffing the whole repository into the model doesn’t improve results — it amplifies noise and cost. Curating relevant files, call chains, and type contracts is what matters.
- Overlooking compliance and security: data redaction before uploading code, license risk in generated content, and sandbox isolation should all be settled at the process-design stage, not patched afterward.
Recommended rollout path: start with the L1 case set for a low-cost screen to narrow down to 2–3 candidates; then use L2 blind review to decide how to split interactive and front-end tasks; finally roll out L3 gradually against a real repository, routing models by task difficulty — simple tasks to lightweight, cost-efficient models, and complex refactors to strong-reasoning models. Model capabilities can be plugged into a unified model-service platform (such as Baidu AI Cloud’s large model service API) as needed, with switching and fallback handled on the configuration side so business code needs no changes.
8. Conclusion
The value of three-tier gradient testing isn’t to rank models, but to convert the vague judgment of “can we use it?” into a set of metrics that are measurable, repeatable, and ready to write into a technical decision document. Once evaluation is reproducible, arguments over selection naturally converge into comparisons of data — and in an era of rapidly iterating large models, that is exactly the infrastructure capability engineering teams should build.
