Three-Tier Gradient Evaluation: Assessing LLM Code Generation and UI Aesthetics
A reproducible three-tier framework for evaluating LLM code generation. Tier 1 uses a 60–120 problem evaluation set with pytest automation to measure first-pass rate, boundary pass rate, and runnable-as-is rate. Tier 2 applies a weighted scoring sheet covering correctness, robustness, maintainability, and security against a checklist of implicit engineering constraints. Tier 3 combines machine checks and a 100-point manual rubric for UI aesthetics, plus dimensions for scoring complex system design. The article also covers cost estimation formulas, tiered model routing by task difficulty, common evaluation mistakes, and failure modes, giving teams a decision basis they can record, compare, and revisit instead of relying on demos or a single benchmark score.
Key Takeaways: What This Framework Delivers
When evaluating an LLM’s code generation ability, developers should not judge by demos alone, and should not trust a single benchmark score either. A more reliable approach breaks “can it write code?” into three verifiable tiers: basic functional correctness, logic robustness, and UI aesthetics plus complex system design ability. This article presents a reproducible three-tier gradient evaluation framework that gives you:
- An evaluation-set design template;
- Tier 1 automated testing criteria;
- A Tier 2 multi-dimensional scoring sheet;
- Tier 3 UI aesthetic and architecture scoring items;
- Cost estimation and a tiered routing approach;
- Common misuse warnings and selection guidance.
If your team is choosing a code generation model, a coding assistant, or a UI generation model, this framework turns subjective impressions into a decision basis you can record, compare, and revisit.
Tier 1: First Verify It Can Actually Run Correctly
The goal of Tier 1 is not to prove how powerful the model is, but to confirm whether it can complete well-defined programming tasks with clear boundaries and verifiable results. This tier best fits high-frequency, standardized development scenarios: sorting and searching, string manipulation, date formatting, collection operations, regular expressions, simple SQL, data structure utility functions, and similar work.
Designing the Evaluation Tasks
Build a fixed evaluation set of 60 to 120 problems. Each problem should contain at least the following fields:
- Task description: state the function’s goal in natural language;
- Input samples: cover normal, empty, and boundary inputs;
- Expected output: used for assertion checks;
- Test code: executed automatically with pytest or unittest;
- Scoring criteria: whether it passes on the first attempt, and whether manual fixes are needed.
Keep tasks close to real business work rather than pure algorithm puzzles. Instead of only asking the model to “implement quicksort,” design something like “given a list of orders, sort them by creation time, and handle an empty list and invalid time formats.”
Running the Evaluation Automatically
Prefer automated testing for Tier 1 to reduce human judgment. With pytest, you can run it locally or in CI:
pytest -q tests/test_grad1.py
For cross-model comparison to be meaningful, hold these variables fixed:
- Model name and version;
- API version;
- Temperature, set to 0 or close to 0;
- top_p, max output length, and timeout;
- System prompt;
- Whether tool calling or web retrieval is allowed.
Only once these conditions are fixed are results across models comparable.
Example: The Even-Squares Function
Suppose the task asks the model to write a function that takes a list which may contain nulls and returns the squares of the even numbers in it. An empty list returns an empty list, and None also returns an empty list.
The model generates something like this:
def even_squares(nums):if not nums:return []return [x * x for x in nums if isinstance(x, int) and x % 2 == 0]
The evaluation test looks like this:
from solution import even_squaresdef test_even_squares():assert even_squares([1, 2, 3, 4]) == [4, 16]assert even_squares([]) == []assert even_squares(None) == []assert even_squares([0, -2, -3]) == [0, 4]
The key point: the test code comes from the evaluation set, and the function under test is generated by the model. If the model’s output cannot run directly, mark it as failed. If it only lacks a few type annotations or some error handling, it can move on to Tier 2 for further assessment.
Tier 1 Scoring Criteria
Record three metrics:
- First-pass rate: the share of tasks that pass all test cases without any manual modification;
- Boundary pass rate: the share of boundary cases such as null values, very long inputs, and invalid parameters that pass;
- Runnable-as-is rate: the share of generated code that runs without adjusting dependencies, function names, or entry parameters.
Suggested pass lines: a first-pass rate of at least 90% on standard cases and a boundary pass rate of at least 75%. If the model frequently fails basic scenarios, it is not yet suitable as the default code generation entry point.
Tier 2: Is It Stable, Maintainable, and Shippable?
Tier 1 only shows that the model “can write code.” Tier 2 determines whether it can handle the complex constraints of real engineering. Typical tasks here include transactional business logic, concurrency control, caching strategies, error retries, logging, legacy code refactoring, and multi-step algorithmic reasoning.
Implicit Constraint Checklist
Real requirements rarely state every condition up front. During evaluation, watch whether the model proactively identifies implicit constraints. A useful checklist:
- Concurrency: does it consider thread safety, locking, and race conditions;
- Transactions: does it consider commit, rollback, and idempotency;
- Caching: does it consider invalidation, cache penetration, and cache avalanche;
- Input validation: does it check for nulls, types, length, and format;
- Error handling: does it distinguish recoverable from unrecoverable exceptions;
- Logging: does it log critical paths and avoid leaking sensitive data;
- Timeouts and retries: does it set timeouts and avoid infinite retries;
- Authorization: does it enforce access control and avoid privilege escalation.
If the model produces code that merely looks reasonable while ignoring key constraints, it should not receive a high score even when the syntax is correct.
Tier 2 Multi-Dimensional Scoring Sheet
Use weighted scoring rather than subjective impression alone:
| Dimension | Weight | 5 points | 3 points | 1 point |
|---|---|---|---|---|
| Correctness | 40% | Complete functionality, covers the main business constraints | Main flow correct, but some boundaries missed | Core logic wrong |
| Robustness | 25% | Validation, error handling, logging, and retry strategy | Basic validation only, missing key error handling | No input validation or error handling |
| Maintainability | 20% | Clear naming, single responsibility, follows conventions | Readable structure, but with duplication or coupling | Messy structure, hard to maintain |
| Security | 15% | Validates input by default and avoids common injection and authorization risks | Some security awareness but insufficient coverage | Obvious security flaws |
Weighted calculation example: a model scores 5 on correctness, 4 on robustness, 3 on maintainability, and 4 on security, giving:
5 * 40% + 4 * 25% + 3 * 20% + 4 * 15% = 4.15
Suggested pass line for Tier 2: a weighted score of at least 4.0. Below that line, the model may be suitable for code completion but should not directly generate core business modules.
Common Deductions
To keep reviews consistent, agree on deductions in advance:
- Not handling empty lists, empty strings, or nulls: −1 point;
- Repeated computation inside a loop causing noticeable performance problems: −1 point;
- Hardcoded configuration: −1 point;
- Logging sensitive information: −2 points;
- No timeout control: −2 points;
- No rollback handling on failure: −2 points;
- Code runs but the semantics deviate from the requirement: −3 points.
Review records should keep the problem ID, score, deduction reason, and improvement suggestions so changes across model versions can be tracked later.
Tier 3: UI Aesthetics and Complex System Design
Tier 3 observes whether the model has higher-level delivery ability, mainly covering frontend UI generation, componentized design, responsive layout, interaction state handling, and complex system architecture design.
UI Aesthetics Is More Than “Does It Look Right”
A common mistake in frontend generation tasks is judging only whether the page opens. A more reasonable evaluation looks at:
- Whether semantic elements are used;
- Whether there is a clear visual hierarchy;
- Whether the layout is aligned;
- Whether mobile and desktop are both considered;
- Whether loading, empty, error, and disabled states exist;
- Whether there is componentized thinking rather than piled-up tags;
- Whether obvious visual crowding or excessive empty space is avoided.
Machine-Checkable Items
For UI generation, start by checking basic quality automatically:
- Whether the HTML is syntactically valid;
- Whether semantic elements such as
header,main,nav, andsectionare used; - Whether basic ARIA attributes are present;
- Whether the CSS compiles;
- Whether there are obvious console errors;
- Whether at least two responsive breakpoints are included;
- Whether it passes a build or preview command.
Machine checks cannot fully represent aesthetics, but they filter out low-quality output.
Manual UI Aesthetic Scoring Sheet
Use a 100-point scale for manual review:
| Item | Points | Evaluation focus |
|---|---|---|
| Semantic structure | 20 | Reasonable use of elements and clear hierarchy |
| Visual hierarchy | 15 | Clear prominence among headings, body text, and buttons |
| Layout alignment | 15 | Stable spacing, alignment, and grid |
| Responsive adaptation | 15 | Readable on both mobile and desktop |
| Interaction states | 20 | Complete hover, loading, empty, and error states |
| Componentization | 15 | Reusable and free of duplicated code |
A UI task should score 80 points or more on the manual evaluation before entering production trial. If the team has the capacity, use two reviewers and bring in a third when the scores diverge significantly.
Scoring Complex System Design
Complex system design usually has no single correct answer, but you can still assess whether the model thinks like an engineer. Dimensions to score:
- Whether the layering is clear;
- Whether module responsibilities are well defined;
- Whether extensibility is considered;
- Whether interface contracts are reasonable;
- Whether the data model is implementable;
- Whether key risks are identified;
- Whether over-engineering is avoided.
For example, ask the model to design a high-concurrency order service. Do not just check whether it lists a database, a cache, and a message queue; check whether it explains the boundaries: which operations must be strongly consistent, which can be eventually consistent, which endpoints need idempotency, and which paths need degradation.
Cost Trade-offs: Decide with Verifiable Numbers
No matter how capable the model is, invocation cost matters. Do not judge cost by general impression; compute it from verifiable data.
Base the estimate on the official pricing page and combine the following factors:
- Input token price;
- Output token price;
- Average input length;
- Average output length;
- Daily call volume;
- Concurrency scale;
- Retry rate;
- Cache hit rate;
- Human review cost caused by task failures.
Per-call cost can be estimated roughly as:
cost per call = input tokens * input price + output tokens * output pricemonthly cost = cost per call * call volume * (1 + retry rate)
Tiered Routing Example
A common engineering practice is to route tasks of different difficulty to different models or capability tiers:
def route_task(task):if task.gradient == 1 and task.risk_level == "low":return "light_model"if task.gradient == 2 or task.token_estimate < 3000:return "balanced_model"return "strong_model"
These thresholds are not fixed standards; adjust them based on your own evaluation results. For example:
- Tier 1 tasks with a high pass rate and low risk can be routed to a light model;
- Tier 2 tasks involving transactions, concurrency, or authorization should use a more stable model;
- Tier 3 UI design or architecture planning should allow stronger models paired with human review.
The goal of tiered routing is not to cut cost blindly, but to optimize total cost while keeping quality risk acceptable.
Common Mistakes and Selection Guidance
Common Mistakes
- Judging only by demo cases: demos are curated and cannot represent real task performance.
- No boundary cases: testing only normal inputs hides flaws the model will show on real, messy data.
- Not pinning the model version: results fluctuate widely across versions, making it impossible to tell whether there is improvement.
- Setting temperature arbitrarily: too much randomness makes evaluation results unstable.
- Relying only on subjective impressions: without a scoring sheet and records, results are hard to revisit.
- Ignoring the cost model: looking only at per-call price while ignoring retries, manual fixes, and latency costs.
- Neglecting security and compliance: when code generation involves injection, privilege escalation, or sensitive data, it must be evaluated separately.
How to Choose More Safely
- Use Tier 1 to filter out models whose basic ability is insufficient;
- Use Tier 2 to judge whether the model can enter business systems;
- Use Tier 3 to assess frontend, interaction, and architecture ability;
- Combine the official pricing page with real call volume to compute cost;
- Keep iterating the evaluation set with real business cases.
Failure Mode Warnings
Watch especially for these failure modes:
- Passes on idealized input, but breaks the moment real data runs;
- Performs well on simple completion, but frequently misses constraints on complex tasks;
- UI that opens, but code that is hard to maintain;
- Architecture diagrams that look complete but cannot be mapped to interfaces and data models;
- A cheap per-call price paired with a high retry rate, so total cost is actually higher.
Conclusion: Turn Evaluation into a Continuous Process
Evaluating an LLM’s code generation ability is fundamentally about judging whether it can become reliable productivity. The three-tier gradient method is not about chasing any model’s absolute score; it helps teams build a stable decision process, moving from basic correctness, to engineering robustness, to UI aesthetics and system design, with clear tasks, scoring criteria, and record-keeping at every step.
The genuinely valuable practice is to bake the evaluation set, test commands, scoring sheets, cost formulas, and routing rules into the process and keep iterating with real business cases. That way, even as models update quickly, the team can rapidly judge which option fits its own scenarios and find a more reliable balance among quality, cost, and maintainability.
