Evaluating LLM Coding Capabilities: A Reproducible Framework for Frontend Interaction and Algorithms
A practical, engineering-oriented method for evaluating how well large language models (LLMs) write code, moving beyond first-impression judgments. It introduces a reproducible scoring framework built around two representative tasks—a single-file Pomodoro timer (frontend interaction) and an interval-merging/max-concurrency problem (algorithms)—and details fixed inputs, first-pass versus fix-round scoring, repeated runs, blind review, environment logging, test procedures, a defect checklist, result interpretation, a reference implementation, and team-level adoption practices.
Evaluating LLM Coding Capabilities: Frontend Interaction and Algorithms
1. Start with the conclusion: don’t judge from a single “looks good” result
When people discuss an LLM’s coding ability, the most common mistake is to hand the same prompt to a few candidate models, generate a snippet from each, and then rank them based on first-glance completeness, visual polish, or a subjective gut feeling. Such conclusions are usually unreliable, because at least the following factors interfere:
- Entry-point differences: the same platform may expose multiple model entry points, free and paid tiers, and normal versus deep-thinking modes. The capabilities actually invoked are not necessarily identical.
- Non-deterministic generation: the same prompt run multiple times can produce noticeably different results, so a single sample cannot represent a stable level.
- Task bias: frontend UI tasks are easily swayed by aesthetics, while algorithm tasks depend heavily on test coverage. Without a unified scoring standard, conclusions drift toward whichever output “looks more agreeable.”
- Missing acceptance criteria: code that displays is not code that runs; code that runs is not code where state transitions, invalid inputs, and boundary conditions all behave correctly.
- Version drift: model versions and product entry points change over time, so today’s evaluation results should not be treated as a long-term basis for selection.
A more reasonable approach is to break “model comparison” down into a reproducible engineering-evaluation problem: under fixed tasks, fixed inputs, and fixed acceptance criteria, observe whether a candidate can consistently produce code that runs, is maintainable, and can be fixed.
This article uses two directions:
- Frontend interaction: implement a Pomodoro timer in a single HTML file, with a custom countdown, pause/resume/reset, an end-of-session alert sound, and a background that gradually shifts color over time.
- Algorithm ability: an interval-merging and max-concurrency problem, used to assess correctness, boundary handling, complexity analysis, and testability.
The evaluation discusses only a general method and scoring framework; it does not center on any specific third-party product.
2. Evaluation design principles
2.1 Fixed task and fixed input
All candidate samples use the same prompt, with no follow-up explanation allowed in the first round. This avoids the situation where, after coaxing an answer out over multiple rounds of prompting, prompt-engineering skill gets mistaken for one-shot code-generation ability.
2.2 Separate first-pass scoring from fix-round scoring
The first round considers only the initial generated result. The fix round then supplies a specific failure symptom—for example, “starting again after pausing creates two timers,” “it still starts after entering abc,” or “no sound plays when the countdown ends”—and observes the model’s ability to locate and modify the code.
The first-pass score reflects initial understanding and implementation; the fix score reflects debugging, regression, and engineering-collaboration ability. The two must not be conflated.
2.3 Generate multiple times and record the variance
Run the same prompt at least two or three times, recording the main defects and score of each run. You can then look at the average, or at the lowest score. For production selection, the lowest score often matters more than the average, because a single serious state error can trigger rework.
2.4 Blind evaluation and two-person review
Assign numbers to the generated results and hide their source before handing them to reviewers. One person runs the functional tests, another performs the code review, and the subjective aesthetic score is given independently by both reviewers and then averaged. When a large disagreement appears, return to the scoring rubric and recalibrate.
2.5 Record explicit environment information
At minimum, record: evaluation date, entry-point type, whether it is paid, whether the model can be switched, whether coding or deep-thinking mode is enabled, output-length limits, browser version, and whether network access is allowed. For entry points whose exact version cannot be determined, mark it as “version uncertain” and do not treat the result as a long-term conclusion about a fixed model.
3. Frontend task: a single-file Pomodoro timer
3.1 The prompt
Please implement a web-based Pomodoro timer in a single HTML file. Requirements:1. Minimalist design, a clean interface, but not crude;2. Support a custom countdown duration;3. Clicking Start begins the countdown, with support for pause, resume, and reset;4. When the countdown ends, play an alert sound and show a completed state;5. The background color shifts smoothly as time passes;6. Validate the contents of the input field; invalid input must show a clear message and prevent starting;7. Do not reference any external CSS, JS, fonts, images, or network resources; it must run directly offline.
This task looks simple, but it covers a lot of ground: DOM events, a state machine, timer management, input validation, audio playback, CSS gradients, responsive layout, accessibility, and the single-file constraint all come into play.
3.2 Scoring rubric
The main score totals 50 points, plus 15 bonus points, normalized to a 100-point scale.
| Dimension | Points | Criteria | Common point losses |
|---|---|---|---|
| Usability | 10 | Clicking Start enters the countdown and the displayed time decreases; if it cannot start, 0 points | Buttons unresponsive, timer not started, page errors |
| Completeness | 10 | Includes custom time, start, pause, resume, reset, end alert sound, and completed state; deduct 5 points for each missing key feature, down to a floor of 0 | Missing pause or reset, cannot restart after finishing, timer keeps running after pause |
| Aesthetics | 10 | Minimal yet with visual hierarchy, reasonable spacing, sufficient contrast, no overflow on mobile | Central card too crude, crowded buttons, muddled font-size hierarchy |
| Defects | 10 | Starts at 10; deduct 5 per confirmed reproducible defect, floor of 0 | Repeated start creates multiple timers, reset fails to clear state, invalid input still runs |
| Dynamic gradient | 10 | Background changes with time progress; a static gradient alone scores 5; smooth, continuous, and tied to remaining time scores 10 | Background set only once, jumps every second, gradient unrelated to progress |
| Validation and robustness | Bonus 5 | Empty, non-numeric, negative, zero, and excessively large values all show clear messages; a visual fallback exists when audio is blocked by the browser | Loose conversion only, missing error messages, no boundary limits |
| Defect fixing | Bonus 10 | Reproducing and locating the problem: 5 points; fixing along the original approach with regression passing: 10 points; fixing with a different reasonable approach: 8 points | Only fixing surface styling, not clearing the old timer, introducing new problems after the fix |
The normalization formula can be written as:
Final score = (Main score + Bonus) / 65 * 100
If the fix round is not run, count only validation and robustness for the bonus, and present fixing ability in a separate list, so that “can fix a bug” is not mixed together with “generated correctly in one shot.”
4. How to run the tests
4.1 First-round functional test
Save the single HTML file locally, open it in a browser while offline, and open the console to watch for errors.
Run the following scenarios with emphasis:
- Click Start with the default time; confirm the countdown starts immediately.
- Set a short custom time, for example 10 seconds or 1 minute, and verify the completed state and alert sound.
- While running, click Pause, wait a few seconds, and confirm the displayed time stops changing; after Resume, confirm it continues from the pause point.
- While running, click Reset, and confirm the time returns to its initial value and the state returns to startable.
- Click Start repeatedly in quick succession; confirm no multiple parallel timers appear.
- Enter an empty string, letters, a negative number, 0, a very large value, and a decimal into the input; confirm each shows a clear message and prevents starting.
- After the countdown ends, click Start again and confirm it does not reuse the old state.
- Resize the browser window to mobile width; confirm the buttons, input, and time display do not overflow.
- Observe whether the background gradient changes continuously over time, rather than being a static gradient only at the start.
4.2 Code review
After the functional test passes, review the code structure. Focus on:
- Whether a unified state machine is used, for example idle, running, paused, finished.
- Whether the timer is cleared on pause, reset, and completion.
- Whether remaining time is computed from an absolute end time, to avoid
setIntervaldrift. - Whether input validation is encapsulated as a separate function.
- Whether audio capability is initialized on the user’s first click, to avoid autoplay restrictions.
- Whether all resources and styles live inside the single file.
- Whether key elements have accessibility labels and state announcements.
4.3 Common defect checklist
In practice, the following problems appear most often:
- Using a decrementing counter instead of a deadline, so timing slows noticeably after a background tab is throttled.
- On pause, only changing the button text without actually clearing or suspending the timer.
- On reset, failing to clear the old timer, so restarting results in double-decrementing.
- Using only loose type conversion for input validation, so something like
12abcis still treated as 12. - Trying to play the end alert before the user has interacted, so the browser blocks it.
- Setting the background gradient only in the initial CSS state, without updating it over time.
- Jumping the gradient once per second, which looks visually discontinuous.
- Buttons too small on mobile, text overflow, insufficient contrast.
- No keyboard operability, and no readable state announcement when the countdown changes.
- Claiming to be a single file while actually referencing external fonts, icons, or scripts.
5. Algorithm task: interval merging and max concurrency
Beyond the frontend task, an additional pure algorithm task should be included, to keep the evaluation from being dominated by visual factors. A problem like this can be used:
Given an array of time intervals, where each element is [start, end), implement a function that:1. Merges all overlapping intervals;2. Returns the merged interval array;3. Returns the maximum concurrency at any point in time.Note: intervals are left-closed and right-open, so [1,2) and [2,3) do not overlap.Requirements: analyze time and space complexity, and cover cases including an empty array, a single interval, invalid intervals, negative times, and intervals that touch at their endpoints.
Suggested scoring:
| Dimension | Points | Criteria |
|---|---|---|
| Correctness | 50 | Both the merged result and the max concurrency are correct |
| Boundary handling | 20 | Empty input, single element, touching endpoints, and invalid intervals are all handled |
| Complexity | 15 | Can explain sorting overhead and avoids an obviously inefficient implementation |
| Readability | 10 | Clear naming, layered logic, appropriate comments |
| Verifiability | 5 | Provides examples, self-tests, or test cases |
Common errors include: treating [1,2) and [2,3) as overlapping; mishandling the order of end and start events at the same position during a sweep-line sort; returning only the merged intervals and forgetting the max concurrency; failing to handle invalid intervals where start >= end; and giving only code without a complexity analysis.
6. How to interpret results
At an abstract level, candidate results usually fall into three categories:
- Basically usable: the page opens and the Start button works, but pause, reset, and invalid-input messages are missing, and the background gradient is mostly static. Such results do not score low on usability but clearly fall short on completeness.
- Feature-complete: the main interactions are all present and the code runs, but the aesthetics are template-like, and boundary handling and state cleanup are not rigorous. Such results are suitable for proceeding to the fix-round evaluation.
- Engineering-robust: the state machine is clear, pause/resume/reset behave consistently, input validation is explicit, the gradient is tied to progress, audio has a fallback, and readability is good. Such results not only “can be demoed” but are closer to deliverable code.
Note, however, that paid versus free entry points, whether deep thinking is enabled, output-length limits, and whether tool calls are available all affect the outcome. A single web-chat result cannot simply be equated with a model’s inherent ability. For formal selection, treat the public experience as a first-pass filter, then combine it with private business samples, API-based regression tests, cost, latency, data compliance, and stability for an overall judgment.
7. A reference implementation skeleton
Below is a simplified illustration of the core timer logic. It emphasizes state management and smooth gradients and does not depend on external resources:
const state = {mode: "idle", // idle | running | paused | finishedtotal: 25 * 60 * 1000,remaining: 25 * 60 * 1000,endAt: 0,raf: 0};function render() {const progress = 1 - state.remaining / state.total;const hue = 200 + Math.max(0, Math.min(1, progress)) * 120;document.documentElement.style.setProperty("--hue", String(hue));document.querySelector("#time").textContent = format(state.remaining);}function tick() {state.remaining = Math.max(0, state.endAt - performance.now());render();if (state.remaining <= 0) {finish();return;}state.raf = requestAnimationFrame(tick);}function start() {if (state.mode === "running") return;if (state.mode === "idle" || state.mode === "finished") {state.remaining = state.total;}state.endAt = performance.now() + state.remaining;state.mode = "running";state.raf = requestAnimationFrame(tick);}function pause() {if (state.mode !== "running") return;state.remaining = Math.max(0, state.endAt - performance.now());state.mode = "paused";cancelAnimationFrame(state.raf);render();}function reset() {cancelAnimationFrame(state.raf);state.mode = "idle";state.remaining = state.total;render();}
Input validation should be handled separately:
function readMinutes(input) {const value = Number(input.value.trim());if (!Number.isFinite(value) || value <= 0 || value > 1440) {throw new Error("Please enter a number of minutes between 1 and 1440");}return value;}
The key point is not the code itself, but that during evaluation you check whether these designs are actually implemented: whether there is a state machine, whether old tasks are cleared, whether remaining time is computed from an absolute time, whether input errors are explicitly fed back to the user, and whether a visual cue is provided when audio playback is not available.
8. Adoption recommendations
For teams and enterprise users, it is recommended to turn LLM coding evaluation into an internal standard:
- Build a fixed question bank: at least one frontend interaction task, one algorithm task, one defect-fixing task, and one security/exception-handling task.
- Build a unified scoring sheet: score functionality, boundaries, maintainability, defect fixing, and explanation quality separately.
- Record version context: note the entry point, mode, date, and limitations, so that outdated conclusions are not cited long-term.
- Validate with regression cases: don’t just look at the generated code—actually run it and execute tests.
- Treat generated results as candidate implementations: they must pass code review, unit tests, security scanning, and human confirmation before entering production.
- For critical business scenarios, use API-based, repeatable regression tests rather than relying on a one-off web-chat experience.
In summary, “which model is easier to use” should not be the only question. The more engineering-valuable question is: under a clear task, a clear mode, and clear acceptance criteria, can a candidate consistently produce runnable code, can it be covered by tests, and can defects be quickly located and fixed once they appear. Using the single-file Pomodoro timer to probe frontend interaction, using interval merging to probe algorithmic thinking, and then recording results with a unified scoring rubric is what turns subjective experience into reusable technical selection criteria.
