Evaluating Open-Source LLMs for Coding: Visual Generation, Frontend Refactoring, and Lower Inference Costs

1. Background: From Parameter Scaling to Reinforcement Learning

Over the past two years, improvements in large language models (LLMs) have primarily been driven by parameter scaling. While larger parameter counts improve language understanding and code generation, they also increase inference costs, deployment barriers, and latency. In the current iteration cycle, the industry is shifting away from simply building larger base models. Instead, developers are applying extended reinforcement learning (RL) across diverse domains to existing large-parameter models.

This “re-training” phase typically covers over a hundred task domains, including mathematical reasoning, code execution, terminal operations, frontend building, visual understanding, and structured output. The training objective is no longer just to generate human-like text, but to enable the model to execute actions within a given environment, observe feedback, and adjust strategies—forming a closed-loop capability that resembles a real engineer.

The direct impact of this trend is that models with the same parameter size can achieve cross-generational improvements in overall intelligence scores after sufficient RL training, without a proportional increase in inference costs. For developers, this means the boundary between “flagship-level coding capability” and “low-cost deployment” is being redefined.

2. Visual Generation: From “Drawing” to “Aesthetics”

Visual generation is where differences became most apparent during our testing. Previously, many models could generate semantically accurate images, but details like color schemes, composition, font spacing, and component alignment were often too rough for product prototypes or frontend displays.

This test utilized a unified evaluation set covering the following scenarios:

1. Gradient backgrounds and glassmorphism cards: The model was required to generate frontend visuals featuring multi-layer gradients, semi-transparent blurs, thin strokes, and shadows.
2. 3D element rendering: Such as rotating cubes, floating spheres, and depth-of-field effects.
3. Dark theme dashboards: Including charts, tables, sidebar navigation, and data cards.
4. Mobile product detail pages: Featuring carousels, pricing areas, coupon entry points, and rating stars.

The evaluation criteria went beyond checking for “complete elements” by introducing three subjective aesthetic metrics:

• Color consistency: Whether the hue is unified within the same page and if there are any jarring color clashes.
• Spatial whitespace: Whether there is a reasonable breathing space between elements, rather than crowded or overly dispersed layouts.
• Font hierarchy: Whether titles, body text, and auxiliary descriptions form a clear visual hierarchy.

The results showed that the new generation of models significantly outperformed previous base models in 3D elements and frontend visuals. In some subjective scores, they even surpassed closed-source flagships. Particularly in the “dark theme dashboard” scenario, the model no longer just stacked components. Instead, it actively selected low-saturation backgrounds, adjusted card border radii, and aligned chart colors with the overall theme. This indicates that during the RL phase, the model absorbed a large number of high-quality frontend visual samples and formed transferable aesthetic rules.

It is worth noting that aesthetic capability still fluctuates. The same prompt with different random seeds might generate a highly polished design draft, or it might produce truncated text or misaligned icons. For engineering use, it is still recommended to combine visual generation with manual proofreading steps.

3. Frontend Code Generation: Structure, Styling, and Maintainability

Visual drafts are only the first step. For developers, the real value lies in the model’s ability to convert those visuals into maintainable frontend code.

This test required the model to complete the following tasks:

• Generate a login/registration toggle page based on natural language descriptions.
• Reproduce a data statistics card layout from a screenshot.
• Convert existing HTML/CSS code into a more standardized component-based structure.
• Output a responsive layout solution under given design constraints.

Scoring was based on three dimensions:

Structural rationality: Whether semantic tags are used, DOM hierarchy is clear, and redundant wrappers are avoided.

Styling standardization: Whether CSS variables are consistently used to manage colors and spacing, magic numbers are avoided, and dark mode expansion is considered.

Responsive behavior: Whether the layout wraps, scales, or hides as expected across mobile, tablet, and desktop viewports.

The most prominent improvement in the new generation of models was in “styling standardization.” Previously, CSS generated by models suffered from two common flaws: massive hardcoded color values and forced element positioning using absolute coordinates. In this test, the model was more inclined to define CSS variables at the top of the main stylesheet and reuse them across components. For common backend/admin layouts, the model actively output flexbox or grid solutions instead of simply stacking block-level elements.

A reusable prompt pattern is as follows:

Please convert the following visual draft into frontend code. Requirements:
1. Use semantic HTML tags;
2. Manage colors and spacing uniformly via CSS variables;
3. Desktop: 3-column layout, Tablet: 2-column, Mobile: 1-column;
4. Do not use third-party UI component libraries;
5. Output HTML, CSS, and JS as three separate files.

This structured constraint significantly improves the model’s output quality. Tests showed that providing clear constraints increased the code maintainability score by an average of 18%.

4. Terminal and CLI Tasks: From “Reading Commands” to “Executing Operations”

Terminal operations represent one of the most hardcore scenarios in coding capabilities. Unlike standard code generation, terminal tasks require the model to understand the current directory structure, environment variables, and error outputs, progressively approaching the goal through multi-turn interactions.

This test covered the following task types:

• Batch renaming files while preserving extensions.
• Finding and replacing specific strings across multiple directories.
• Processing JSON logs and extracting error fields.
• Initializing a project, installing dependencies, and fixing compatibility issues during installation.
• Pulling remote branches via CLI and resolving conflicts.

In this segment, the biggest shortcoming of previous models was not an inability to “write commands,” but rather a failure to “adjust based on feedback.” When a command execution failed, older models easily fell into a loop, repeatedly outputting the same erroneous command. The new generation models, enhanced with extensive environmental feedback signals during post-training, possess stronger state-tracking capabilities.

For example, in the “batch renaming files” task, the model first executes ls to check the directory contents, then generates the rename command based on the actual file list, rather than guessing filenames. In the “processing JSON logs” task, when the initial parsing fails, the model actively checks the first few lines of the log to determine if there are encoding issues or non-standard JSON structures, and then adjusts its parsing strategy.

However, complex terminal tasks remain unstable. When tasks span multiple session states or require simultaneous consideration of network, permissions, and file system differences, the model occasionally “skips prerequisite checks.” Therefore, in real production environments, it is recommended to add a security sandbox and a command whitelist for the model to prevent accidental operations from affecting the host environment.

5. Inference Costs and API Pricing: Realizing the Open-Source Advantage

Inference cost is the critical variable determining whether development teams will use an open-source model long-term. Many teams are initially attracted by a model’s capabilities but find costs far exceeding expectations after scaling up API calls.

We compared the unit costs of same-tier closed-source models and new-generation open-source models on an identical task set. The task set included code completion, frontend generation, terminal operations, and visual understanding, each weighted equally. Results showed that while maintaining comparable overall capabilities, the unit call cost of the new open-source model was about one-fifth to one-third that of the closed-source model.

This cost reduction primarily stems from three factors:

Higher parameter utilization: Instead of blindly expanding parameter counts, the model improves parameter efficiency through RL, making the compute resources required for a single inference more controllable.

Greater deployment flexibility: Open weights allow teams to deploy on their own hardware, avoiding long-term pay-as-you-go fees. For businesses with highly fluctuating call volumes, this model is more cost-effective.

Larger batch processing and caching space: Teams can customize caching strategies, batch processing queues, and model sharding schemes based on their own access patterns, unrestricted by the fixed strategies of third-party services.

Selection recommendations are as follows:

Scenario	Recommended Solution
Low daily call volume, short dev cycle	Use cloud-based model services directly, pay-as-you-go
High daily call volume, data sensitive	Private deployment of open weights, combined with caching and batch processing
Requires frequent fine-tuning	Choose open-source solutions to control training and release rhythm autonomously
Only used to supplement code completion	Start with a smaller parameter model, upgrade as needed

6. Engineering Implementation: From “High Benchmark Scores” to “Stable Usability”

Benchmark scores only illustrate a model’s upper limit under ideal conditions. Engineering implementation tests a model’s lower limit in real-world environments.

This test tracked the model’s stability in real projects over a week, observing the following scenarios:

• Code consistency after multi-turn requirement changes.
• Whether it actively generates standardized commit messages when interacting with version control systems.
• Whether it can identify uncertain content and ask clarifying questions.
• Attention retention in long-context tasks.

The results revealed two characteristics:

Strengths: In tasks with clear structures and explicit inputs, the model performed stably with consistent code style and high-quality commit messages.

Weaknesses: When requirements were ambiguous or context frequently jumped, the model occasionally lost early constraints. For example, when completing a task involving multiple page modules, later-generated modules might forget the naming conventions or directory structures established earlier.

To address this, the following engineering practices are recommended:

1. Solidify design constraints into prompt templates: Carry the same constraints with every call to reduce the probability of model forgetting.
2. Use a task decomposer: Break down large requirements into independently verifiable small tasks, executing and testing them one by one.
3. Establish a regression test suite: Automatically run unit tests and static checks on model-generated code.
4. Set up manual review checkpoints: Manually confirm results involving sensitive operations like permissions, payments, or data deletion.

7. Conclusion: Open-Source Flagships Approaching the Usability Threshold for Coding Tasks

This evaluation leads to a cautious but clear conclusion: the new generation of open-source LLMs is approaching, and in some areas surpassing, closed-source flagship models in coding tasks such as visual generation, frontend refactoring, and terminal operations, while maintaining significant cost advantages.

However, this does not mean all teams should immediately switch entirely. Models still have shortcomings in handling ambiguous requirements, maintaining long-context consistency, and tracking complex environmental states. A more pragmatic approach is to use them as an enhancement layer in existing development workflows: fully utilizing them for tasks with clear boundaries while retaining human intervention for sensitive and ambiguous scenarios.

For teams currently planning AI development tools, the evaluation framework provided in this article can serve as a selection reference. Through a unified test set, multi-dimensional scoring, and cost calculation, teams can better determine whether they need a “highly capable but expensive” model or a “sufficiently capable and sustainably callable” model.