Open-Source Coding Models Level Up: Hands-On Testing of Visual Generation and Agent Workflows

From Parameter Races to RL Post-Training: A New Divide for Coding Models

Over the past year, the logic of competition among large language models in coding has shifted noticeably. Early on, the discussion centered on parameter counts, context length, and single-turn code completion accuracy. But once models entered real engineering environments, developers came to realize that what determines whether a model can take on genuine development work is often not “can it write a snippet of code that runs,” but “can it keep making the right decisions in a complex environment.”

This shift has given rise to a new training paradigm: large-scale reinforcement learning (RL) post-training on top of base models. Unlike conventional supervised fine-tuning, RL emphasizes a model’s ability to experiment, correct itself, and choose the right path across multi-step tasks. This matters most in scenarios such as command-line operations, file system interaction, browser automation, and frontend layout adjustments, where the model must keep adjusting its next action based on environmental feedback rather than producing a static answer in one shot.

Public benchmark data shows how significant this training strategy can be. Take terminal operations: some early base models achieved completion rates of only around 13% on Terminal-Bench-style benchmarks, while versions that underwent extended RL post-training improved that figure several times over. More importantly, the gains are not confined to a single leaderboard — they generalize across tasks and domains.

The lesson for developers: when evaluating a coding model, don’t stop at code completion accuracy. Pay attention to its stability, error-correction ability, and tool-calling efficiency in multi-step agent tasks. In long-running projects in particular, whether a model can recover on its own after a mid-step failure often matters more to the actual experience than the quality of its first output.

3D and Frontend Visuals: From “Can Generate” to “Can Design”

One of the most notable changes observed in this round of testing was the model’s performance on visually oriented coding tasks. Traditional code generation models handling frontend pages, 3D scenes, or data visualizations could usually guarantee “structurally correct” output, but fell clearly short on aesthetic dimensions such as spacing, color, hierarchy, and whitespace. Developers typically had to invest substantial manual tuning after receiving the generated result.

The new generation of models shows a qualitative leap in visual generation. Take frontend pages: given a screenshot or design mockup of a product landing page and asked to reproduce it, the model’s output gets remarkably close to professional designers’ work in title hierarchy, card spacing, color contrast, button states, and responsive breakpoints. In some tests, the visual completeness of the generated pages even surpassed that of certain closed-source flagship models.

In 3D scene construction, the model demonstrated skilled command of Three.js-style code. Lighting setups, material definitions, camera paths, particle systems, and geometry combinations can all be turned into directly runnable examples within a short time. More importantly, its understanding of 3D spatial relationships no longer stops at “placing a few cubes” — it can generate scenes with clear visual intent from natural-language descriptions, such as “a day-night cycle in a low-poly forest” or “a tech showroom with mirror reflections.”

Behind this improvement in visual ability lie changes in training data and optimization objectives. The model no longer learns only “how to write code” but also “what kind of code produces visually appealing results.” For frontend engineers, 3D developers, and data visualization teams, this means the model is evolving from a “code generator” into a “visual collaboration partner.”

Agent Workflows: Coordinating the Terminal, Browser, and File System

The next capability frontier for coding models lies in agent workflows. An agent workflow means the model is no longer confined to single-turn Q&A or single-file generation; it can execute multi-step tasks within a complete environment that includes the terminal, browser, file system, and version control.

A typical task might look like this: pull project code from a remote repository, read multiple files to understand the project structure, locate a functional defect, modify the relevant code, run the test suite, review the failure logs and keep fixing until all tests pass. Throughout the process, the model has to cycle repeatedly through “understand – act – observe – correct.”

In testing, the new generation of models handled these tasks impressively. It can execute commands in the terminal, parse the output, adjust its strategy based on error messages, and open a browser when necessary to check documentation or run a preview. Compared with earlier models, the most striking change is a sharp drop in the probability of getting “stuck.” Even when a step fails, the model tends to proactively try alternative approaches instead of repeating the same failing operation.

This capability is particularly well suited to automated testing, continuous integration, code migration, and dependency upgrades. Developers can hand repetitive, procedural work to a model agent and focus on architectural decisions and business logic. That said, agent workflows place higher demands on a model’s controllability and safety. In production use, they should be paired with sandbox isolation, permission controls, and audit logs to prevent the model from executing high-risk operations.

Cost Efficiency: Can Flagship-Level Capability Really Go Mainstream?

Beyond performance gains, cost is the other key factor in whether a technology can be adopted at scale. In the past, the pricing of top coding models put them out of reach for many small and mid-sized teams. While closed-source flagship models still lead on some benchmarks, their per-task cost can run several times — even tens of times — higher than open-source alternatives.

The trend observed here is that the new generation of open-source models has significantly improved coding and visual capabilities while further lowering API costs. Some public information indicates that certain flagship open-source models now cost one-fifth or less of comparable closed-source models. This means a five-person team using open-source models for everyday coding assistance, code review, and test generation could keep total monthly costs to the equivalent of just a few hundred RMB.

The direct effect of falling costs is a wider range of use cases. Cost constraints once confined many teams to using LLMs only in “core development stages”; now models can cover more peripheral scenarios, such as generating unit tests, supplementing code comments, producing changelogs, translating technical documentation, and assisting with code review. Tasks with low unit value but extremely high frequency become economically viable with low-cost models.

That said, cost calculations shouldn’t focus solely on API unit prices. Teams with private deployment needs also need to factor in inference hardware, operations staffing, and model update strategies. Open-source models offer a controllable technology stack, data that never leaves your domain, and the ability to fine-tune against specific codebases; their downside is that maintaining inference services requires a certain level of engineering capability. For enterprises without a dedicated platform team, a mature cloud-hosted model service is often more cost-effective than building your own inference cluster.

Practical Advice: Building Your Own Model Evaluation Pipeline

Rather than relying on a single leaderboard, it’s better to build a model evaluation pipeline tailored to your own business. Here is a four-step process you can reference:

Step 1: Build a private task set. Extract 100 to 200 tasks from the historical data of real projects, covering frontend, backend, database, DevOps, testing, and other categories. Each task should include a clear input, expected output, and acceptance criteria.

Step 2: Set multi-dimensional metrics. Beyond code correctness, also record average steps to completion, number of failed retries, terminal command accuracy, and visual similarity scores. For frontend tasks, you can introduce structural similarity metrics or human blind review.

Step 3: Introduce an agent evaluation framework. Connect the model to a sandboxed environment and let it execute tasks in a real terminal and browser. Automatically log every action and observation for later analysis and regression testing.

Step 4: Continuously track cost and latency. For each evaluation, record token consumption, end-to-end elapsed time, and the extra overhead of failed tasks. Only by comparing performance and cost in the same table can you make a selection decision that fits your business reality.

With this pipeline in place, teams can break free from over-reliance on public leaderboards and genuinely answer the core question: “Does this model actually work well in my project?”

Boundaries and Outlook: How Far Are We from Autonomous Development?

Despite the strong performance of the new generation of models on coding and visual tasks, fully autonomous development remains some distance away. Current models have three main limitations. First, memory and planning over long-horizon tasks are still limited; once a project structure grows complex enough, the model tends to lose global context. Second, understanding complex business logic depends on clear comments and documentation, and the model still misjudges messy legacy code. Third, in multi-agent collaboration scenarios, the mechanisms for communication and conflict coordination between models remain immature.

Over the next two to three years, directions worth watching include more efficient long-context memory management, automated fine-tuning toolchains tailored to specific codebases, and frameworks for multi-model collaborative development. For developers, the most valuable thing to do right now is to accumulate evaluation data from real-world scenarios, gradually upgrade models from “coding assistants” to “automated development agents,” and, along the way, work out the new division of labor between human judgment and machine execution.