Beyond Code Completion: Evaluating LLMs for Visual Generation and Real Terminal Agents

1. Model Capabilities Are Moving from “Writing Code” to “Completing Engineering Tasks”

Over the past two years, competition among programming-focused large language models has centered on code completion, function generation, and unit testing. But as model scale, training strategies, and toolchain capabilities continue to improve, a new dividing line has emerged. Models are no longer expected to only answer “How should I write this code?” They are increasingly asked to handle broader engineering tasks such as “How should I structure this interface?”, “Will this command actually run in a real terminal?”, and “Can the model complete a multi-step task autonomously?”

From a developer’s perspective, this means evaluation criteria need to evolve as well. In addition to established benchmarks such as HumanEval and MBPP, three categories of capability now deserve closer attention:

1. Multimodal interface generation and visual consistency: whether the model can generate frontend interfaces with a sound structure and consistent visual style based on natural language descriptions.
2. Terminal command execution and real environment interaction: whether the model can correctly operate a shell, file system, and package manager inside an isolated environment, rather than merely producing commands that look plausible.
3. Long-horizon task planning and state retention: whether the model can maintain context across multi-step tasks, plan subgoals autonomously, and recover from failures.

This article presents a systematic hands-on evaluation of current open and closed models in these areas, with a focus on visual generation and terminal execution in a recently released open-source model version. To remain neutral, model names are kept generic, using expressions such as “K-series models” or “a flagship open-source model.”

2. From Base Model to Post-Trained Version: Rebuilding Capability in 99 Days

2.1 Background of the Version Evolution

In July 2025, an open-source base model with 2.8T parameters was released, becoming the first publicly available open-source model at that scale. It performed strongly in geek-oriented scenarios such as compute optimization and compiler theory, but still showed clear limitations in real command-line execution. On Terminal-Bench 4.0, the initial version scored only 13%, leaving a significant gap compared with leading closed models at the time.

Over the following three months, instead of rushing out an even larger next-generation base model, the development team advanced along two parallel tracks:

• External toolchain development: iterating on a desktop programming tool, turning the command-line interface into a graphical environment with a built-in browser and visual annotation capability. Plan and Goal modes were also introduced to prepare for longer-running agent tasks.
• Internal reinforcement learning: applying joint reinforcement learning post-training to the 2.8T base model for 99 consecutive days, covering more than one hundred domains.

This “stabilize the base model first, then invest heavily in post-training” approach contrasts with the industry’s more common pattern of rapidly scaling parameters. From an engineering standpoint, post-training often has a more direct impact on real-task performance than simply increasing model size.

2.2 The Core Value of Post-Training

The core purpose of post-training is to teach the model to act within an environment, rather than only generate text in context. With reinforcement learning, for example, the model no longer just imitates answers found in training data. It must interact with an environment, receive reward signals, and gradually optimize its policy.

In programming scenarios, such reward signals may include:

• Whether a command executes successfully without unintended side effects;
• Whether the generated frontend passes automated visual regression tests;
• Whether a multi-step task is completed within a limited number of steps;
• Whether the model can recover autonomously after an error.

This kind of training pushes model output from “semantically correct code” toward code that is executable, verifiable, and recoverable.

3. Hands-On Evaluation of Visual Generation

3.1 Test Dimensions

To systematically evaluate the model’s frontend visual capabilities, the following test items were designed:

Test Item	Description	Evaluation Focus
Layout reconstruction	Generate a three-column card layout from natural language descriptions	Grid system, spacing, responsiveness
Style consistency	Generate a multi-component page containing a navbar, buttons, and forms	Consistent color, typography, border radius, and shadow
Visual annotation	Annotate problems in an existing interface screenshot and suggest modifications	Understanding and locating visual details
Design restoration	Convert a design mockup screenshot into frontend code	Pixel-level approximation

3.2 Results Analysis

In the post-trained version, the model’s frontend visual capabilities improved significantly. Taking the generation of a homepage hero section as an example, the post-trained version outperformed the baseline in the following ways:

• Spacing system: it tended to use a unified spacing scale such as 4 / 8 / 16 / 24 / 32, instead of scattered hardcoded values.
• Color management: it more frequently used CSS variables to centralize color definitions and reduce hardcoded color values.
• Maintainability: component decomposition was more reasonable, and content was less often piled into a single div.

For visual annotation tasks, the new version could accurately identify insufficient contrast, overly small click targets, and inconsistent spacing, while also providing corresponding CSS corrections. This capability is practically useful for design reviews and frontend defect localization.

That said, visual generation remains a probabilistic process. The same prompt can produce noticeably different styles under different sampling temperatures. Therefore, visual regression testing and human review are still necessary in production environments.

4. Terminal Commands and Agent Capabilities

4.1 Why Real Terminal Execution Matters

Traditional code generation benchmarks often check only whether the model’s output text matches a reference answer. They ignore a more important question: does the code actually run in a real environment?

Package management is a good example. A model that outputs pip install does not necessarily know whether the current system is Debian or Alpine, whether a virtual environment already exists, or whether version conflicts need to be handled. These environmental details determine whether an agent can complete real engineering tasks.

4.2 Environment Isolation and Safety

Terminal capability tests must run inside an isolated sandbox. The following structure is recommended:

# Create an isolated working directory
mkdir -p /tmp/model-agent-test && cd /tmp/model-agent-test
# Use a container or virtual machine to restrict permissions
# Do not mount the host file system directly
# Create a fresh environment snapshot for each test

This isolation mechanism protects the host machine and ensures that every test starts from the same initial state, improving reproducibility.

4.3 Observed Performance

The post-trained version improved its Terminal-Bench 4.0 score from 13% to above 50%. Specific improvements included:

• Correct package manager selection: it could choose apt or yum based on the operating system distribution instead of defaulting to pip.
• Error recovery: when a command failed on the first attempt, the model read the error output and adjusted its approach instead of repeating the same command.
• Path handling: when dealing with directory names containing spaces or relative paths, it no longer failed because of missing quotes.

Remaining weaknesses include:

• Managing long-running service processes is still unstable;
• Version pinning for binary dependencies is insufficiently mature;
• Timeout and retry handling in complex network environments remains relatively simple.

5. Cost and Model Selection

5.1 What the Price Change Actually Means

With the release of the post-trained version, the API price for this open-source model dropped by roughly one third. For enterprise users, this means more rounds of model calls under the same budget, or the ability to use longer context windows in more complex agent tasks.

However, a lower unit price does not automatically mean lower total cost. If the model tends to make more exploratory calls during an agent task, total token consumption can increase rather than decrease. Therefore, cost evaluation should focus on “total tokens required to complete a unit of work,” not only the input and output unit price.

5.2 Choosing Between Open and Closed Models

In the current model ecosystem, open and closed models each fit different scenarios:

• Strict requirements for private deployment and data compliance: open-source models are often the preferred option, but teams need to handle inference optimization, hardware scheduling, and security hardening themselves.
• Rapid prototyping with limited model operations capability: closed APIs remain a more cost-effective choice.
• Hybrid architecture: use open-source models for sensitive stages and API services for general stages, with routing handled through a unified gateway.

Whichever path is chosen, teams should avoid being driven solely by a single benchmark score. Instead, they should build internal evaluation sets based on their own tasks and continuously monitor model performance on real business data.

6. Final Thoughts

A substantial model version upgrade reveals more than just moving benchmark scores. It reflects a broader trend: large language models are evolving from tools that answer questions into agents that execute tasks. This shift is driven by post-training strategies, toolchain design, and engineering practice working together.

For developers, this brings both opportunities and higher requirements:

• Understand model capability boundaries and avoid over-trusting outputs;
• Build reliable testing and rollback mechanisms;
• Pay attention to cost structure, not just unit prices.

Models are evolving, and engineering practices must evolve with them. Only under reasonable system architecture and engineering constraints can model capabilities be turned into reliably delivered productivity.