A New Benchmark for Open-Source LLMs: Flagship Performance in Vision and Coding Capabilities

From Quantitative to Qualitative: A 99-Day Post-Training Evolution

Over the past three months, the open-source large language model sector has experienced a silent but intense competition. In mid-July, a leading open-source model set an industry record with a scale of 2.8 trillion parameters, demonstrating remarkable reasoning capabilities in hardcore coding scenarios such as operator optimization and compilation principles. However, in terminal benchmark tests that evaluate foundational command-line operations, the initial version achieved only a 13% pass rate. This exposed a significant shortcoming: while it could understand complex algorithmic problems, it frequently stalled when executing multi-step system administration tasks.

Over the subsequent 99 days, the research and development team did not blindly chase larger parameter scales. Instead, they chose a more pragmatic path: conducting large-scale reinforcement learning post-training on the existing model while continuously iterating on the peripheral toolchain. This involved refining engineering capabilities such as sandbox environments, terminal emulators, and browser interactions. This dual approach of internal model optimization and external tooling refinement ultimately led to the new version of the model under review.

From an external evaluation perspective, this upgrade is cross-generational. The comprehensive intelligence index jumped from 44 in the initial version to 55, placing it directly into the global top four. More notably, its visual aesthetic capabilities outperformed a top-tier overseas closed-source model in blind tests, and it also overtook them in coding benchmarks. Meanwhile, the API invocation cost was reduced to two-thirds of the initial version, delivering a flagship-level experience at approximately one-fifth of the price.

I. Visual Capabilities: Breaking the “Weaker in Arts than in Sciences” Stereotype

In traditional perception, open-source models perform well in logic-intensive tasks like coding and mathematics but often fall short in visual tasks requiring aesthetic judgment. The model in this review breaks that convention.

1. Frontend Visual Restoration: Pixel-Level Intuition

In practical tests, we asked the model to generate equivalent HTML/CSS code based on a complex UI design draft. The initial version produced pages with correct layouts but rigid color schemes, rough shadows, and poorly handled rounded corners. The new version, however, generated pages almost identical to the design draft—featuring natural gradient transitions, comfortable spacing rhythm, appropriate font weight contrasts, and even reasonable light and shadow feedback for micro-interactions in hover states.

This improvement is not simply a result of “memorizing more cases” but stems from modeling “aesthetic feedback” during the reinforcement learning process. The training team introduced multi-dimensional visual evaluators that not only checked layout correctness but also quantified and scored visual hierarchy, color harmony, and whitespace proportion. This forced the model to actively consider “what the user sees” rather than merely whether the “structure is correct” when generating code.

2. Cross-Modal Consistency: More Reliable Image-to-Code

Another significant progress is reflected in multimodal understanding. We input a complex architectural diagram and asked the model to output the corresponding code framework. The new version accurately identified module boundaries, arrow directions, and data flows in the image. The generated code structure was clear, naming was semantic, and it even completed error-handling branches not explicitly marked in the diagram. This capability is highly valuable for developers reading and understanding legacy projects or quickly generating scaffolding code.

II. Coding Capabilities: From “Solving Problems” to “Doing the Work”

If vision is a pleasant surprise, the evolution of coding capabilities is the core competitive advantage of this upgrade.

1. Code Generation Quality: Complex Logic in One Pass

In tests with LeetCode Hard-level algorithmic problems, the new model not only provided correct solutions but also actively analyzed time and space complexity, accompanied by an easy-to-understand complexity derivation process. More impressively, when asked “how to optimize if the input scale increases by 100 times,” it provided multiple optimization paths—such as divide and conquer, parallelization, and approximation algorithms—and compared the applicable scenarios of each. This “engineer-style” thinking, rather than acting as a “problem-solving machine,” is a key transformation brought about by post-training.

2. Real-World Engineering Scenarios: Multi-File Collaborative Editing

We simulated a real frontend engineering task: adding a comment component with permission validation to an existing React project. The model not only generated the component code but also autonomously performed the following operations:

• Modified routing configuration files to add new routes and lazy loading logic
• Registered relevant modules in the state management repository
• Updated the API service layer to add request interceptors and error handling
• Added unit test cases covering both permission-denied and normal submission paths

Throughout the process, the model demonstrated a strong awareness of project structure. It knew which files should be modified and which should not, and it even read relevant configuration files first to understand project specifications before making changes. This multi-file collaborative editing capability closely approaches the workflow of a mid-level developer.

3. Terminal Operations: From “Armchair Strategy” to “Real Execution”

The dismal 13% pass rate of the initial version in terminal benchmark tests was the primary focus of this optimization. In tests involving real operational tasks such as system administration, software installation and configuration, log troubleshooting, and network diagnostics, the new version’s pass rate increased to 51%. We tested a typical scenario: asking the model to troubleshoot a process with excessively high CPU usage in a sandbox environment. It sequentially executed top to check load, ps to locate the PID, lsof to inspect associated files, and systemctl to check service status. Ultimately, it identified a runaway log rotation task and provided a reasonable fix plan.

This end-to-end operational capability enables the model to serve as an “autopilot” operations assistant rather than just a conversational tool providing code suggestions.

III. Cost Analysis and Selection Recommendations

1. Pricing Strategy Analysis

The new version’s API is priced at 0.6 yuan per million input tokens and 2.5 yuan per million output tokens, which is only about one-fifth of the price of comparable closed-source models. For mid-sized teams with an average of 10,000 daily invocations, monthly costs can be reduced from the tens of thousands to the thousands level. This makes it a highly cost-effective choice for budget-sensitive startup teams and independent developers.

2. Applicable Scenarios

Recommended use cases:

• Frontend interface development and design draft restoration
• Full-stack scaffolding and code generation
• Code review and refactoring suggestions
• Automated test case writing
• Operations script writing and troubleshooting assistance

Not recommended scenarios:

• Production environments requiring extremely high security and compliance (e.g., core financial trading systems)
• Customized agents highly dependent on private knowledge bases
• Interactive scenarios requiring millisecond-level low-latency responses (current inference latency is approximately 2 seconds per 1,000 tokens)

3. Synergy Strategy with Closed-Source Solutions

A pragmatic approach is to build a “dual-model” workflow: use this open-source model for daily development tasks to control costs, and fall back to closed-source flagship models for complex reasoning and critical decision-making scenarios. By designing a unified abstraction layer, requests can be dynamically routed based on task difficulty, achieving an optimal balance between cost and quality.

IV. Engineering Implementation Practices

1. Integration and Deployment

The model offers two usage methods:

• Online API: Suitable for rapid validation and small-to-medium traffic scenarios, without needing to manage underlying infrastructure.
• Private Deployment: Suitable for data-sensitive enterprises. Official commitments state that full weights will be released within two weeks of launch, allowing deployment using open-source inference frameworks. Given the model’s scale of 2.8T parameters, a multi-node, multi-GPU parallel solution is recommended, combined with quantization techniques to reduce memory pressure.

2. Prompt Engineering Recommendations

Based on testing experience, the following prompt strategy can significantly improve output quality:

[Task] Generate a form component for the user management module of an e-commerce backend system
[Requirements]
1. Use React + TypeScript, following the existing project's component specifications
2. Include three states: field validation, duplicate submission prevention, and error prompts
3. Keep styles consistent with the Card component in the design files
4. Additionally consider accessibility (a11y) support
[Reference] The project uses the Ant Design component library and already has a encapsulated FormItem

Key points: specify the tech stack, set constraints, provide context references, and quantify acceptance criteria.

V. Limitations and Future Outlook

Despite its impressive performance, the model still has clear boundaries:

• Long-context hallucinations: In conversations exceeding 30,000 tokens, it occasionally references non-existent APIs or incorrect file paths.
• Version sensitivity: Support for the latest framework versions (such as a newly released language framework) lags, and it may generate outdated syntax.
• Creative execution: When task descriptions are vague and lack precedent, it tends to make “reasonable guesses” rather than proactively asking for clarification, which can lead to directional deviations.

However, it is undeniable that the generational gap between open-source models and closed-source flagships is narrowing at a visible pace. This upgrade proves that investment in algorithmic innovation and engineering optimization can completely compensate for the physical limits of parameter scale. For developers and technical decision-makers, closely monitoring the evolution of this technical path may well be the key to grasping the next generation of software productivity tools.