Open-Source Flagship Face-Off: Hands-On Testing a New Contender's Real Strengths and Weaknesses

The open-source large model space recently gained a significant new entrant. A major cloud vendor released its latest flagship model, drawing widespread attention with an official claim of being “the first to rank among the strongest open-source models.” When third-party evaluation leaderboards were published, however, the public narrative split in a subtle way: on one side, the vendor proclaimed a technological leap; on the other, third-party data showed a gap in composite intelligence scores compared with top-tier products, along with a relatively high hallucination rate.

This gap between marketing and measured results is not unusual in the tech world. As developers, what we really care about is whether this upgrade represents marketing language running ahead of actual capability, or whether the model has genuinely achieved a qualitative leap. To cut through the noise, I applied for internal testing access as soon as possible and conducted an in-depth evaluation of the flagship model, called Hy4, across three dimensions: architecture and parameters, pricing and positioning, and hands-on testing on complex tasks.

1. Origins and Positioning: From “Reading Text” to “Taking Action”

Before diving into hands-on testing, it is worth clarifying the model’s fundamentals. According to the official documentation, Hy4 uses a Mixture-of-Experts (MoE) architecture with 770B total parameters, but only 49B parameters activated per inference. To put that scale in perspective: compared with the previous generation’s total parameter count of under 300B, this is a 2.6x increase and the largest model in the vendor’s history. A larger parameter count means stronger knowledge storage and representation capacity, but it also imposes higher demands on inference cost and deployment efficiency.

Beyond the larger size, the biggest additions in the official release compared with the preview version are capabilities in two dimensions:

1. Visual understanding: The model introduces native multimodal support for the first time, allowing it to directly read and analyze images and charts rather than being limited to text-only input.
2. Computer operation: It adds support for cross-application operation of graphical interfaces and command-line tools, meaning the model has preliminary potential as a “digital employee” capable of executing some basic automated workflows.

In terms of positioning, this model is clearly not content to be just a “chatbot”; it is attempting to evolve toward an “agent.” For context window, its API supports input of up to 1 million tokens, enough to digest a large code repository or a lengthy technical document in a single pass.

2. Pricing and Usability: The “Sweet Spot” and the “Traps” Behind the Appeal

For developers, model capability matters, but cost is always an unavoidable decision variable. Hy4’s API pricing is aggressive: 6 yuan per million input tokens, 18 yuan per million output tokens, and as low as 0.3 yuan for cache hits. Taking the output price as an example, this is significantly lower than同级 competitors (such as a leading domestic large model Pro version at roughly 70 yuan per million tokens). Among flagship models, Hy4’s cost-effectiveness advantage is indeed plain to see.

That said, it is important to distinguish the use cases of the “flagship version” from those of the “lightweight version.” Hy4’s 1 million token context window is the “sweet spot” for processing ultra-long text; but in production environments that require extremely low latency and high-frequency calls, a large-parameter model like this will take longer to infer and consume substantial GPU memory. For individual developers or small-to-medium teams, the more pragmatic path may be to use it for offline batch processing or complex reasoning tasks rather than for high-concurrency real-time interfaces.

3. Hands-On Testing: The Real Level of Code Generation and Multimodality

To verify the accuracy of the official claims, I selected two scenarios that best test a model’s caliber: complex logic code generation and converting hand-drawn sketches into web pages.

Test 1: Complex Logic Code Generation — Game Development

My test task was to generate a classic “Gold Miner” mini-game. The requirements included a complete mine, a miner character, a claw that grabs objects, a winch reeling mechanism, and a Low Poly cartoon style. With “high thinking intensity” mode enabled, Hy4’s output was impressive. It not only generated runnable Python code but also reasonably used the pygame library to build the game loop and, through control of vector math, implemented the physical logic of the claw extending and retracting. The generated code had a clear structure and thorough comments, and it could run with almost no manual modification.

This demonstrates that Hy4 has a solid foundation in complex engineering code generation. It does not simply stitch syntax together; it can understand the specific business logic of “winch reeling” and translate it into a mathematical model.

Test 2: Visual Understanding — Reconstructing a Hand-Drawn UI Sketch

The second test exercised its “new eyes.” I hand-drew a simple web layout sketch (including a navigation bar, a two-column layout, and content cards), photographed it, and uploaded it, asking the model to directly generate the corresponding HTML/CSS code. Hy4 correctly recognized the structural divisions and text labels in the sketch, accurately generated Flexbox-based layout code, and automatically matched a suitable color palette. Although recognition of some scribbled visual elements was still a bit rough, its structural fidelity had already reached a level suitable for entering a design review.

4. A Rational View of the Tension Between “Benchmarks” and “Hallucinations”

Returning to the controversy mentioned at the start of this article: why is Hy4’s composite score lower than Kimi K3.1 Pro in third-party evaluations? Here it is necessary to distinguish the dimensions of “composite intelligence” evaluation. Third-party leaderboards typically cover multiple composite metrics such as math, reasoning, code, and instruction following, whereas Hy4’s strengths are more evident in long-text processing and multimodal understanding, which tend to have diluted weight in a single “composite intelligence” ranking.

As for the conclusion of a “high hallucination rate,” this needs to be stated objectively: the hallucination problem in large models is essentially a side effect of probabilistic output. When pursuing diversity and creativity in generated content, Hy4 may indeed sacrifice some factual accuracy. For developers, this means that when relying on it to generate technical documentation or factual answers, additional verification mechanisms need to be added; but in open-ended tasks such as creative generation and brainstorming, this “hallucination” may instead become a source of inspiration.

5. Conclusion: Who Is It For?

Based on the hands-on experience, this new open-source flagship is not unworthy of its name, but its “strength” is conditional.

• If you are a developer who needs to process massive document volumes, perform complex code refactoring, or explore multimodal applications, Hy4’s extremely high cost-effectiveness and 1 million token context window are genuinely useful tools.
• If you are building a high-concurrency customer service chatbot or a real-time online Q&A system, you should perhaps consider a lighter, lower-latency specialized model rather than simply piling on parameters.

The release of Hy4 marks another leap forward for open-source models in the “multimodal” and “agent” directions. It may not have taken first place on some traditional leaderboards, but in terms of productivity release in specific vertical domains, it has delivered a highly convincing result. In an era where the boundaries of model capability are increasingly blurred, choosing the model that best fits your business scenario is far more meaningful than chasing benchmark numbers.