AI Model Comprehensive Evaluation: Open-Source Models Compete with Closed-Source Flagships

    This article explains how composite intelligence index scores are built from multiple task dimensions. It reviews a recent evaluation in which a Chinese open-source model scored 60 points, on par with several closed-source flagship models, and discusses the technical drivers behind open-source progress and the remaining gaps. It also offers practical guidance for developers and enterprises on model selection, building on open-source weights, and reading leaderboard results.

    1. Background: Why Model Evaluation Matters

    As large language models improve rapidly, measuring the overall capability of different models objectively and fairly has become a key step in technology selection for enterprises and in tool decisions for developers. Traditional single-task benchmarks, such as code generation, mathematical reasoning, and common-sense question answering, reflect specific capabilities. However, they do not show how a model performs across real-world scenarios. In recent years, the industry has gradually adopted composite intelligence indexes as the core of evaluation. These indexes aggregate scores across multiple dimensions to produce a single quantitative score for horizontal comparison. Such results directly shape users’ trust in and willingness to adopt a model. They also indirectly guide the development direction of open-source and closed-source models.

    Recently, a composite intelligence index published by an independent evaluation organization showed that a Chinese open-source model scored 60 points. This places it on par with several closed-source flagship models and tied for first place among open-source models. The result challenges the long-held view that open-source models lag significantly behind closed-source ones. It also suggests that the open-source ecosystem has entered a new stage in model capability. A single score, however, cannot fully represent a model’s practical suitability. The methodology, data distribution, and weighting behind the score are essential for interpreting the result.

    2. Evaluation Methodology: How the Composite Intelligence Index Is Built

    Before interpreting the score, it helps to understand the components of the evaluation framework. A composite intelligence index is usually not a single benchmark. It is a weighted aggregate of multiple task sets. Typical dimensions include:

    • Knowledge and understanding: world knowledge, common-sense reasoning, and multilingual ability, often tested through question answering and fill-in-the-blank formats;
    • Code generation and execution: generating runnable code from natural language descriptions, or fixing existing defects;
    • Mathematical reasoning: algebra, geometry, and logical recurrence, which require symbolic manipulation and step-by-step derivation;
    • Instruction following and dialogue: breaking down complex instructions, keeping responses consistent across multi-turn conversations, and declining inappropriate requests;
    • Long-context processing: memory, retrieval, and summarization over very long contexts.

    Each dimension’s score is normalized and then combined through a preset weighted sum, producing an index score from 0 to 100. The evaluator typically sets the weights based on real user usage frequency and business importance. For example, code capability may carry more weight than pure knowledge question answering. Different evaluation organizations may emphasize different things, so the same model can rank very differently across leaderboards. Developers should not focus only on the total score. They should look at the sub-scores for the dimensions they actually need.

    In this evaluation, the open-source model’s 60 points is an overall score, and its sub-scores are uneven. According to the summary of the evaluation report, its mathematics and code abilities are relatively strong. Its long-context processing and some multilingual tasks may be somewhat weaker than those of closed-source models. This kind of uneven strength is common among open-source models, because training data and optimization goals often focus on popular benchmarks. Identical totals therefore do not mean equivalent capabilities. The result must be judged against the specific scenario.

    3. Results: Drivers and Bottlenecks of Open-Source Progress

    1. Why open-source models have made breakthroughs

    Over the past year, open-source model training has changed in several key ways:

    • Refined data engineering: high-quality data, deduplication, and difficulty-based sampling have become common practice, giving the open-source community access to more usable data;
    • Better training alignment: techniques such as supervised fine-tuning, reinforcement learning from human feedback, and reinforcement learning with verifiable rewards have been quickly adopted by open-source models;
    • Faster architecture innovation: techniques such as mixture-of-experts (MoE), sparse attention, and long-context extension have moved from papers to open-source implementations within months;
    • Ecosystem synergy: open-source repositories provide reusable training frameworks, evaluation tools, and quantization solutions, which reduce duplicated effort.

    These factors have significantly improved the efficiency of open-source models per unit of parameters. Models at the scale of hundreds of billions of parameters can now compete on overall capability with closed-source flagships, which may use sparsely activated models at the trillion-parameter scale. Meanwhile, long-tail improvements from the open-source community, such as domain-specific fine-tuning, have also improved the practical usability of these models.

    2. Gaps with closed-source flagships remain in the details

    Although the total scores are level, the industry widely believes that closed-source flagships still hold advantages in several areas:

    • System-level optimization: closed-source models are often deeply integrated with inference engines and hardware scheduling, which tends to give more stable performance in low-latency and high-concurrency scenarios;
    • Safety guardrails: closed-source vendors invest heavily in red-team testing and content filtering, which makes their models harder to manipulate into generating dangerous content;
    • Long-tail knowledge updates: closed-source models can keep refreshing their knowledge through continuous learning, while open-source models have frozen weights once released;
    • Modality expansion: some closed-source flagships already support visual and audio inputs, while most open-source models remain primarily text-based.

    “On par” should be understood to mean that, across the text-based capability dimensions covered by this evaluation, the two groups have the same composite score. Engineering maturity, safety, and multimodal ability are either not covered by the evaluation or carry low weight, and they still show noticeable differences. For enterprises with strict compliance requirements, closed-source products may remain the safer choice. For teams that need control, private deployment, or research flexibility, open-source models are becoming increasingly attractive