Open-Source vs. Closed-Source LLMs: How Big Is the Gap Really?
Benchmark data from ~15,000 unified test questions shows how the latest flagship closed-source model reclaimed the top spot, why its accuracy gains came with steep cost increases and capability regressions, and how the real open-source vs. closed-source gap has shifted from raw capability to engineering, cost, and control trade-offs.
1. Why a Unified Benchmark Is the Only Fair Way to Measure the Gap
Claims about model capability usually come from vendors themselves. What truly reflects the gap between open-source and closed-source models is horizontal data collected under the same evaluation set and the same invocation method. A common industry practice today is to build a question bank spanning education, healthcare, reasoning and math, code generation, instruction following, and agent/tool-use scenarios, then run automated evaluations at a scale of tens of thousands of questions while recording four core metrics: accuracy, average response time, token consumption per call, and converted cost.
This approach delivers value in three ways:
- Comparability: All models are scored on the same question set under the same criteria, avoiding the benchmark pollution problem where everyone reports on their own terms.
- Multi-dimensionality: Beyond the total score, you can see strengths and weaknesses in specific domains, enabling targeted model selection for different business scenarios.
- Cost perspective: No matter how high the accuracy, if the per-call cost is more than ten times that of a competitor, the cost-effectiveness of deployment must be recalculated.
In a recent large-scale evaluation (roughly 15,000 test questions), a new-generation flagship closed-source model topped the leaderboard with a total accuracy of 72.5%, up 3.6 percentage points from its predecessor, ending a long period in which Chinese closed-source models dominated the rankings. The number itself is worth analyzing, but the structural changes behind it are even more valuable.
2. Measured Performance of the New Flagship Closed-Source Model
2.1 The Jump in Overall Metrics
Compared with its predecessor, the new flagship model shows a combined profile of “significantly higher accuracy, slightly faster speed, and notably higher cost”:
- Total accuracy rose from 68.9% to 72.5% (+3.6 percentage points), jumping from 7th place to 1st.
- Average per-call latency dropped from about 67 seconds to 64 seconds, an engineering efficiency gain of roughly 4.5%.
- Average token consumption per call rose from 2,810 to 3,119, an increase of about 11%.
- Converted cost per 1,000 calls climbed from about 189 yuan to about 247 yuan, an increase of over 30%.
This shows that capability gains are not a free lunch: the model tends to trade longer chains of thought and more reasoning tokens for higher accuracy, and the combination of a higher output unit price and increased token consumption drives up total cost.
2.2 Domain Breakdown: Surprises and Regressions Side by Side
Breaking down the total score by dimension reveals more interesting patterns:
Areas of major improvement are concentrated in deep reasoning tasks. Accuracy on education questions jumped from 49.1% to 64.8%, a gain of nearly 16 percentage points; healthcare and mental health rose from 77.8% to 89.5%; reasoning and mathematical computation went from 72.2% to 81.8%. What these domains share is the need for multi-step reasoning, knowledge recall, and logical chain verification — the “PhD-level reasoning” marketing claim is partially validated by these numbers.
Areas of regression deserve caution. Language understanding and instruction following dropped from 70.6% to 67.5%, and agent/tool-use fell from 61.0% to 58.6%. A reasonable explanation is that as the model’s reasoning ability was strengthened, the training focus and post-training strategy shifted, creating a trade-off against basic language understanding and instruction alignment. This is a warning for applications that depend on strict output formatting and multi-turn tool orchestration — before upgrading model versions, always run regression tests on your own business data.
3. Cost Structure: The Real Price of the Closed-Source Leader
The new flagship model is currently the most expensive commercial model in the evaluation: about 247 yuan per 1,000 calls, nearly 16 times the second-ranked model on the same leaderboard (about 15.6 yuan per 1,000 calls), and far above other mainstream closed-source flagships (in the roughly 88 yuan per 1,000 calls tier). Its cost structure can be broken down into two parts:
- Increased token consumption: Reasoning-oriented models tend to generate longer thinking processes, pushing total input/output tokens up by about 11%.
- Higher output unit price: The output price rose from about 72.5 yuan per million tokens to about 85.2 yuan per million tokens, an increase of roughly 17.5%.
The takeaway for developers: gains in reasoning capability usually come at the cost of longer reasoning chains. Calling the strongest reasoning model for simple tasks can be pure waste. A more sensible approach is model routing — send simple requests to a low-cost model and escalate only complex reasoning to the flagship model. An illustrative strategy:
def route_request(prompt: str, complexity: float):if complexity < 0.4: # Simple Q&A, formatting tasksreturn call_model("light", prompt)elif complexity < 0.75: # Routine business reasoningreturn call_model("standard", prompt)else: # Complex math / multi-step planningreturn call_model("reasoning", prompt, budget_tokens=8192)
4. Open-Source vs. Closed-Source: The Real Picture of the Gap
Back to the title question: how big is the gap between open-source and closed-source models? Several conclusions can be drawn from the evaluation data:
1. The total-score gap is narrowing, but the frontier still has a moat. The strongest open-source models can already approach, and in some areas locally surpass, certain mid-to-high-tier closed-source models, and they perform particularly well in domains like code and math where public training data is abundant. But in cutting-edge capabilities such as multimodal reasoning and long-context stability, closed-source flagships still hold a lead.
2. The nature of the gap is changing: from a “capability gap” to an “engineering and cost gap.” Open-source model weights can be self-hosted, so marginal call cost is mainly GPU compute; in high-concurrency or data-sensitive scenarios, total cost of ownership may actually undercut closed-source APIs. Closed-source models win on higher single-point accuracy and fully managed service capabilities.
3. Asymmetric iteration speed. A single version upgrade of a closed-source flagship can deliver a 3.6-point jump in total accuracy, while open-source progress is scattered across different vendors’ release cadences. That said, historically the time lag between the closed-source frontier and the best open-source model has shrunk from over a year to a matter of months.
4. Selection advice. For most teams, the pragmatic strategy is a “hybrid architecture”: let open-source models carry high-frequency, routine, and data-sensitive workloads, route the small share of high-difficulty reasoning requests to closed-source flagship APIs, and continuously monitor the gap between the two on your own business dimensions through an evaluation set, dynamically adjusting the ratio.
5. Conclusion
Evaluation data under a unified benchmark tells us this: closed-source flagships are still pushing the frontier forward, but they are beginning to show a structural trade-off of “stronger reasoning, regressing basic capabilities, and surging costs”; meanwhile, the open-source camp keeps applying pressure on cost-effectiveness, controllability, and iteration speed. The so-called “gap” is no longer a single-dimension race — it is a multi-dimensional balance among capability, cost, and control. For developers, rather than picking sides between camps, it is better to build your own evaluation and routing system — and let the data make the model selection decisions for you.
