Native Multimodal Reasoning Model Evaluated: Accuracy, Performance, and Cost Analysis

Evaluation Background: Why Focus on the “Logic Core”?

Over the past year, numerous large language models have emerged, but the true test of a foundational model lies in its performance under extreme complexity rather than marketing claims. The subject of this evaluation is a preview version of a model officially positioned as a “highly powerful native multimodal reasoning model.” Its core advertised advantage is the ability to simultaneously process massive volumes of video, audio, images, and code repositories—a promising cross-modal capability.

However, for this evaluation, we deliberately selected a highly rigorous test set focused on pure text and logical reasoning in Chinese, rather than mixed multimodal tasks. This approach was chosen for two reasons. First, the complexity of Chinese grammatical structures and deep logical nesting effectively strips away the model’s reliance on visual signals, directly targeting its foundational semantic understanding and reasoning capabilities. Second, for the vast majority of current enterprise application scenarios, pure-text logical reasoning remains the most frequent and core requirement. This “stress test” approach provides a more authentic measure of whether the model’s linguistic foundation is solid.

Evaluation Framework and Core Metrics

This evaluation was conducted in a controlled environment to ensure data reproducibility. The test set included approximately 15,000 human-verified questions covering subdomains such as logical reasoning, mathematical operations, code comprehension, and long-text summarization. Each API call used a unified parameter configuration and an identical high-concurrency framework to ensure fairness.

The core performance indicators are defined as follows:

• Accuracy: The percentage of model outputs that exactly match the standard answers, measuring semantic correctness.
• Average Latency: The total time from initiating an API request to receiving the complete response, reflecting actual response speed.
• Average Token Usage: The total number of tokens (including input and output) processed per call, directly impacting cost calculations.
• Average Cost (per 1K calls): The cost in CNY per 1,000 calls calculated at standard billing rates, providing a reference for enterprise budgets.

Core Data Overview and Key Findings

This evaluation completed approximately 15,000 calls. The aggregated key data is as follows:

• Overall Score (Accuracy): 74.8%
• Average Latency (per call): 53 seconds
• Average Token Usage (per call): 3,157 tokens
• Average Cost (per 1K calls): 250.5 CNY

Visually, a 74.8% accuracy rate is considered upper-tier among similar complex logic tests, but the structural changes behind the data are more noteworthy. Compared to the previous generation, the new version achieved comprehensive capability leaps, with the most surprising being a qualitative change in reasoning efficiency.

Key Data Comparison: Synchronized Performance and Efficiency Optimization

Metric	Previous Version	New Version	Trend
Overall Accuracy	72.5%	74.8%	+2.3 percentage points
Comprehensive Leaderboard Rank	6th	2nd	Jumped 4 places
Average Response Latency	64 seconds	53 seconds	Reduced by 17.2%
Average Token Usage	3,119 tokens	3,157 tokens	Slight increase of 1.2%

This data reveals two core conclusions:

1. Substantial Enhancement in Logical Reasoning: Maintaining the same question difficulty, a 2.3 percentage point increase in accuracy is no small feat. The new version demonstrates greater robustness, particularly in understanding complex long Chinese sentences, multiple-negative reasoning, and intricate mathematical word problems. This indicates targeted optimization of its underlying semantic representation network rather than mere parameter scaling.
2. “Counter-intuitive” Optimization in Reasoning Efficiency: Typically, model capability improvements come with increased computational overhead and longer inference times. However, despite a slight increase in token usage, the new version improved its response speed by 17%. This implies the model architecture likely introduced more efficient inference optimization mechanisms (such as speculative decoding or attention mechanism optimizations), significantly reducing the computational overhead per token. For developers requiring high-concurrency processing of complex tasks, this characteristic is arguably more valuable than the accuracy improvement.

Deep Dive: Technical Logic Behind the Performance Leap

1. Accuracy: From “Literal Understanding” to “Logical Insight”

In this evaluation, we found the new version performed exceptionally well on “implicit context-based reasoning” questions. For example, given a complex long text containing causal and transitional relationships, asking the model to infer the author’s unstated attitude or predict subsequent developments, the new version’s score was significantly higher than its predecessor. This suggests the model’s attention mechanism is stronger at capturing long-distance dependencies, capable of integrating scattered information fragments across paragraphs to form a coherent reasoning chain.

However, the error rate remained high when interpreting metaphors in very short Chinese proverbs and idioms. This reveals a blind spot in the model’s corpus coverage for highly condensed traditional cultural semantics, reflecting the data distribution characteristics of its underlying Chinese training corpus.

2. Response Time: The Practical Experience of Efficiency Gains

The response time drop from 64 seconds to 53 seconds means tasks that previously required a minute of waiting can now be completed in under a minute. For developers, the intuitive experience translates to:

• Interactive Applications: For intelligent customer service or real-time programming assistants, the shortened wait time provides a smoother interactive experience.
• Batch Processing Tasks: With fixed throughput, the volume of tasks processed per unit time increased by approximately 20%, directly improving computational resource utilization.

This efficiency gain was not achieved by truncating output length (token usage actually increased slightly) but through optimizing the model’s internal computational paths. For developers, this means achieving higher concurrent processing capacity under the same budget.

3. Token Usage and Cost Model: The Economics Behind the Slight Increase

Average token usage increased slightly from 3,119 to 3,157, a rise of about 1.2%. This minor fluctuation is within the normal range and likely stems from the model constructing longer, more rigorous intermediate reasoning steps to improve accuracy. However, because the response time was significantly shortened, the actual computational cost per unit of time decreased.

Regarding cost estimates, the average expense of 250.5 CNY per 1,000 calls is mid-tier among similar complex reasoning models. But considering its accuracy has ranked in the top two, its cost-effectiveness is highly competitive. For enterprises that need to invoke models extensively for data analysis and document review, this cost structure is acceptable.

Developer Practice Guide: How to Efficiently Utilize the Model

Based on the evaluation above, we provide the following recommendations for developers with different needs:

Scenario 1: Intelligent Summarization and Knowledge Extraction for Complex Long Documents

• Suitability: High. The model excels at global understanding of long texts.
• Recommended Configuration: Input ultra-long text in segments, adopting a two-stage strategy of “paragraph-by-paragraph summarization - global aggregation.” Leverage its strong context window to reduce the omission of key information.
• Cost Control: Since single-call token usage is relatively high, control output length by setting an upper max_tokens limit. Prioritize efficient prompt templates to avoid wasting tokens.

Scenario 2: Multi-turn Dialogue and Task-oriented Agents

• Suitability: Medium-high. The model performs well in maintaining consistency across multi-turn dialogues.
• Recommended Configuration: Enable streaming output mode to return results as they are generated. The processing speed improvement in the new version results in lower time-to-first-token for streaming, offering a better user experience.
• Performance Tuning: Consider compressing historical dialogue into key information summaries to reduce input token redundancy and further lower latency.

Scenario 3: Deep Reasoning for Structured Data and Code

• Suitability: High. It performs excellently in pseudocode logic completion and algorithmic idea generation.
• Recommended Configuration: For tasks requiring executable code, pair the model with an external code interpreter or sandbox environment for validation. Use the model to generate multiple candidate solutions, then filter for the best result through automated testing.

Limitations Analysis and Critical Reflection

While acknowledging the performance leap, the evaluation also identified several noteworthy shortcomings:

1. Overconfident Logical Hallucinations: On some extremely difficult questions, the model produces a logically fluent but entirely incorrect analysis with high “confidence,” lacking self-correction capabilities. Developers building high-risk decision-making applications must introduce additional verification mechanisms rather than blindly trusting the output.
2. Deep Cultural Gaps in the Corpus: There are still notable shortcomings in understanding classical Chinese and identifying industry-specific jargon. In specialized vertical domains, it is recommended to supplement proprietary knowledge through fine-tuning or Retrieval-Augmented Generation (RAG).
3. “Middle Forgetting” in Long Contexts: Although the model has a large context window, memory for details in the middle of the context degrades after exceeding a certain length. In practice, place important information at the beginning or end, avoiding placement in attention decay zones.

Conclusion: A New Perspective on Model Selection Post-Performance Leap

This evaluation demonstrates that the new generation of native multimodal reasoning models has made substantial progress in performance. The optimization of reasoning efficiency, in particular, offers developers a more attractive cost model. With accuracy entering the top two on the leaderboard, its underlying logical reasoning capability is now in the industry’s first tier.

For developers, model selection should not be based solely on parameters or marketing claims. Instead, as shown in this evaluation, establish a comprehensive assessment matrix correlating accuracy, latency, token usage, and cost. In complex tasks requiring deep reasoning and long-text processing, this model demonstrates significant efficiency advantages. Future attention should be paid to its further performance in streaming output engineering quality, function calling stability, and additional cutting-edge reasoning optimization features.