3D Scientific Visual Reasoning Benchmark Exposes Key Gaps in Large Model Capabilities: How Far From “Looking Right” to “Truly Understanding”

1. A Problem Masked by Visual Polish

Generative models have repeatedly sparked discussion with their performance in engineering drawing demonstrations. In some demos, models output 3D parts with assembly relationships and support rotation and disassembly animations, achieving impressive visual completeness. Social media has even seen debates about whether related jobs will be replaced. Yet feedback from frontline engineers is far more sober: none of these model outputs can go directly onto a production line. The reason is simple—looking right does not mean being right.

From a professional perspective, whether a 3D object is “correct” requires satisfying at least three levels simultaneously:

• Scientific principles: Do material properties, force relationships, chemical valence bonds, and biological structural constraints conform to domain common sense?
• Geometric structure: Are dimensions, tolerances, topological connectivity, and surface continuity self-consistent?
• Part relationships: Are assembly constraints, kinematic joints, and hierarchical organization explainable and verifiable?

Interface replication and visual realism only cover part of the surface appearance at the third level. The first two levels depend almost entirely on the model’s genuine understanding of 3D data. The question follows: Is there a professional benchmark that can test a model’s ability to observe and understand 3D scientific objects, and evaluate whether its generated results meet scientific and engineering requirements?

2. Benchmark Design Logic: Following the Workflow Like an Engineer

To address these questions, a benchmark called ScienceVision offers a systematic answer. Its starting point can be summed up in one sentence: Can models observe and reason like scientists?

The benchmark was jointly built by a research institution and a visual reasoning lab, positioned as a cross-disciplinary 3D understanding and reasoning evaluation based on expert-level visual question answering (VQA). Its core design philosophy is not “let the model look at an image and talk about it,” but rather let the model autonomously operate 3D objects and complete a full inspection workflow—starting from structural relationships, verifying geometric details item by item, and finally judging whether results meet requirements. This closely mirrors how professional engineers actually inspect parts.

In terms of coverage, the benchmark spans 5 scientific domains and 22 sub-domains, including biology, chemistry, medical imaging, architecture, and mechanical CAD, with 250 questions built from real 3D data. The questions are not hand-drawn schematics but come from real 3D source data, meaning the noise, scale, and structural complexity models face are close to real-world scenarios.

The evaluation covers 12 mainstream open-source and closed-source frontier models. Results show that the best-performing model combination scores only 0.53 out of 1.0, and no single model leads across all five disciplines. Capabilities that once approached perfect scores in visually oriented CAD benchmarks barely pass half when tested on 3D scientific understanding and reasoning.

This gap itself is the conclusion: visual generation ability and 3D scientific understanding are two capability curves whose correlation is far lower than intuition suggests.

3. Task Breakdown: Three Typical “3D Inspection” Tasks

To understand why models lose points, we first need to see what the questions actually test. The benchmark’s demonstration samples show recorded inspection operations across several disciplines. The 3D tools used are modeled after how domain experts actually process 3D source data, aiming to recreate realistic observation scenarios. Three typical task types are described below.

3.1 Biology: Measuring Spatial Distances Between Local Structures

A typical biology question format: hide the entire protein structure, keep only one ligand small molecule and a few nearby ions, and ask the model to measure the spatial distances between them and judge whether interactions are valid.

The difficulty of such tasks lies in:

1. Selective visibility: The model must establish a correct spatial reference frame even when much structural information is masked;
2. Scale sensitivity: Judging intermolecular distances depends on grasp of Angstrom-scale dimensions, not pixel-level visual similarity;
3. Semantic binding: Measurement results must be bound to chemical semantics (e.g., coordination, hydrogen bonds) to answer the question.

In other words, the model cannot just “see” two points—it must know what those two points mean scientifically.

3.2 Chemistry: Layer-by-Layer Stacking of Unit Cells and Symmetry Operations

Chemistry samples have relatively concise steps, typically around 8: first select a unit cell, then observe through symmetry operations how it stacks layer by layer into a periodic structure.

This pipeline tests the reasonability of spatial transformations:

• Can the model correctly extrapolate overall periodicity from a local unit?
• Can it understand the effects of symmetry operations (translation, rotation, mirroring) on structure?
• Can it maintain tracking of the original coordinate system after multiple steps?

For models that excel at sequence generation, this type of “multi-step spatial state maintenance” task is a clear weakness—each operation changes the global state, and once the state drifts at an intermediate step, subsequent conclusions fail entirely.

3.3 Engineering and Architecture: Item-by-Item Verification of Structural Relationships and Geometric Details

In mechanical CAD and architecture, questions require models to proceed in an engineer’s inspection order: first confirm structural relationships between parts (who connects to whom, what constraint types apply), then verify geometric details (whether dimensions close, whether topology is connected, whether surfaces are continuous), and finally give a compliance judgment.

The key here is that the inspection order itself has methodological significance: overall structure before local details, constraints before dimensions—this avoids fine-grained computation on wrong assumptions. If a model skips structural relationships and directly compares geometry, it often misses fatal errors beneath a seemingly reasonable appearance.

4. Root Cause Analysis: Four Common Weaknesses

Combining performance across disciplines, current model weaknesses can be summarized into four categories.

4.1 Insufficient 3D Spatial State Maintenance

Multi-step 3D operations require the model to update and retain complete spatial state after each step. Current models are better at processing “single-frame” information and lack stable memory for coordinate, orientation, and scale changes during continuous transformations, causing errors to accumulate across intermediate steps.

4.2 Weak Measurement and Quantitative Reasoning

3D understanding is not just qualitative recognition—it includes quantitative judgments such as distance, angle, volume, and connectivity. Without explicit geometric computation tools, models tend to “guess” numbers using language priors rather than actually performing measurements.

4.3 Disconnect Between Domain Semantics and Geometric Representations

Even when models correctly extract geometric features, they often cannot map them to domain semantics. For example, they may know two atoms are close but cannot judge whether this constitutes valid coordination; they may know two faces are adjacent but cannot judge whether assembly constraints are reasonable.

4.4 Poor Cross-Disciplinary Consistency

No model leads across all five disciplines, indicating a clear “uneven specialization” distribution. This suggests a single training distribution struggles to cover the diversity of 3D scientific reasoning, and domain adaptation remains necessary work.

5. Practical Implications for Developers

For developers building 3D-related applications, these evaluation results offer at least three actionable insights.

First, do not substitute visual metrics for correctness metrics. In 3D generation or reconstruction pipelines, introduce structured validation steps such as topological connectivity checks, dimension closure checks, and constraint satisfaction checks. This idea can be expressed in simplified pseudocode:

def validate_3d_asset(asset, spec):
    # 1. Structural relationship validation: assembly constraints satisfied
    assert check_assembly_constraints(asset, spec.constraints)
    # 2. Geometric detail validation: topology and dimensions
    assert check_topology(asset)          # connectivity, manifoldness
    assert check_dimensions(asset, spec)  # dimension closure, tolerance range
    # 3. Domain semantics validation: done by domain rule plugins
    assert domain_rule_engine.validate(asset, spec.domain)
    return True

Second, treat “multi-step spatial operations” as an independent capability to build. If applications involve 3D inspection, assembly simulation, or structural analysis, equip models with explicit geometric computation and state management modules rather than expecting a language model to complete all spatial reasoning in a single forward pass. A common approach is to wrap measurement, transformation, and intersection operations as tool calls, with the model orchestrating the workflow.

Third, evaluation sets should stay close to real 3D source data. High scores on schematics and rendered images do not transfer. When building internal evaluations, prioritize real scan data, real CAD models, and real medical images, and design multi-step tasks following expert workflows rather than single-turn Q&A.

6. Conclusion

The value of benchmarks like ScienceVision is not in ranking models but in separating “3D understanding” from the narrative of visual effects and restoring it as a measurable, decomposable, and improvable engineering problem. A score of 0.53 is not a discouraging number—it is a clear capability map: it tells developers where the three weak spots are—3D spatial state maintenance, quantitative measurement, and domain semantic binding—and where to invest next.

For enterprise users, the conclusion is equally pragmatic: in 3D-related production pipelines, visual realism can serve as an interaction layer, but correctness must be backstopped by structured validation and domain rules. Building generation capability and verification capability separately is the more prudent engineering choice at the current state of technology.