Evaluating 3D Reasoning in AI: From Impressive Renders to Engineering-Grade Validity

In 3D design and scientific computing, generative AI performance is often inflated by visual impressions: models can generate complex parts, show assembly animations, simulate disassembly, and even output building or mechanical structures that look complete. Yet what engineering practice actually cares about is not whether something is rendered realistically, but whether the structure holds, dimensions are accurate, part relationships satisfy constraints, and results can be manufactured or used for scientific analysis.

A cross-disciplinary evaluation containing 250 real 3D data tasks showed that even strong frontier models achieved a top composite score of only about 0.53 on 3D scientific understanding tasks. Results like this indicate that strong performance in 2D image recognition, interface operation, or text explanation does not mean a model has developed stable 3D spatial cognition and engineering reasoning capability.

1. 3D Generation Capability Is Not 3D Understanding Capability

A common mistake when evaluating AI-generated CAD models is treating “similar appearance” as “correct model.” In fact, 3D tasks involve at least four independent capability levels.

1. Appearance Reproduction: Does the Model Look Like the Target Object?

This level mainly checks contours, colors, materials, local features, and visual layout. For presentation-oriented tasks, appearance reproduction already has value, such as concept design, proposal demos, early communication, and virtual scene construction.

But the bar for visual similarity is relatively low. A part with offset holes, uneven wall thickness, or incorrect connection relationships can still “look very similar” from a fixed viewpoint.

2. Geometric Correctness: Are Dimensions, Topology, and Shape Valid?

Engineering models must satisfy stricter geometric conditions, including but not limited to:

• Whether solids are closed, and whether there are broken faces, overlapping faces, or self-intersections;
• Whether features such as holes, slots, chamfers, and fillets meet dimensional requirements;
• Whether normal directions are consistent and boundaries are continuous;
• Whether unintended penetration occurs between parts;
• Whether coordinate systems, unit systems, and scale relationships are correct;
• Whether the model meets input requirements for downstream meshing, simulation, machining, or printing.

A visually complete model may fail to enter downstream manufacturing workflows because of tiny gaps, non-manifold edges, or abnormal surfaces. Geometric validity is therefore the first threshold for AI-assisted design to move from “presentation-ready” to “production-ready.”

3. Structural Understanding: Do Objects Satisfy Spatial Relationships?

Objects in the 3D world do not exist in isolation. Building components need to align with axes and floor elevations, mechanical parts need to form correct assembly relationships, biomolecules need to maintain specific bonding or proximity relationships, and tissue regions in medical imaging need to match the relative positions of anatomical structures.

Typical structural relationship judgments include:

• Whether two parts touch, intersect, contain, or separate;
• Whether a hole is located on a specified surface;
• Whether a component is parallel, perpendicular, or at a specific angle to a target plane;
• Whether shafts, supports, and connectors are in reasonable positions;
• Whether an object is fully occluded or partially enclosed by another object;
• Whether the shortest distance between two molecules, tissues, or parts meets a threshold.

These questions cannot be reliably answered from a single screenshot. They require support for rotation, zooming, sectioning, hiding, measurement, and multi-view observation.

4. Engineering Reasoning: Does the Structure Follow Domain Rules?

The hardest part of engineering and scientific tasks ultimately comes down to reasoning. For example:

• Whether an assembly structure can move properly;
• Whether a support bears a reasonable load path;
• Whether building component layouts have obvious collisions;
• Whether the volume, boundary, and position of a medical region meet observational requirements;
• Whether intermolecular distances are sufficient to support a certain interaction;
• Whether machining features of a part satisfy process accessibility.

This requires the model not only to recognize “what is there,” but also to understand “why it is connected this way,” “whether risks exist,” and “which constraints are violated.”

2. Why 3D Scientific Evaluation Is Harder Than Ordinary Image QA

Traditional visual question answering is usually based on a single 2D image, with questions often focused on object category, color, count, or salient positional relationships. 3D scientific understanding involves hidden surfaces, spatial scale, coordinate transformations, and domain knowledge, making it significantly harder.

1. A Single View Has Severe Information Gaps

In 2D images, occlusion is an inherent problem. A circular hole may be a through-hole or just a recess; two parts may appear to touch in the current view, but rotation reveals a gap between them; a structure may look left-right symmetric but actually have offset features on the back side.

Therefore, an evaluation system should allow the model to actively operate on 3D objects rather than only providing fixed screenshots. Reasonable operations include:

• Changing the viewing angle;
• Adjusting zoom level;
• Switching between orthographic and perspective projection;
• Hiding or isolating specified parts;
• Using section planes to inspect internal structures;
• Reading attributes such as dimensions, angles, areas, and volumes;
• Querying part hierarchy and object identifiers.

2. Autonomous Operation Exposes the Model’s Planning Ability

In real engineering scenarios, engineers do not rotate models randomly; they design inspection steps around the question.

For example, to determine whether a hole goes through a part, a typical workflow might be:

1. Locate the face where the hole is;
2. Rotate the model to confirm the hole axis;
3. Hide occluding parts or create a section plane;
4. Check whether the hole channel extends to the other side;
5. Measure hole depth and part thickness;
6. Compare against design constraints and give a conclusion.

If AI cannot plan this path, it easily falls into the problem of “seeing a local feature and answering directly.” Even if the answer happens to be correct, it lacks stability and verifiability.

3. Scientific Objects Are More Sensitive to Numerical Precision

In everyday image tasks, natural language descriptions such as “near,” “larger,” or “on the left” are usually sufficient. But in engineering and research tasks, conclusions often depend on explicit numerical relationships.

For example:

• Whether the clearance between two parts is smaller than the design tolerance;
• Whether a component deviates from the axis beyond the allowed range;
• Whether the volume change of a region reaches an anomaly threshold;
• Whether the distance between molecules falls within a specific interaction range;
• Whether the curvature of a surface meets smoothness requirements.

Therefore, 3D evaluation cannot only count whether the final answer matches. It should also check whether the model performed effective measurements, referenced the correct objects, and produced reasoning consistent with measurement results.

3. Key Principles for Building Reliable 3D AI Benchmarks

A truly valuable 3D benchmark should not simply convert screenshots of 3D models into multiple-choice questions. It should reproduce as closely as possible the inspection workflow of professionals.

1. Data Sources Should Cover Real Tasks, Not Just Single Graphic Samples

Evaluation samples should cover multiple domains, including building structures, mechanical assemblies, biomolecules, medical volumetric data, and chemical structures. Core challenges differ across domains:

• Architecture emphasizes spatial layout, component collisions, hierarchy, and dimensional standards;
• Mechanical engineering emphasizes assembly constraints, motion relationships, feature dimensions, and manufacturing feasibility;
• Biology emphasizes structural hierarchy, binding sites, and molecular proximity;
• Medicine emphasizes voxel space, tissue boundaries, lesion localization, and volume measurement;
• Chemistry emphasizes atomic connectivity, bond angles, conformations, and distance relationships.

Multi-domain data prevents models from memorizing only one type of modeling convention or visual style while lacking general spatial reasoning ability.

2. Questions Should Cover Observation, Measurement, and Analysis

Tasks can be designed into the following categories:

Task Type	Typical Question	Core Capability
Object localization	Which region contains the specified part?	Semantic recognition and spatial localization
Relationship judgment	Do two objects touch or intersect?	3D relationship understanding
Geometric measurement	What is the hole diameter, distance, angle, or volume?	Precise operation and numerical reading
Structural analysis	Which component provides connection or support?	Structural semantic reasoning
Constraint verification	Does it satisfy parallelism, coaxiality, clearance, or other requirements?	Engineering rule judgment
Process planning	Which operations should be used to verify the conclusion?	Tool invocation and task planning

Relying only on final multiple-choice scores cannot fully distinguish whether a model truly understands or just guessed the answer. A better approach is to record operation traces, measurement data, and reasoning chains.

3. Evaluation Metrics Should Expand from Single Accuracy to Multiple Dimensions

At minimum, the following metrics are recommended:

• Answer accuracy: Whether the final conclusion is correct;
• Measurement error: Deviation between numerical results and reference values;
• Operation effectiveness: Whether model operations actually contribute to solving the question;
• Path efficiency: Whether completing the task requires too many or irrelevant steps;
• Constraint consistency: Whether conclusions are consistent with acquired geometric data;
• Robustness: Whether answers remain stable after changing the initial view, object order, or minor perturbations;
• Explainability: Whether the model can provide reviewable objects, positions, dimensions, and judgment basis.

For example, if a model answers “the two components do not collide,” but its operation trace shows no hiding of occluders, no view switching, and no distance or interference check, then even if the answer is correct, its credibility score should be reduced.

4. Common Failure Modes of AI in 3D Design

From an engineering validation perspective, common problems in current 3D AI systems are not only “wrong answers,” but also failure modes that are hidden.

1. Mistaking Visual Proximity for Real Connection

Two parts appearing close on screen does not mean they touch in 3D space. Perspective projection, occlusion relationships, and view compression can all cause misjudgment.

The remedy is to use shortest-distance calculation, cross-section observation, or collision detection rather than judging directly from a single view.

2. Ignoring Internal Structures and Hidden Features

Outer shell models are often the easiest to generate, but internal channels, ribs, cavities, and fastening structures are key to engineering correctness. For tasks with internal requirements, sectioning, transparency, or part isolation must be included in the verification workflow.

3. Letting Text Priors Override Actual Geometric Evidence

A model may infer an answer directly because “common parts usually have a certain structure” without verifying whether the current model actually does. This phenomenon can be called prior substitution for observation.

In high-risk scenarios, models should be required to output evidence-based conclusions, for example:

Conclusion: The specified hole was not found to go through the side wall.
Basis:
1. Observed along the hole axis;
2. After sectioning, solid material exists between the hole bottom and the outer surface;
3. Measured hole depth is less than the corresponding wall thickness.
Confidence: Medium; further review with design dimensions is required.

This format forces the model to convert “guessing” into “judgment based on operation evidence.”

4. Confusing Coordinate Systems, Units, and Scale

Different data sources may use different units and coordinate directions. If the system does not unify unit systems during import, the model may produce order-of-magnitude errors in interpretation even when measured numbers are correct.

Therefore, the data processing pipeline should explicitly record:

• Original units and converted units;
• World coordinates and local coordinates;
• Model scale;
• Calculation methods for attributes such as length, area, and volume;
• Measurement precision and error range.

5. A 3D AI Validation Workflow for Enterprise Applications

For teams planning to introduce AI into design review, proposal generation, model question answering, or quality inspection, a closed loop of “generate—validate—review—trace” can be established.

Step 1: Define Business Constraints

First clarify which conditions are hard constraints that must not be violated and which are soft constraints that can be optimized.

Hard constraints may include:

• Parts must not interfere;
• Critical hole diameters must be within allowed tolerances;
• Load-bearing components must satisfy specified connection relationships;
• Building components must not intrude into equipment maintenance space;
• Models must be usable solids or comply with downstream data specifications.

Soft constraints may include appearance style, layout preferences, material suggestions, and presentation effects.

Step 2: Have AI Output Structured Results

Do not only ask AI to return natural language descriptions or rendered images. Require it to output checkable structured information, such as object lists, hierarchy relationships, key dimensions, constraint conditions, and risk items to be confirmed.

An illustrative format is as follows:

{
  "objects": [
    {
      "id": "part_01",
      "role": "support",
      "constraints": ["vertical", "connected_to:base"]
    }
  ],
  "checks": [
    {
      "type": "clearance",
      "target": ["part_01", "part_02"],
      "status": "needs_verification"
    }
  ]
}

The point here is not fixed field names, but enabling AI output to be consumed by rule engines, geometry kernels, or manual review workflows.

Step 3: Use Deterministic Rules for Automated Validation

AI is well suited to proposing candidate designs, identifying potential issues, and explaining complex structures; tasks such as distance calculation, collision detection, volume calculation, and topology checking are better handled by deterministic algorithms.

A recommended division of responsibilities is:

• AI handles semantic understanding, task planning, anomaly description, and interactive question answering;
• Geometry algorithms handle intersection, distance measurement, volume, angle, closure, and topology detection;
• Rule systems compare detection results against industry standards and enterprise specifications;
• Human engineers handle high-risk decisions, boundary case confirmation, and final sign-off.

This hybrid architecture is more consistent with the reliability requirements of engineering systems than “relying entirely on model judgment.”

Step 4: Record the Entire Process and Support Traceability

For important model inspections, version numbers, input data sources, operation steps, measurement results, rule hits, and manual confirmation records should be saved. When subsequent design changes, quality disputes, or model regressions occur, teams can quickly identify whether the problem comes from data, algorithms, prompting strategy, or rule definitions.

In cloud collaboration scenarios, model versions can be stored in object storage, operation processes recorded through logging services, and abnormal failure rates identified through monitoring and alerting. These capabilities should be designed around data permissions, audit requirements, and protection of sensitive models.

6. Conclusion: The Core Competitiveness of 3D AI Is Verifiability

Advances in 3D generation capability can indeed improve design efficiency, but the value of engineering applications lies not in producing a stunning render, but in outputting results that are measurable, checkable, traceable, and deliverable.

As AI enters professional fields such as architecture, mechanical engineering, life sciences, and medical analysis, evaluation standards must upgrade from “does it look right” to “is it correct.” A mature 3D intelligent system should be able to actively observe objects, select appropriate tools, perform necessary measurements, identify structural relationships, explain judgment basis, and accept review by deterministic rules and professionals.

For development teams, building cross-disciplinary 3D benchmarks, accumulating engineering constraint libraries, introducing geometric validation mechanisms, and preserving complete operation evidence are key paths for moving AI from demo capability to real productivity.