AI Can Assemble Building Blocks, but Struggles to Design Something Impressive
This article explores why AI systems can assemble brick-based models yet still fail to produce visually compelling designs. It breaks down brick design as a multi-constraint problem spanning part availability, connection rules, geometry, and physics, then explains why "buildable" and "beautiful" are separate goals. It examines why general-purpose coding agents with generate-verify-fix loops may outperform specialized models, outlines a reliable evaluation framework beyond final images, and offers practical architecture guidance for developers building generative AI systems for constrained, real-world design tasks.
When AI starts handling 3D modeling tasks, many people first think of having it generate a program: give it a scene description, and it automatically creates geometry, assigns materials and lighting, and finally outputs a visualization. A similar approach works for brick-based design. A user only needs to state a natural-language requirement, such as “design a bird riding a bicycle,” and the system can try to select parts from a parts library, determine their spatial positions, and output a structural description that modeling tools can read.
On the surface, this looks like just translating text into a model. But anyone who has actually done complex brick assembly knows the problem is far harder than ordinary image generation or simple 3D modeling. The system must not only “look right”; it must also use real, existing parts, follow the connection rules between parts, manage collision relationships, and ensure the final structure can stand, bear weight, and move. More importantly, even if a model is sound from an engineering standpoint, that doesn’t mean it has good visual expression or design finish.
1. Brick Design Is Fundamentally a Multi-Constraint Problem
A complete brick work involves at least four categories of constraints.
The first is part constraints. Real-world bricks are not infinitely variable geometry; they are discrete objects with fixed dimensions, connection points, orientations, and functions. A parts library may contain tens of thousands of shapes, from basic bricks and slope pieces to joints, axles, and decorative elements, each suited to different use cases. If a system selects a part that doesn’t exist, or generates a form that can’t be produced in reality, the model may look reasonable on screen but can never be realized.
The second is connection constraints. Two parts being close in space does not mean they can connect. A connection requires alignment of positions, matching orientations, and compatible interfaces. For example, some pieces can only be inserted along a specific axis, while others rely on clips or axles to be secured. An intelligent system must understand the difference between “touching” and “connecting,” or it will generate a large number of structures that appear to fit but actually can’t be assembled.
The third is geometric constraints. A model must not have obvious intersections, overlaps, or floating elements. For a 3D structure, collision detection is only the baseline; you also need to judge whether a part blocks a key connection point, whether it obstructs later assembly, and whether a local structure is too crowded. A single part being placed correctly doesn’t mean the overall assembly order is reasonable.
The fourth is physical constraints. The work must remain stable under gravity and cannot topple due to a top-heavy design, a base that is too narrow, or insufficient connection strength. For wheeled vehicles or movable mechanisms, you also need to consider rotational clearance, force directions, and range of motion. In other words, what the system ultimately produces is not a static image but a discrete structure with real-world assembly properties.
2. “Usable” and “Good-Looking” Are Two Different Goals
Many intelligent design systems can already produce basic structures: the part count is roughly reasonable, the main silhouette matches the description, and some parts can connect. This shows that models have developed a degree of spatial reasoning and procedural generation ability.
But engineering feasibility is not the same as design quality.
A “usable” work usually only needs to meet the minimum conditions: the model doesn’t fully collapse, the main body has no severe collisions, and the user can complete assembly in some order. A “good-looking” work additionally needs coordinated proportions, a clear silhouette, consistent colors, rich detail, and defined visual focus. It must answer not only “what is this” but also “why is it designed this way.”
For example, when designing a vehicle, a low-quality solution might just piece together a few rectangular parts into a body and add four wheels. It can be semantically recognized as a vehicle, but it lacks the layered relationships among the hood, cabin, wheel arches, chassis, and rear structure. A high-quality solution uses color zoning, curved transitions, and local detail to actively guide the viewer in understanding the vehicle’s orientation, function, and sense of motion.
This shows that aesthetic evaluation is not a simple binary judgment but a multidimensional problem. Work quality can be broken down into the following metrics:
- Semantic consistency: Does the work accurately express the object and action described by the user?
- Structural completeness: Are the main body, supports, connections, and functional parts all present?
- Geometric legality: Are there collisions, intersections, floating elements, or unassemblable situations?
- Physical stability: Can the work maintain balance under reasonable conditions?
- Visual coherence: Are proportions, colors, silhouette, and detail consistent?
- Assembly feasibility: Can the work be assembled in the real world following clear steps?
- Design originality: Is it just mechanical stacking, or does it form a recognizable, distinctive solution?
The first few items are well suited to verification through rules and programs, while the latter few require visual models, human review, or more complex preference modeling.
3. Why General-Purpose Coding Agents May Perform Better
Models trained specifically for brick design often have advantages on fixed datasets and fixed task formats. But when tasks involve new part combinations, complex spatial relationships, or unseen semantic descriptions, relying purely on training memory can hit a bottleneck.
By comparison, the advantage of general-purpose coding agents doesn’t necessarily come from “understanding bricks better”; it comes from a more complete workflow. Such an agent can break the task into multiple stages:
- Parse the natural-language requirement, extracting objects, actions, colors, and poses;
- Query available parts along with their geometry and interface information;
- Generate an initial structure;
- Run collision detection and connection checks;
- Render the model or analyze its views;
- Modify the code and layout based on the check results;
- Repeat until the result meets the goal.
This “generate–verify–fix” loop is crucial. Traditional models may output a structure in one shot, whereas agents with tool-calling capabilities can handle brick design the way developers debug programs: first generate a runnable version, then fix it step by step based on error logs, geometry check results, and visual feedback.
From an engineering perspective, this approach is closer to a real design process. Designers rarely complete a final solution on the first try; they converge through sketches, prototypes, testing, and iteration. If an agent can combine a renderer, a geometry checker, a rule engine, and an evaluation model, it has a chance to compensate for the instability of single-shot generation.
4. Building a Reliable Evaluation System Takes More Than the Final Image
If quality is judged only by images, evaluation results can easily be misled. A model might generate a work that looks visually similar but actually uses non-existent parts; it might also hide structural flaws through clever viewing angles. Therefore, brick design evaluation should cover static appearance, geometric structure, and assembly logic at the same time.
1. Part Legality Check
The system needs to verify that every part belongs to the given parts library and that its model, dimensions, and orientation are valid. For automatically generated scripts, it should also check whether there is a problem of arbitrarily creating fictitious parts.
2. Collision and Intersection Check
The system should compute the intersection relationships between parts and distinguish acceptable contact, legal mating, and obvious geometric penetration. Simple bounding-box detection is fast but limited in accuracy; for critical areas, more precise mesh or solid detection is needed.
3. Connection Graph Analysis
You can treat each part as a node and each valid connection as an edge, building a connection graph of the work. By analyzing the graph structure, you can identify problems such as isolated parts, overly long cantilevers, single-point connections, or broken key structures.
4. Stability Estimation
For static models, you can use metrics such as center-of-gravity projection, support areas, and connection strength to make approximate judgments. While this can’t fully replace real physics simulation, it can filter out many obviously unstable designs. For movable structures, further analysis of range of motion and interference between parts is required.
5. Semantic and Aesthetic Evaluation
Visual models can judge whether a work matches the text description, while human review is better suited to judging whether a design is natural, layered, and distinctive. The ideal evaluation approach is not to let a single metric decide the outcome, but to combine objective checks with subjective assessment.
5. What’s Still Missing to Go from “Auto-Assembly” to “Auto-Design”
The biggest shortcoming today is that systems are often better at satisfying local rules than at establishing an overall intent. A system may know whether two parts can connect, but not necessarily understand why a certain detail should be placed where it is; it can make a model stand, but not necessarily know how to reinforce a visual theme through structure.
To improve design quality, three directions can be pursued.
First, establish a layered representation. First generate the overall silhouette and functional zoning, then generate the main structure, and finally add decorative details. This avoids having the system get absorbed in local parts from the start, which leads to imbalanced overall proportions.
Second, introduce multi-objective optimization. A model shouldn’t pursue only geometric legality; it should also consider part count, assembly difficulty, stability, appearance quality, and semantic consistency. Different tasks can set different weights—for example, an educational model values simple assembly, while a display model values form and detail more.
Third, strengthen process feedback. The system should retain the results of each generation, check, and modification, recording which parts caused collisions, which structures affected stability, and which visual regions lacked features. Through this intermediate feedback, an agent can truly form an explainable iterative process rather than repeated random attempts.
6. What This Means for Developers
Intelligent brick design is just one concrete case. It reflects a common problem of generative AI in complex real-world tasks: output that looks reasonable doesn’t mean the result is usable, and satisfying hard constraints doesn’t mean delivering a good experience.
For developers building such systems, a basic architecture can be followed:
- Use a language model for requirement understanding and solution planning;
- Use a structured database to store parts, interfaces, and constraints;
- Use a geometry engine to handle collision, connection, and dimension checks;
- Use a physics or rule module to evaluate stability;
- Use rendering and visual models to analyze appearance;
- Use a loop controller to orchestrate the generate–verify–fix process;
- Use logs and an evaluation system to record each round of changes and the final result.
This architecture also applies to tasks such as robotic assembly, mechanical design, interior layout, and 3D scene generation.
Ultimately, what AI truly needs to break through isn’t “whether it can put things together,” but whether it can understand goals, follow real-world constraints, and choose the better option among multiple feasible solutions. A runnable result is only the starting point; a stable, easy-to-use, clear, and well-designed result is where the real value of an intelligent design system lies.
