AI Can Build Usable Brick Structures, but Still Struggles to Make Them Look Good
Brick building asks an AI model to turn a short text prompt into a physical model by choosing parts, poses, and colors. This article explains why that task is harder than simple assembly, why a general code agent can outperform a specialized model, where existing evaluations fall short, and how a stricter four-part score can separate usable builds from well-designed ones.
1. What the Task Is
Today’s AI systems can write 3D modeling scripts to generate scenes and use code to describe mechanical parts in computer-aided design. Brick building is a more comprehensive task. The model receives a one-sentence description, such as “a pelican riding a bicycle,” then selects parts from a library, sets each part’s position, orientation, and color, and produces a physical model that can sit on a table.
The task looks like simple assembly, but it tests spatial reasoning, rule constraints, and aesthetic judgment at the same time.
2. Why Building Is Hard
2.1 A Very Large Part Library
A standard brick library contains more than 14,000 part types, with different shapes, sizes, and connection methods. The model must first decide the overall form, then filter a huge set of candidates for both shape and interface fit. If the wrong part is chosen, every later position calculation fails.
2.2 Physical Feasibility
A finished build must meet at least two physical constraints:
- No collisions: two parts cannot occupy the same space.
- Load-bearing structure: the assembly must not fall apart under gravity, and every part must connect to other parts through valid connection points.
Both rules can be checked precisely in code, so they form the baseline for a usable build.
2.3 Aesthetics
Two builds may both match the text and both be structurally stable, yet one has balanced proportions and careful detail while the other looks rough and perfunctory. That difference is hard to capture with a single number, and it is the most difficult part of evaluation.
3. Specialized Models vs. General Agents
Earlier research teams trained models specifically on brick data for this kind of generation. Newer findings show that a general code agent, not trained specifically on brick data, performs better. On the existing benchmark, its alignment score is only 0.02 points below human-designed reference builds, and it clearly leads the specialized models.
The result is similar to cooking: a cook who only follows a recipe is not necessarily better than an apprentice who tastes the dish and adjusts along the way. The general agent’s advantage is that it can run code, inspect results, find errors, and correct them. For tasks that need multi-step reasoning and repeated checking, this closed loop often matters more than simply adding more domain data.
4. Limits of Existing Evaluation
If a benchmark contains only easy problems, a general agent may succeed by chance without truly mastering design. Evaluation therefore needs to cover complexity, the number of constraints, and aesthetic requirements together, so it can distinguish “can assemble something” from “can design.”
5. A Stricter Evaluation Approach
Evaluation can be split into four dimensions, scored separately and then judged together:
- Semantic consistency: whether the build reflects the subject, action, and key attributes in the description.
- Physical validity: whether it is collision-free, fully connected, and stable overall.
- Structural soundness: whether proportions are balanced and substructures have a clear function.
- Visual quality: whether color, symmetry, and detail look natural.
The first two can be checked automatically. The last two need multi-view renders plus human review. The following code sketches the evaluation structure:
def evaluate_build(build):report = {"semantic": score_semantic_match(build.prompt, build.render_views),"collision_free": not has_collisions(build.parts),"connected": is_fully_connected(build.parts),"stable": passes_gravity_check(build.parts),"aesthetic": score_visual_quality(build.render_views),}report["valid"] = all([report["collision_free"],report["connected"],report["stable"],])return report
The functions above are illustrative. A real implementation depends on the specific system. The key is to treat hard constraints as a gate: only builds that pass the gate enter aesthetic scoring. That prevents an attractive but unusable build from hiding structural problems in the overall score.
6. Implications for Developers
- Build a generate–check–revise loop. Run validation immediately after each generation, feed the errors back to the model, and let it iterate.
- Encode physical constraints as executable checkers. Collisions, connections, and stability can all be judged by code and should not depend only on the model’s self-assessment.
- Combine automatic metrics with human review. Aesthetics are hard to quantify fully. A small set of human-labeled samples can be used to calibrate automatic scores.
- Test by difficulty tier. Keep a fixed set of easy, medium, and hard prompts, and watch how performance falls as complexity rises. That curve is more informative than a single average score.
7. Summary
Getting AI to assemble a usable brick model is not the hard part. The hard part is making a build that is both stable and well designed. Evaluation should test physical constraints and aesthetic quality together, and use tiered tasks to expose the real limits of capability. For engineering work that needs spatial reasoning and multi-step checking, closed-loop iteration is usually more effective than simply adding more domain data.
