Brick Assembly Generation: The Technical Gap Between "Functional" and "Impressive"
This article examines why generating brick-based assemblies that are merely buildable is far easier than generating ones that look genuinely designed. It frames the core problem as the difference between a structure that holds together and one that reads as an intentional object, then breaks the task into two independent layers: a physical layer (legal connections, collision-free geometry, stability, buildability) that can be checked deterministically, and a perceptual layer (semantic alignment and modeling finish) that has no single formula. It explains why the search space explodes at scale, why a closed-loop agent that can run checks and revise tends to outperform a one-shot specialized model, how to design a layered benchmark that resists gaming, and how to build a modular generation pipeline. It closes with engineering tradeoffs and the remaining gap between "usable" and "worth looking at."
The problem isn’t whether it can be assembled — it’s what counts as a finished design
Give a program a piece of text and ask it to pick components from a large part library, place them with positions, orientations, and colors, and output an assemblable structure. It sounds like retrieval plus geometric placement. The real difficulty is that the result has to satisfy several conditions at once: manufacturable connection relationships, no interference, stability under gravity, and — hardest of all — it has to look like a designed object rather than a pile of colored blocks that barely fit the description.
The first three constraints can be approximated with geometry and mechanics. The last one has no single formula. Two solutions can both “hold together” and both cover the subject and action in the prompt, yet one still looks badly proportioned, stuffed with parts, and vague in silhouette. Treating “it works” as “the design succeeded” is the most common evaluation mistake in systems like this.
This article narrows the topic to a general technical problem: given discrete parts, rigid connections, and open-ended descriptions, how do you build a generate–verify–revise loop, and how do you measure physical feasibility separately from perceptual quality?
Why the search space explodes
Parts are not continuous surfaces. They are discrete components with connection points, occupied volume, and orientation constraints. Once the library reaches tens of thousands of items, types, sizes, and connection methods all differ, and whether any two pieces snap together depends on whether their local interfaces align — not just on whether their bounding boxes are adjacent.
A complete assembly has to decide at least four things:
- Which types to use, and how many of each;
- Whether each piece’s translation and rotation land on the connection grid;
- Whether color serves the form rather than being spread randomly;
- Whether the build order lets later pieces get in, instead of sealing off interfaces first.
If you sample the overall shape, the local parts, and the colors all at once, errors propagate along the connection chain: one piece off by a unit, and the limbs, wheels, or supports downstream are all misaligned. A specialized model that only learns “placements that look like the training set” on static samples has no way to correct errors on the fly. Faced with a new description, it tends to settle on a solution that is locally feasible but globally half-hearted.
Two layers of constraints must be separated
The physical layer: can it stand up?
The physical layer answers whether the structure is assemblable and placeable — not whether it looks good. At minimum, an engineering check covers:
- Legal interfaces: connection point types match, insertion direction agrees with the grid step, and no floating pieces are “glued” together.
- No collisions: solid volumes do not intersect; allowed contact happens only at the intended mating surfaces.
- Stability: under gravity and small perturbations, the center-of-mass projection falls within the support region, and critical cantilevers have enough connection redundancy — otherwise the structure falls apart at a touch.
- Buildability: there exists an assembly order that never passes through solid material, not just geometry that is valid in the final pose.
These checks lend themselves to deterministic programs: voxel or exact-mesh intersection, connectivity of the connection graph, simplified rigid-body or quasi-static equilibrium. On failure, the system should return a locatable reason — “the front-left support is floating” or “part #37 interferes with the base” — rather than a single aggregate score. An actionable failure signal is the prerequisite for any later automated revision.
The perceptual layer: does it look right, does it look good?
The perceptual layer includes at least semantic alignment and modeling finish, and neither can substitute for the other.
Semantic alignment asks whether the subject, pose, and key parts are all expressed. For example, “a bird riding a bicycle” should let you make out the torso, wings, frame, and wheels — not just an approximate blob of a silhouette. You can evaluate this with vision–language alignment over multi-view renders, key-part detection, or human scoring, but you must fix the viewpoint, lighting, and background so the rendering style itself doesn’t inflate or deflate the score.
Modeling finish is harder. Common failures include: proportion cobbled together by “sticking on a few more pieces,” a stair-stepped silhouette with no figure–ground hierarchy, color unrelated to structure, support pieces exposed at the visual center, and inconsistent part counts on the left and right of a symmetric object. When two solutions have equal physical scores and near-identical alignment scores, people can still reliably prefer one over the other. So the evaluation must report “finish” or “preference” as a separate axis — you cannot pass off a stability score as design quality.
Why closed-loop agents often beat one-shot specialized models
A model trained specifically on brick data has one advantage: priors. It knows common proportions, common supports, and common color schemes. Its weakness is that at inference time, if it cannot call geometric checks and revise the design, errors never get absorbed during generation.
A general-purpose coding agent isn’t necessarily “better at modeling,” but it can break the design into runnable steps: retrieve candidate parts, write a placement program, call collision and stability checks, read failure logs, swap in parts locally, and retry. This resembles an apprentice who can read a spec, use calipers, and rework based on a test fit — as opposed to a process that can recite recipes but never tastes the food. The gap lies mostly in the closed loop, not in raw parameter count.
On simpler, in-distribution problems, this trial-and-error is often enough to push alignment close to a human reference. That doesn’t prove the model has learned design; it only shows the problem hasn’t yet separated “accidentally legal” from “intentionally shaped.” Once the part count climbs into the hundreds or thousands, or the description contains rare combinations or explicit style requirements, one-shot generation breaks down more visibly — and a closed loop without a good revision strategy will exhaust its budget patching local problems, ending up at “good enough to work.”
How to design an evaluation that can’t be gamed
A benchmark with real discriminative power should report results separately, not merge them into one uninterpretable score.
- Legality: the fraction of illegal connections, collisions, and unassemblable results.
- Stability: whether the structure collapses under static equilibrium and small perturbations; optionally, whether connection redundancy is too low.
- Alignment: how consistently the subject, action, and parts from the text description are covered, compared against a human reference rather than just against other models.
- Finish: silhouette clarity, proportion, reasonableness of symmetry and repetition, whether color serves structure, and whether stray parts interfere with the subject. Blind preference or ranking is recommended, to avoid letting the same generator grade its own aesthetics.
- Efficiency: part count, retry count, and check time. Trading a hundred random collision checks for a legal structure does not count as design ability.
Difficulty should be tiered: rigid objects with few parts, support-bearing structures with a moderate number of parts, open descriptions with style constraints, and reconstruction tasks that “must match a reference silhouette but cannot copy its parts list.” Being close to a human reference only on the easy set does not extrapolate to open-ended design.
Failure-type distributions should also be published. If errors concentrate in collisions, prioritize strengthening the geometric solver; if results are legal but ugly, neither adding data nor adding evaluation solves the problem of “beauty not being in the objective function.”
A generation pipeline that can actually be deployed
The recommended design makes the system a set of replaceable modules rather than an end-to-end black box.
- Parse intent: extract the subject, parts, action, style, and elements to avoid, forming a structured spec — rather than dumping the whole sentence into a sampler.
- Coarse form: first use low-resolution voxels or a few bounding blocks to fix proportions and center of mass, then attach real parts. Define the skeleton first, then add the skin.
- Part retrieval: retrieve by function, not by fuzzy name matching. Wheels, thin beams, plates, slopes, and connectors should enter different candidate pools, and the type of single-step swaps should be constrained to avoid large jumps.
- Placement and coloring: keep poses on the connection grid; generate color from a few primary tones plus accents, avoiding independent randomness for each piece.
- Programmatic verification: collision, connection, stability, assembly order; return structured failure information.
- Local revision: change only the failing subgraph. If something floats, add support or lower the center of mass; if there’s interference, swap in a thinner part or offset by one layer; if the silhouette is muddy, merge scattered pieces and emphasize the outer contour.
- Select among candidates: keep several physically valid candidates, then rank them by alignment and finish, with spot-checks by a human if needed.
A sketch of the flow (independent of any specific runtime):
spec = parse(prompt)skeleton = layout_proportions(spec)for round in range(budget):assembly = fill_parts(skeleton, library)report = check(assembly) # collision, connection, stability, orderif report.ok and score(assembly, spec) >= threshold:return assemblyskeleton, assembly = revise(report, spec)return best_valid_or_none
The revision strategy matters more than “just randomize again.” Random resampling keeps hitting the same class of geometric error; graph edits driven by failure locations spend the budget on the constraints that remain unsatisfied.
Engineering tradeoffs and common pitfalls
Version the part library. Errors in mesh precision or connection-point annotation for the same logical part will make the checker and the generator disagree on what “snaps together” means. The library, the checker, and the renderer must share the same unit and axis conventions.
The stability model doesn’t have to start with full dynamics, but it must state its approximations: quasi-static analysis, ignoring friction, and ignoring clip elasticity all mean that “check passed” does not guarantee the structure won’t fall apart on a desk. Reports meant for external use should spell out these assumptions.
Rendering cannot be the sole judge. Lighting, materials, and camera choices can hide a stair-stepped silhouette. Perceptual scores should use multiple views and be cross-checked against geometric metrics; if the two diverge over the long term, inspect annotations and camera setup before suspecting the model.
The division of labor between humans and machines should be explicit. The automated system handles the lower bound of legality and alignment; style, brand feel, and narrative detail are better kept as optional constraints or handled by a final human review. Handing aesthetics entirely to an uncalibrated scoring model writes its biases into the iteration target.
On cost, the checker should be cheap and run incrementally: recompute collisions and connections only for modified substructures, avoiding a full simulation every round. When the budget runs out and the result is still illegal, return the closest failure report rather than outputting a result that looks complete but actually interferes.
From “usable” to “worth looking at”: what’s still missing
The current evidence supports one cautious conclusion: a closed loop with tools can bring structural feasibility very close to usable, but it does not automatically bring modeling judgment. Impressive results usually come from active subtraction and emphasis — using fewer parts, strengthening the silhouette, hiding supports, letting color serve the form. If these goals don’t enter the spec or the preference data, optimization won’t move toward them.
The more valuable next step isn’t building yet another leaderboard that reports a single score. It’s publishing reproducible, tiered problem sets, structuring the failure reasons, separating physical from perceptual metrics, and recording the revision trajectory. Only then can we distinguish “search that calls a checker” from a “strategy that has actually formed a design.” For an engineering team, first making legality a gatekeeper and then building a separate review for finish is closer to a deliverable system than chasing an all-capable generator.
