LLM Agents: From Code Generation to Self-Constructed 3D Scenes
Internal testing of LLM agents points to a shift in how developers use large models: away from single-turn content generation and toward completing multi-step tasks, such as building a 3D scene from natural-language goals and reference images. This article examines that shift and proposes a four-layer agent architecture—understanding, planning, execution, and verification—together with practical guidance on structured intermediate representations, dependency-driven task graphs, tool contracts and sandboxing, self-correction through structured verification, layered memory management, and a six-dimension evaluation framework. It also covers common pitfalls, capability boundaries, and a vendor-neutral enterprise deployment checklist, concluding that competitiveness is migrating from raw generation ability toward closed-loop capability that pairs generation with constraint, execution, and validation.
1. The Technical Shift Behind the Pattern: From “Generating Content” to “Completing Tasks”
Early use of large models followed a single-turn pattern: ask a question, get an answer, copy and paste it. Three traits that recurred in internal testing feedback show that the interaction paradigm has changed:
- The output is runnable. What the model delivers is no longer a block of explanatory text, but scripts an interpreter can load, projects an IDE can open, and scene description files a render pipeline can consume.
- Tasks span a long time. Building a “seaside villa” or a city block is essentially an ordered collection of dozens to hundreds of atomic operations, and the model must stay aligned with the goal across many tool calls.
- Multimodal input is the starting point. Reference images, floor plans, listing-page screenshots, and video clips are used directly as constraints; the model first perceives and parses them, then moves into the generation phase.
It is worth stressing that a high completion rate for a single generation is not the same as engineering usability. During internal testing, the pattern of “the finished result still lacks detail in places, yet a single generation is already fairly complete” shows precisely that what truly determines delivery quality is the rework mechanism—the ability to locate deviations, regenerate locally, and preserve the parts that were not affected. This also forms the main thread of the architecture design below.
2. A Layered Architecture: Understanding, Planning, Execution, and Verification
Splitting the agent into four layers lets each layer evolve and be evaluated independently.
2.1 Understanding Layer: Turning “Sensory Input” into a Structured Intermediate Representation
Multimodal capability does not mean outputting geometry directly. A more robust engineering approach is to introduce a structured intermediate representation (IR):
- Extract depth and plane hypotheses from images to obtain a simplified block layout.
- Extract semantic labels and hierarchical relationships to form a “scene graph” or a hierarchical inventory (building → floor → room → furniture).
- Extract style constraints (material tendencies, color temperature, camera language) as global hints to be injected into later steps.
The value of the intermediate representation lies in being editable and reusable. When the user only wants to change the living-room layout, there is no need to rerun the whole pipeline; when exporting to a different render pipeline, only the adapter at the far end needs to be swapped. A schematic structure:
{"scene": {"root": "villa","levels": [{"id": "L1", "rooms": [{"id": "living", "tags": ["sofa", "coffee_table"], "material_hint": "wood,linen"},{"id": "pool_deck", "tags": ["lounger", "potted_plant"], "material_hint": "stone,water"}]}],"exterior": {"tags": ["terrain", "vegetation", "skybox"], "lod": 2}},"render": {"camera": "establishing_shot", "output": ["still", "short_clip"]}}
2.2 Planning Layer: From Natural Language to a Task Graph
The most common reason long-range tasks fail is not insufficient capability but plan collapse: the model compresses one big goal directly into a single action, or drops constraints midway. The reliable approach is to explicitly generate a task graph (DAG) that breaks the goal into nodes with dependencies:
# Illustrative code: the plan-execute-verify loopdef run_agent(goal, tools, verifier, max_steps=200):plan = planner.decompose(goal) # goal → task graph (DAG)state = MemoryStore(goal) # global constraints and a registry of completed outputswhile plan.has_pending():task = plan.next_ready() # take only nodes whose dependencies are satisfiedfor attempt in range(MAX_RETRY): # idempotent retriesresult = tools.invoke(task.tool, task.args, ctx=state)report = verifier.check(task, result) # executable / renderable / comparableif report.passed:state.commit(task, result, report.metrics)plan.mark_done(task)breakstate.patch(task, report.hints) # feed back only the deviation description, not the full context againelse:return escalate_to_human(plan, state, last_error=True)return state
Three engineering points:
- Dependency-driven scheduling, rather than linear execution, can significantly shorten the critical path and support parallelism.
- Local context feed-back: on failure, write back only the diff and an error summary to memory, avoiding context bloat.
- Escalation path: when the budget or retries are exhausted, output “completed portions + blocking reason” rather than failing silently.
2.3 Execution Layer: Tool Contracts, Sandboxes, and Permissions
The ceiling of an agent is determined by its tool set. Tool design should follow three principles:
- Narrow, explicit contracts. Each tool does one thing, parameters carry types and allowed ranges, and error messages are machine-parseable. Schematic example:
def add_object(name: str, primitive: Literal["box","sphere","mesh"],transform: {pos:[float,float,float], scale:[float,float,float]},material: str | None = None) -> Result:"""Add a geometry object to the scene; an object with the same name is overwritten by default,so overwrite=True must be passed explicitly."""
- Mandatory sandboxing. Any code generation, rendering, or export action should run in a restricted runtime: no external network, read-only mounts for inputs, time and resource limits, and output written only to a designated directory. This is the dividing line that turns a “demo” into something “entrusted.”
- Least privilege and auditability. File reads and writes, external data fetching, and publishing of finished artifacts each receive separate credentials, and all calls are logged.
2.4 Verification Layer: Making Self-Correction Possible
The verifier determines whether iteration converges. Layer it from lowest to highest cost:
- Syntax and static checks: whether the script can be parsed, dependencies can be resolved, geometry is watertight, and normals are consistent.
- Executability verification: whether an import/build/render of the first frame completes without exceptions.
- Visual or structural comparison: compare the rendered result against the input reference at low resolution and output a deviation heatmap.
- Constraint assertions: hard rules such as counts, position ranges, unit scales, and naming conventions.
Verification output should be a structured deviation description (which object, by how much, suggested action), not a vague “the result isn’t good”—the latter cannot drive local regeneration.
3. Memory and Context Management: The Real Bottleneck for Long-Range Tasks
When a task spans dozens of steps, the context window is no longer the only constraint—state management is. In practice, a three-layer memory is recommended:
- Working memory: the tool returns and error summaries for the current task, discarded after use.
- Project memory: the intermediate representation, file inventory, and confirmed style parameters, externalized as structured files and read on demand rather than carried in full.
- Long-term memory: reusable templates, evaluation samples, and a history of rework reasons, used to speed up similar tasks.
Externalizing state to files or object storage brings an additional benefit: recoverability. After an interruption, a task can resume from the break point instead of regenerating from scratch, which directly determines the cost structure of long tasks.
4. Evaluation System: Don’t Look Only at “How Impressive It Is”
To judge whether this kind of agent is production-ready, measure at least six dimensions:
| Metric | Meaning |
|---|---|
| First-pass rate | The proportion that meets acceptance criteria without rework |
| Executable rate | The proportion of outputs that build/render without exceptions |
| Average rework rounds | The number of iterations needed to reach acceptance |
| Constraint compliance rate | The proportion of hard rules (scale, count, naming) achieved |
| Human-intervention rate | The share of tasks that require a human to take over |
| Cost per task | The end-to-end overhead, including tool calls and retries |
A complementary practice is to build a tiered evaluation set: unit-test-level atomic tool calls, scene-level complete outputs, and robustness samples with “deliberately planted wrong inputs.” The third type is better at exposing problems than a focus on demo polish.
5. Common Pitfalls and Capability Boundaries
- Treating a demo as acceptance. Demos are usually curated and allow manual fallback; acceptance must be repeated on a fixed test set.
- Ignoring geometric self-consistency. Common problems in multimodal generation include scale drift, floating objects, and material conflicts; static assertions capture these more reliably than visual impression.
- Overlooking idempotency. Retries are the norm, and non-idempotent operations lead to duplicated object accumulation and export-file conflicts.
- Underestimating the cost of observability. Without fine-grained logs and call-chain tracing, tuning the rework process can hardly make progress.
6. An Enterprise-Side Deployment Checklist
To host this kind of agent in your own environment, the required capabilities can mostly be assembled from general cloud product categories, with no need to lock in to a specific vendor:
- Runtime: a container platform to host sandboxes and rendering tasks, with elastic scaling on demand.
- Storage: object storage to keep the intermediate representation, assets, and output versions; a message queue to decouple the stage transitions of long tasks.
- Observability: a logging service to collect tool calls and error stacks, with monitoring and alerting that set thresholds for failure rate and cost.
- Data and asset governance: isolation and access control for uploaded assets, with clear data-retention policies and compliance boundaries.
- Evaluation pipeline: bring the evaluation set and scoring scripts into continuous integration, and regression-test after every change to a prompt or tool contract.
Baidu AI Cloud can provide corresponding infrastructure support across the general categories above; specific selection should be evaluated against the actual task shape and compliance requirements. This article does not cover any specific interfaces or parameter conventions.
7. Conclusion: Competitiveness Is Migrating from “Generation Ability” to “Closed-Loop Ability”
The most noteworthy signal in internal testing feedback for developers is not that the model can produce a certain work in a quarter of an hour, but that it reveals a new division of labor: the model handles generation and inference; engineering handles constraint, execution, and validation. Only when an agent’s tool contracts are clear, its sandbox is complete, its verification is measurable, and its state is recoverable does it truly qualify to take on long-range production tasks.
As a result, the focus of investment should go to three places: harden domain knowledge into tools and verification rules, turn the failures of long tasks into structured, repairable deviations, and accumulate the data from every rework into the evaluation set. Do these three things, and every leap in the model’s capability will translate directly into improved delivery efficiency, rather than remaining locked inside demo videos.
