From Generative AI to AI Agents: Compute and Spatial Challenges

    This article explains why AI agents raise compute costs, and why controlled workflows, explicit tool contracts, and budgets are needed for reliable execution. It also covers why 2D images do not provide dependable 3D spatial understanding, how to build research and evaluation processes that others can review, and how to connect agent decisions with spatial perception.

    1. From Generating Answers to Completing Tasks

    Generative AI is good at organizing text, producing images, and explaining code. Businesses, however, usually need a finished task rather than just an answer. For example, a system should not only explain the likely causes of an equipment fault. It should also read operating logs, check maintenance standards, propose inspection steps, and, when permissions allow, create a work order.

    This shift moves systems from single-step content generation to a closed loop of understanding, planning, execution, observation, and error correction. An agent is not simply a model with a few tools added. It embeds the model in a stateful, controllable, and traceable execution system.

    Real-world AI also faces a different difficulty. A model that can recognize objects in a photo cannot necessarily determine where those objects sit in a room, or their sizes and relationships to each other. The execution loop in language tasks and the spatial modeling in vision tasks seem to belong to different directions. In practice, both point to the same engineering problem: how to connect model outputs to verifiable facts in the external world.

    Therefore, discussing the next stage of AI systems requires more than checking whether model responses are fluent. It also requires examining computing cost, execution reliability, and environmental understanding.

    2. Why Agents Increase Compute Overhead

    2.1 From single generation to multi-round decisions

    Traditional question answering usually involves one input and one output. A complex agent may first break down a task, then call tools to gather information, adjust its plan based on the results, and finally verify the outcome.

    For example, analyzing a set of anomaly alerts may involve the following steps:

    • Read the alert summary
    • Query related metrics
    • Retrieve historical events
    • Propose cause hypotheses
    • Verify key evidence
    • Generate handling recommendations

    Each step may trigger a new model inference. Tool outputs are also added to later context, so compute overhead keeps growing.

    However, it is not valid to conclude that every agent task costs a fixed multiple of a single call. The growth depends on model size, context length, number of execution rounds, number of branches, caching strategy, and the task itself. Applying estimates from one scenario to all business cases can mislead capacity planning.

    2.2 Planning, search, and verification add work

    To improve success rates, a system may generate several candidate plans and select one to execute. It may also re-plan after a failure. These steps can improve performance on some tasks, but they carry their own cost.

    A simplified model for the cost of a single task:

    Total task cost = sum of model inference costs across all rounds + tool execution cost + data access cost + runtime environment cost + failure retry cost

    This expression is a way to classify costs, not a unified billing formula. In practice, accounting should also consider resource occupation during waits, limits on external system calls, and human review effort.

    2.3 Output volume is not business value

    The number of text units (tokens) a model processes is often used to measure inference scale, but it cannot directly represent task value. A system may produce longer outputs and perform more reasoning, yet still fail the business task because of insufficient evidence or incorrect tool use.

    A more useful metric is the comprehensive cost per effectively completed task:

    Comprehensive cost per effectively completed task = total investment in the statistical period ÷ number of tasks that pass acceptance

    This metric links success rate with cost. It avoids optimizing only for a low per-call price or adding inference rounds without limit.

    3. Building Controllable Agents: Design the Workflow Before Connecting the Model

    3.1 Manage execution with explicit state

    An agent should have observable task states, such as pending planning, pending execution, pending verification, awaiting approval, completed, and terminated. Every state transition should record the reason, inputs, outputs, and related evidence.

    For highly deterministic processes, use a fixed workflow first. Introduce model decisions only at nodes with real uncertainty. For example, reading configuration, validating fields, and calculating thresholds can usually be done by deterministic programs without repeatedly asking the model to judge them.

    This reduces unnecessary computation and narrows the scope of error propagation.

    3.2 Define contracts for tool calls

    Each tool should clearly specify its input structure, output structure, permission scope, timeout rules, and failure handling. The model proposes a call, and the execution layer checks whether the call is allowed. A model generating an operation does not automatically grant permission to execute it.

    Actions that write data, delete data, move funds, or change production configuration should require approval or confirmation based on risk. For operations that may be retried, design idempotency so that repeated execution does not cause duplicate writes or state conflicts.

    3.3 Set exit conditions and budgets

    Autonomous execution should not mean endless loops. Before a task starts, define the maximum number of steps, the allowed retry scope, the overall time budget, and the resource budget.

    When key data is missing, tools fail repeatedly, evidence conflicts, or the budget runs out, the system should stop automatic execution. It should return the completed steps, unresolved issues, and suggested points for human takeover. A system that can explain why it stopped is usually better suited to production than one that keeps trying without converging.

    4. Automated Research: Turning Exploration into Reviewable Chains of Evidence

    AI can help retrieve literature, organize hypotheses, design experiments, write experiment code, and summarize results. These abilities do not directly equal independent, reliable research.

    The core of a research process is not producing a plausible conclusion. It is ensuring that conclusions are supported by evidence, can be reviewed by others, and can be overturned when necessary.

    4.1 Distinguish facts, inferences, and hypotheses

    When a system organizes source material, it should label each piece of information as a raw observation, an established conclusion, an indirect inference, or an untested hypothesis. Statements that cannot be traced to a source should not be promoted to facts automatically.

    When sources conflict, the system should preserve the conflict and the conditions under which each source applies. It should not remove differences just to produce a tidy summary.

    4.2 Separate experiment execution from result interpretation

    Models can help write experiment code, but results must come from actual execution. Numbers, charts, and statistical conclusions in a report must trace back to input data, run logs, and analysis steps.

    Execution environments should be isolated, with restrictions on network access, file reads and writes, and resource consumption. Model-generated code should be treated as content to be checked. Correct syntax alone does not make it safe or reliable.

    4.3 Prevent contamination of evaluation

    If a system repeatedly sees the final test samples during development, its evaluation results may stop being representative. Keep development data, debugging cases, and final evaluation data separate, and record versions.

    For exploratory analysis, avoid reporting only successful experiments. Failed results, limitations, and phenomena that could not be reproduced are also valuable, and they should be recorded in the research log.

    5. Why 2D Recognition Cannot Directly Replace 3D Spatial Understanding

    A photo is a two-dimensional projection of a three-dimensional scene produced by the imaging process. Projection loses part of the depth information, so different real scenes can produce similar images.

    A model that can recognize a “table” and a “wall” does not necessarily answer re