From Generation to Action: The Compute and Spatial Cognition Challenges of AI Agents
As large models move from single-turn generation toward end-to-end task completion, AI agents are reshaping both how compute is consumed and how machines perceive the physical world. This article breaks down the compute economics of action-oriented AI, explains why per-task costs can rise by orders of magnitude, and presents architecture patterns such as tiered model routing, budget guards, observability, and reliability engineering. It also examines the gap between 2D vision and 3D spatial understanding—covering representations, multi-sensor fusion, and scene graphs—and closes with a practical action checklist for enterprise teams. All code samples are for architectural illustration only.
1. Introduction: From “Generation” to “Action”
In recent years, the capabilities of large models have kept expanding. At the same time, the role of AI is undergoing a fundamental shift: it is no longer just an application that gets called on demand, but a force reshaping how the computing industry divides its work and giving rise to new product forms and business models. The prevailing view across the industry is that the focus of competition is moving from “single-turn generation quality” to “end-to-end task completion capability,” and the suppliers that can consistently secure high-quality compute therefore hold a more advantageous position.
This shift can be summed up as “from generation to action.” In the generative phase, a model’s output is text, images, or audio. In the agentic phase, the model’s output is a sequence of decisions: first understand the goal, then break it into steps, then call tools such as search, code execution, and data queries, observe the results, revise the plan, and finally deliver an outcome. The longer the chain of understanding, reasoning, planning, and tool use, the greater the compute cost of a single task. Industry observers have noted that completing this full chain requires roughly three orders of magnitude more compute than simple generative interaction. Whether or not the exact multiple is precise, the trend is clear:
- Shifting value anchor: The measure of success changes from “content quality” to “task success rate × unit cost × response latency.”
- Downward shift of the technical stack’s center of gravity: Prompts are only the surface; tool contracts, execution sandboxes, state management, and observability determine the ceiling of the system.
- Architecture must be scalable: Compute consumption has a non-linear relationship with task complexity, so cost governance must be part of the design from day one.
Note: The terms “compute,” “token,” “inference,” and “KV cache” used throughout are industry-standard concepts. The code is for architectural illustration only and does not depend on any specific vendor’s API.
2. Compute Economics: Why the Cost of Action-Oriented AI Rises by Orders of Magnitude
2.1 Call Amplification from Multi-Step Loops
A typical agent run consists of several phases: planning consumes inference compute, execution consumes tool compute, and verification consumes inference compute again. The cost of a single step may seem limited, but the steps multiply. As a result, cost should be accounted for per “successful task” rather than per “request.”
# Illustrative code: cost accounting based on "cost per successful task"# Break a task into phases and estimate inference and tool consumption separatelydef estimate_task_cost(task):unit = { # prices are per 1K tokens; these are illustrative values without volume discounts, etc."reasoning_in": 0.002,"reasoning_out": 0.008,"small_in": 0.0002,"small_out": 0.0004,}reasoning = task["model_calls"] * (task["avg_input_tokens"] * unit["reasoning_in"]+ task["avg_output_tokens"] * unit["reasoning_out"])tools = task["small_model_calls"] * (task["avg_input_tokens"] * unit["small_in"]+ task["avg_output_tokens"] * unit["small_out"])cost = reasoning + tools# Failed retries, timeouts, and human handoffs also consume compute and must be counted in the denominatorsuccess_rate = task["succeeded"] / task["attempted"]return cost / success_rate # cost is amplified as the success rate drops
2.2 Hidden Costs Within a Single Call
- Prompt length: The system prompt, tool descriptions, conversation history, and knowledge snippets are re-fed as input at every step, forming a fixed overhead.
- Output length: The longer the explicit reasoning chain, the more output tokens; under pricing models that distinguish input from output, long reasoning significantly drives up cost.
- Long-context attention: Attention computation grows superlinearly with context length. Although the KV cache can reuse already-computed keys and values, GPU memory usage continues to rise with length and concurrency.
- Result re-injection: Uncontrollable content such as search results, logs, and API responses can inflate the next round of input. This calls for summarization and compression, along with upper limits on result truncation.
2.3 Cost Amplification Mechanisms
- Autoregressive generation: Tokens are produced one by one, and both time-to-first-token and total duration grow with output length, making full parallelization difficult.
- Retries and verification: Because output is stochastic, multi-candidate sampling, self-checking, and retries after failure are needed, so actual consumption is often far higher than the theoretical minimum.
- Product of depth and breadth: Multi-agent collaboration, reflection loops, and search-tree-style exploration multiply the number of calls, so upper limits on depth and breadth must be set.
| Cost item | Primary drivers | Optimization approach |
|---|---|---|
| Inference calls | Steps × context length | Tiered routing, history compression, budget caps |
| Explicit reasoning | Chain-of-thought length | Disable deep reasoning for simple tasks, output structured results |
| Tool calls | External APIs, execution environment | Result summarization, local caching, bounded concurrency |
| Failures and retries | Failed validation, timeouts | Strict output constraints, idempotent execution, limited retries |
| Long context | Documents / dialogue / tool results | Retrieval augmentation, sliding window, summary memory |
3. Architecture Design: Tiered Routing and Budget Control
3.1 Tiered Routing: Use the Right Model for the Right Job
Not every step needs the strongest model. You can tier by task complexity:
- Lightweight tier: Highly deterministic tasks such as intent classification, format conversion, field extraction, and result summarization.
- Standard tier: Multi-turn dialogue, routine code generation, and code refactoring.
- Deep tier: Complex planning, architecture design, and difficult troubleshooting that require explicit reasoning and tool coordination.
# Illustrative code: tiered routing with a budget guardCOMPLEXITY_RULES = [# (task type, allowed tiers, whether explicit reasoning is enabled, max steps)("extract", ["small"], False, 1),("chat", ["small", "standard"], False, 2),("codegen", ["standard", "deep"], True, 6),("debug", ["deep"], True, 10),]def select_tier(task, form):for kind, tiers, reasoning, max_steps in COMPLEXITY_RULES:if task["type"] == kind:return {"tier": tiers[-1] if task["complexity"] >= 0.7 else tiers[0],"reasoning": reasoning,"max_steps": max_steps,"input_tokens": len(form["messages"]), # coarse estimate; production should use a tokenizer"output_tokens": 0,}def charge(plan, usage, limit):plan["input_tokens"] += usage["input_tokens"]plan["output_tokens"] += usage["output_tokens"]return plan["input_tokens"] <= limit["input"] and plan["output_tokens"] <= limit["output"]
Key implementation points:
- Quantifiable routing basis: Prefer rules and structured features (task type, retrieval hit rate, historical success rate). Use model self-assessment cautiously, because it too consumes compute and is not stable enough.
- Front-loaded budgets: Set token and cost caps for each session and each task; when a threshold is reached, downgrade or hand off to a human to prevent abnormal loops from creating a “compute black hole.”
- Outcome-based billing: For external users, prefer billing by task or by subscription, while controlling abnormal consumption through rate limiting and quotas.
3.2 Observability and Attribution
Inference calls have black-box characteristics, so the following capabilities must be in place before launch:
- Full-chain tracing: Record the model tier, input/output tokens, tool latency, and error codes for every call.
- Cost attribution: Break down costs by tenant, business line, and task type to locate high-consumption scenarios.
- Quality metrics: Task success rate, first-attempt success rate, average number of steps, and human handoff rate.
- Evaluation regression: Build task-level evaluation sets and run regressions after every prompt or model change to prevent quality regressions from being masked by cost optimization.
When deploying on the cloud, these capabilities are typically assembled from general-purpose cloud services: inference and task execution can run on a container platform with automatic elastic scaling; large files and tool artifacts can be stored in object storage; asynchronous tasks can be smoothed by message queues; and call logs and metrics can feed into logging services and monitoring alerts. Cloud providers such as Baidu AI Cloud offer similar foundational capabilities, and enterprises can choose based on their own architecture.
4. Spatial Cognition: From 2D Pixels to the 3D World
4.1 The Gap Between 2D Vision and Real Space
Current vision capabilities are impressive: image classification can complete in milliseconds, scene segmentation can distinguish objects pixel by pixel, generative models can render photorealistic interior images, and vision-language models can even give plausible descriptions of places they have never visited. Yet if you put the same models into a real room and ask three questions:
- Which item sits on which shelf?
- How far is the table from the wall?
- Where does the ceiling end, and where does the window begin?
The answers are often inadequate. The reason is that most vision benchmarks are defined on a 2D plane: classification, detection, and segmentation measure “what is in the pixels,” not “what is in the space.” Models learn statistical patterns of pixels rather than the geometric structure of a scene.
4.2 Root of the Task Difference
| Dimension | 2D vision tasks | 3D spatial understanding tasks |
|---|---|---|
| Input | A single image | Multi-view images, depth maps, point clouds, or continuous video streams |
| Annotation | Category, 2D bounding boxes, pixel masks | 3D coordinates, orientation, scale, scene layout |
| Key challenges | Occlusion, scale variation, class imbalance | View consistency, depth ambiguity, occlusion and visibility, metric scale |
| Typical outputs | Labels, bounding boxes | Object poses, spatial relations, layout maps, free space |
| Typical applications | Content moderation, retrieval, quality inspection | Robot navigation, indoor modeling, industrial inspection |
4.3 Scale Ambiguity in Monocular Views
A single image cannot directly provide absolute distance: a small nearby object and a large distant object may be equivalent in the image. To recover true scale, one of the following must be added:
- A depth sensor or stereo disparity;
- Camera calibration parameters (such as a known focal length and sensor size);
- A reference object with known size or pose;
- A scene prior obtained from 3D reconstruction.
4.4 3D Representations and Multi-Sensor Fusion
Common 3D representations each involve trade-offs:
- Point clouds: Generated by depth cameras or LiDAR, well suited to expressing fine geometry but sparse and noisy.
- Voxels: Structurally simple and easy to process with conventional operators such as convolution, but resolution and GPU-memory overhead constrain each other.
- Neural radiance fields: High quality for rendering novel views, but they require jointly optimizing camera poses and scene parameters, and training costs are relatively high.
- 3D Gaussian representation: Balances rendering efficiency and image quality, and is relatively friendly to incremental mapping.
The core of multi-sensor fusion is calibration and spatiotemporal synchronization: only with pixel-level alignment can category, instance, and geometry information be unified into the same coordinate system, which in turn supports queries about “object–supporting surface–spatial relation.”
4.5 From Scene Representation to Spatial Reasoning
A practical approach is to organize perception results into a structured scene graph, making spatial relations computable entities:
{"room": "kitchen","layout": {"length": 4.2, "width": 3.6, "height": 2.7},"objects": [{"id": "cabinet_3", "label": "wall cabinet", "layer": 3, "center_m": [0.8, 0.1, 1.6], "size_m": [0.6, 0.35, 0.7]},{"id": "counter_1", "label": "countertop", "top_height_m": 0.9, "center_m": [0.5, 0.1, 0.45]},{"id": "mug_7", "label": "cup", "center_m": [0.62, 0.09, 0.96], "relation": "on", "support": "counter_1"}],"openings": [{"label": "window", "sill_height_m": 1.0, "top_height_m": 2.2, "wall": "east"},{"label": "door", "sill_height_m": 0.0, "top_height_m": 2.1, "wall": "south"}]}
With a unified coordinate system and semantics, complex questions can be broken down into verifiable computational steps:
# Illustrative code: spatial relation queries based on a scene graph# Coordinates are in a metric right-handed system with one corner of the room as the originimport mathdef find_nearest(obj_id, pred, scene):src = next(o for o in scene["objects"] if o["id"] == obj_id)p = src["center_m"]cands = [o for o in scene["objects"]if o["id"] != obj_id and o["label"] == pred]if not cands:return Nonedist = lambda o: math.dist(p, o["center_m"])return min(cands, key=dist)def wall_offset(scene, obj_id):obj = next(o for o in scene["objects"] if o["id"] == obj_id)x, y, _ = obj["center_m"]length, width, _ = scene["layout"].values()sx, sy = obj["size_m"][:2]return min(x - sx / 2, length - x - sx / 2,y - sy / 2, width - y - sy / 2)scene = {...} # scene graph output by the perception and mapping modulesprint(find_nearest("mug_7", "wall cabinet", scene)["id"]) # Which object is nearest to the cupprint(wall_offset(scene, "mug_7")) # Distance from the cup to the nearest wall, in meters
4.6 Evaluation Must Also “Add a Dimension”
The evaluation system needs to be upgraded in parallel: in addition to conventional metrics such as intersection-over-union and mean average precision, it should introduce 3D position error, orientation error, reconstruction completeness, scale error, and spatial-relation query accuracy. Only when evaluation dimensions are aligned with business goals can model improvements translate into real-world usability.
5. Reliability Engineering: Making Agents “Usable, Manageable, and Traceable”
5.1 Structured Output and Tool Contracts
Every action a model outputs should be programmatically verifiable. Common practices include:
- Using strict structured output formats, such as JSON Schema or a restricted grammar, to lower the parse-failure rate;
- Defining clear names, parameter types, allowed ranges, and error codes for tools;
- Validating parameters and checking permissions before execution, and validating the result shape after execution;
- Introducing idempotency keys for write operations to prevent duplicate submissions from retries;
- Adding a human confirmation step for high-risk actions.
5.2 Execution Environment and State Management
Tool execution should take place in an isolated environment (such as a container or sandbox) that restricts filesystem, network, and system-call permissions. The intermediate state of long-running tasks should be written to a cache or object storage to support interruption recovery and result reuse, avoiding repeated computation.
5.3 Task-Level Evaluation
Evaluation granularity should be upgraded from “single-turn QA accuracy” to “task completion rate,” specifically including:
- Task success rate and first-attempt success rate;
- Average number of steps and timeout rate;
- Human handoff rate;
- Per-task cost and latency distribution (P50, P95).
5.4 Observability and Auditing
Record every model call, tool call, and decision basis to form a traceable chain. Connect anomalies and high-risk actions to monitoring alerts, providing a basis for post-incident review, accountability, and compliance auditing.
5.5 A Robust Execution Loop
# Illustrative code: an execution loop with budget and step guardsdef run_task(task, model, tools, policy):state = {"messages": [task["instruction"]], "steps": 0, "done": False}while not state["done"] and state["steps"] < policy["max_steps"]:state["steps"] += 1resp = model.chat(messages=state["messages"],tools=tools,schema=ACTION_SCHEMA, # structured output for easy validationreasoning=policy["reasoning"], # enable explicit reasoning only for deep tasks)action = resp.actionif action["type"] == "finish":state["done"] = Truereturn action["result"]if action["type"] == "ask":return {"status": "need_user_input", "question": action["question"]}# Validate parameters and permissions before calling, then execute once it passes; write operations carry an idempotency keyresult = execute(action, policy=policy)state["messages"].append({"tool_result": result, "id": action["id"]})if result["error"]:state["messages"].append({"hint": "Tool call failed; correct the parameters or use a different tool"})# When the budget is exhausted or the task is incomplete, degrade safely: return partial results and hand off to a human rather than continuing to consume computereturn {"status": "escalated", "state": state}
6. Trend Analysis: Three Signals Worth Watching
6.1 Compute Becomes a Core Production Factor
Compute is becoming a foundational resource much like energy, and its supply capacity directly determines the boundaries of AI products. On the supply side, capacity expansion, energy-consumption governance for AI computing centers, and innovation in compute scheduling and pricing mechanisms will directly affect the industry landscape. For enterprises, the pragmatic approach is to treat compute as a cost center that requires careful accounting: establish usage baselines, set budget guardrails, and continuously evaluate the cost per task.
6.2 The Divergence of Developer Roles
AI coding tools are widening the gap between developers, and this change is real:
- Proactive users: They apply AI to boilerplate code, test cases, API documentation, and log analysis, concentrating their energy on architecture design, boundary judgments, and quality control, so their output is significantly amplified.
- Passive outsourcers: They hand understanding, debugging, and trade-offs over to the tool entirely, lack the ability to localize problems when facing complex issues, and over time their core skills are weakened.
As a result, teams need to clearly define the boundary between “what AI can automate” and “what must be human-supervised”: generated code should go through review and automated testing, and safety-critical paths should have a clear human fallback mechanism. When judging technology trends, one should also return to primary sources such as original papers, official documentation, and public benchmarks, to avoid being misled by truncated conclusions.
6.3 Spatial Intelligence and Embodied Intelligence
Moving from 2D recognition to 3D understanding is a key leap from “seeing images” to “understanding the world.” It relies on multi-sensor fusion, 3D representation learning, and large-scale real-world scene data, and it directly determines whether directions such as robotics, autonomous driving, and industrial inspection can truly enter the physical world. For developers, mastering foundational skills such as point cloud processing, 3D reconstruction, scene graphs, and coordinate transformations will be a general skill worth investing in over the coming years.
7. Implementation Recommendations: An Action Checklist for Enterprise Teams
- Do the math: Establish a baseline centered on “cost per successful task” and incorporate it into regular operational reporting.
- Tier usage: Route models by task complexity; do not enable explicit reasoning or long context for simple tasks.
- Set guardrails: Set token budgets, step limits, and timeout policies for sessions and tasks; downgrade or hand off to a human when anomalies occur.
- Build evaluation: Construct task-level evaluation sets, and require regression whenever the model or prompts change.
- Strengthen contracts: Strictly define tool inputs and outputs, isolate the execution environment, and make write operations idempotent.
- Keep traces: Maintain full-chain logging and monitoring alerts to ensure traceability and auditability.
- Add spatial capability: Introduce 3D perception and scene understanding into vision-related businesses, and validate results with 3D metrics.
The value of AI lies not in how impressive a single answer is, but in whether it can reliably complete real tasks at controllable cost and run dependably in the physical world. Understanding the economics of compute, the boundaries of spatial cognition, and the essentials of engineering governance is the key to building a long-term advantage for developers and enterprises in this round of change.
