Beyond VLA/WAM: An Agentic Approach to Embodied Manipulation
This article examines a practical alternative to end-to-end VLA policies and world-model-driven action generation for embodied manipulation. It explains how a multimodal agent can decompose tasks, reason about spatial relationships, invoke geometry and control tools, and recover from failures while leaving low-level execution to the existing robotics stack. The core idea is to convert manipulation into structured contact constraints—contact position, contact pose, approach direction, grasp axis, phase state, success criteria, and retry strategy—and then solve those constraints through spatial computation. The article covers the feasibility of this agentic route, a contact-pose computation pipeline, engineering considerations such as calibration, pose construction, safety, failure recovery, and evaluation, and the boundaries of the approach.
1 Redefining the Problem: Embodied Manipulation Is Not Direct Action Generation
General multimodal models are moving from the digital world into the physical world. Common technical routes fall into two broad categories. One is end-to-end vision-language-action policies that directly map images and language instructions to action sequences. The other is world-model-driven action generation, which predicts environment evolution to assist decision-making. Both routes are advancing robot manipulation, but in engineering deployment they often face strong data dependencies, deeply coupled action spaces, and difficult cross-embodiment transfer.
Another route worth attention is the agentic approach. It does not require a base model to directly output low-level joint angles. Instead, the model handles task decomposition, target localization, spatial relationship reasoning, tool invocation, and failure recovery, breaking “manipulation” into intermediate constraints that can be computed and verified. The core idea can be summarized as follows: determine the contact position and contact pose for each manipulation phase. Consider grasping an object. Recognizing the target is only the first step. The system also needs to decide which part to grasp, from which direction the gripper should approach, and with what orientation to hold it. Once these judgments are converted into a target pose for the robotic arm and passed to motion planning and the low-level controller, the manipulation problem becomes a spatial constraint-solving problem.
Therefore, the key to agentic embodied manipulation is not to make a large model “generate actions out of thin air,” but to make it output structured constraints: contact point, approach direction, grasp axis, phase state, success criteria, and retry policy. Low-level control remains handled by mature kinematics, dynamics, and safety modules. This division of labor is closer to a layered software architecture: the high level handles semantics and spatial reasoning, while the low level handles real-time control and safe execution.
2 Feasibility of the Agentic Route: Planning, Perception, Memory, and Tool Use
General multimodal agents have several capabilities that naturally fit embodied manipulation.
First is task planning. Complex manipulation usually contains multiple phases, such as approach, alignment, contact, grasp, move, and place. An agent can split a natural language instruction into a phase state machine and define goals, constraints, and exit conditions for each phase. Compared with outputting a long action sequence in one pass, phased planning makes it easier to insert verification and recovery.
Second is spatial perception. Multimodal models can identify objects, parts, and operable regions from images and combine depth information to estimate spatial positions. The model does not necessarily need to output precise 3D coordinates directly. It can provide semantic constraints such as “grasp the handle,” “contact the rim of the cup,” or “approach from above,” and then a geometry module can compute precise coordinates.
Third is memory and feedback. Manipulation failures are often not caused by a single action error, but by contact point offset, collision along the approach direction, or unstable grasp orientation. An agent can record observations, actions, results, and failure reasons for each phase, forming short-term context and a long-term experience store for retries and strategy adjustment.
Fourth is tool invocation. An agent should not try to handle every computation itself. It should call tools such as segmentation, depth back-projection, normal estimation, hand-eye transformation, inverse kinematics, collision detection, trajectory planning, and low-level control. The model’s value lies in orchestrating tools and interpreting results, not in replacing all geometry and control algorithms.
The basic assumption of this route is that general multimodal agents already have strong semantic understanding and programmatic tool-calling capabilities. If embodied manipulation is abstracted into a suitable intermediate representation, complex tasks can be completed with lower data coupling. This approach does not reject end-to-end policies. Rather, it offers a complementary option that is interpretable, debuggable, and easy to integrate with existing robot stacks.
3 Core Method: Contact Pose Solving Driven by Spatial Computation
Once manipulation is converted into contact constraints, the core computation chain can be summarized as: pixels to space, space to contact, contact to pose, and pose to action.
The following illustrative pseudocode emphasizes the data flow rather than specific interfaces:
def solve_contact_pose(rgb, depth, K, T_wc, task):# 1. Semantic localization: obtain the target region and candidate contact regionmask = segment_target(rgb, task.target_name)contact_px = select_contact_point(mask, task.phase, task.prior)# 2. Pixel back-projection: from pixel and depth to camera coordinatesp_cam = backproject(contact_px, depth, K)# 3. Coordinate transformation: from camera coordinates to world coordinatesp_world = transform_point(T_wc, p_cam)# 4. Local geometry: estimate the contact point normal and tangent planen_cam = estimate_normal(depth, contact_px, K)n_world = transform_vector(T_wc, n_cam)# 5. Pose construction: build an orthonormal basis from the normal, approach direction, and grasp axisR_tcp = build_grasp_orientation(n_world, task.approach_hint)# 6. Output structured contact constraints for planning and control toolsreturn ContactPose(position=p_world, rotation=R_tcp,approach=task.approach_hint,stage=task.phase)
Pixel back-projection can be abstracted as follows: given the camera intrinsic matrix K, pixel coordinates (u, v), and depth d, obtain a 3D point in the camera coordinate frame; then transform it into world coordinates using the camera-to-world extrinsic matrix T_wc. A contact pose usually cannot be determined by a single normal alone. It also needs approach direction, grasp axis, or gripper symmetry constraints, ultimately constructing an orthonormal rotation matrix. For a two-finger gripper, the sign of the grasp axis direction may be equivalent, so pose verification should account for gripper symmetry.
The key to this chain is its “bridge” role. The multimodal agent converts semantic goals into candidate pixels, phase constraints, and tool invocation order. Geometry computation converts pixels and depth into spatial coordinates. Motion planning converts the target pose into an executable trajectory. Only with clear responsibilities across the three can generalization and reliability coexist.
4 Engineering Deployment: Calibration, Pose Construction, and Closed-Loop Recovery
In engineering practice, agentic embodied manipulation needs to address the following issues.
Calibration and coordinate consistency. Camera intrinsics, camera extrinsics, hand-eye transformation, and time synchronization must be managed consistently. Depth holes, reflective objects, and transparent objects all affect back-projection accuracy. In engineering, establish coordinate-frame naming conventions and output a traceable chain of coordinate transformations at each phase.
Pose construction and reachability verification. The contact point is not the gripper center, so a tool center point offset must be added. The approach direction must be checked against collision detection, joint limits, and workspace constraints. For symmetric grippers, equivalent poses should be allowed so that the planner does not incorrectly mark a pose unreachable due to sign differences.
Tool boundaries and safety constraints. The agent should output structured constraints, not unchecked low-level control commands. Before execution, the system should apply workspace limits, speed limits, force limits, collision detection, and emergency-stop logic. The high-level model is responsible for “what to do and where to contact,” while low-level control is responsible for “how to reach it safely.”
Phase state machines and failure recovery. Each phase should have clear success criteria, such as contact point error, grasp detection, torque threshold, or visual confirmation. After a failure, handle it by category: an approach failure can trigger another approach attempt; a contact offset can trigger re-estimation of the contact point; a grasp failure can trigger pose adjustment or selection of a different grasp region. The memory module should record failure reasons to avoid repeating ineffective actions.
Evaluation design. In addition to task success rate, evaluation should consider contact point error, pose error, number of retries, execution time, safety violations, and cross-object generalization. Simulation evaluation should randomize lighting, texture, pose, and physical parameters. Real-robot evaluation should cover occlusion, reflection, and dynamic disturbances.
5 Boundaries and Conclusion: Combining Spatial Reasoning with Embodied Know-How
The agentic route is not a universal solution. Contact dynamics, deformable objects, transparent and reflective objects, severe occlusion, and tasks with high real-time requirements still need specialized perception and control methods. Large models may also produce semantic misjudgments or spatial reasoning hallucinations, so geometric verification, tool feedback, and safety constraints must provide fallback protection.
A more reasonable conclusion is this: the boundary of a general large model’s physical action capability depends on whether it can translate semantic understanding into verifiable spatial constraints and form a stable interface with low-level embodied know-how. Spatial computation does not replace control. Instead, it turns “contact position and contact pose” into a common language between the high-level model and low-level actuators. For teams that want to explore complex physical manipulation with lower data coupling, agentic embodied manipulation offers a layered route worth practicing: let the model handle planning and spatial reasoning, let tools handle geometric computation, and let controllers handle safe execution.
