How General-Purpose Large Models Are Reshaping Embodied AI

1. The First Impact Is on Situational Judgment, Not Just Mechanical Motion

Embodied AI systems have long faced a core problem: a robot’s ability to execute an action does not mean it can continue to complete a task after the environment changes.

In experimental environments with a fixed structure and unchanged object positions, engineering teams can write extensive rules in advance. For example, the target object is at a specified coordinate, the robotic arm moves along a fixed trajectory, the gripper closes at a preset height, and the object is then placed along a fixed path. Such approaches are deterministic, low-latency, and easy to validate, but they are highly sensitive to environmental changes.

Once an item on the table is moved, the orientation of a container changes, the target is occluded by another object, or the originally planned grasp position becomes unavailable, the robot must reassess:

• What task actually needs to be completed;
• Which objects are relevant to the task;
• Whether the operation order needs to change;
• Whether an alternative approach should be used after the original action fails;
• Whether the current information is sufficient, or whether the viewpoint needs to be moved or the scene observed again.

The value of general-purpose multimodal models is mainly reflected in these stages that require semantic understanding and on-the-fly reasoning. They can combine natural-language goals, visual observations, and historical execution results to form a higher-level judgment of task state, reducing the need to write separate rules for every scenario.

However, this capability should be understood as “improving system adaptability,” not as “the robot already has human-level autonomy.” A model can judge “which object should be picked up” without necessarily knowing how to complete the grasp with stable force and a precise trajectory.

2. From “Understanding the Task” to “Completing the Task,” Several Technical Layers Remain

A complete embodied AI task can be divided into at least four layers.

2.1 Task Understanding Layer

This layer parses user intent. For example, if a user says, “Put the tools that need to be used into the box, but do not move the hammer,” the system needs to identify:

• Which objects are the targets;
• Which object is excluded;
• How the final state of the task is defined;
• Whether there are implicit ordering constraints;
• Which actions could cause damage or safety risks.

General models have clear advantages in language understanding, goal recognition, and scene description, especially for tasks with incomplete expressions or flexible instructions.

2.2 Task Planning Layer

The planning layer needs to break the goal into steps, such as first clearing occlusions, then grasping the target tool, and finally checking whether any items are missing from the box. Here, the system must consider not only “what to do,” but also “what to do first” and “what to do after a failure.”

A model can output candidate plans, but a plan cannot be directly equated with an executable trajectory. Planning results still need constraint checks, including whether the target is reachable, whether the object can be grasped, whether the path has collision risks, and whether the current device supports the required action.

2.3 Skill Execution Layer

The skill execution layer converts abstract actions such as grasping, inserting, screwing, pushing, and transporting into concrete motion. It usually needs to combine modules such as visual localization, motion planning, inverse kinematics, trajectory generation, and gripper control.

Compared with asking a general model to directly generate actions for every control cycle, using validated skill modules is usually more reliable. A general model can select skills, provide target positions, or adjust parameters, while concrete execution is handled by real-time control programs.

2.4 Feedback and Safety Layer

This layer directly faces contact between the robot and the physical world, including information such as force, displacement, speed, joint state, collision signals, and grasp stability.

For example, if an object begins to slip during grasping, the control system must adjust the grip force or stop the action within a very short time. If every adjustment required waiting for the upper-layer model to reason again, the system would likely miss the best response time. Therefore, safety protection and low-latency control must remain in the local closed loop.

3. General Models Can Improve Generalization but Cannot Replace Real-Time Control

General models are best suited to solving “semantic and policy problems,” not all low-level control problems.

Consider a simple desktop organization task. A model can determine which items should be placed in a container and can decide, based on instructions, which object to move first. But during the actual grasp, the robot still needs to solve a series of details:

• Where the precise edges and graspable regions of the object are;
• Whether the gripper will hit nearby objects as it approaches;
• Whether the object surface is smooth, soft, or easily deformed;
• Whether sufficient friction is generated after the gripper closes;
• Whether the current arm pose is close to a singular position;
• Whether the object will hit the container edge during placement;
• Whether the original trajectory remains valid after the object moves.

These problems usually require high-frequency perception and control. A model’s reasoning cycle may be much longer than the low-level control cycle, and its output has some uncertainty. Therefore, a more reasonable design is to let the model provide goals, constraints, or policies, while dedicated motion and control modules perform real-time execution.

4. A More Feasible Near-Term Architecture: Layered Collaboration

At the current stage of technology, embodied AI is better suited to an architecture in which “general models handle upper-layer decisions, and specialized modules handle lower-layer execution.”

Upper Layer: Semantic Understanding and Task Orchestration

The upper-layer model is responsible for:

1. Parsing natural-language instructions;
2. Identifying key objects in the scene;
3. Establishing task state;
4. Generating candidate operation sequences;
5. Determining the cause of failures;
6. Switching to alternative strategies when necessary;
7. Deciding whether to continue, retry, or request human confirmation based on execution results.

The key at this layer is not to generate a complete plan in one pass, but to continuously update the plan as the environment changes.

Middle Layer: Skill Selection and Action Parameterization

The middle layer can maintain a set of validated robot skills, such as grasping, transporting, pushing, rotating, inserting, and placing. The upper-layer model does not need to directly generate control values for every joint. Instead, it calls these skills and provides the target object, target region, operation direction, and tolerance requirements.

This approach has two benefits: first, it reduces the direct impact of model output errors on the equipment; second, it makes it easier for engineering teams to independently test, tune, and safety-certify each skill.

Lower Layer: Motion Planning and Real-Time Control

The lower layer is responsible for computing executable trajectories and handling real-time feedback. It should provide:

• Collision detection;
• Speed and acceleration limits;
• Joint range constraints;
• End-effector pose constraints;
• Force control or impedance control;
• Grasp stability assessment;
• Abnormal stop and safe fallback.

When collision risk, force anomalies, or target loss are detected, the lower-layer system should be able to pause or undo the action directly without waiting for the upper-layer model to respond.

5. From Experimental Demos to Reliable Delivery, the Key Gap Is Continuity

Many robot systems can already complete demonstration tasks such as screwing, transporting, sorting, or simple assembly. But a single success does not prove that a system is suitable for production environments. The real concern is continuous operation capability.

A complete evaluation should at least examine the following metrics:

5.1 Single-Step Success Rate

This determines whether the robot can complete an independent action, such as picking up an object and placing it in a specified area.

5.2 Long-Sequence Success Rate

When multiple actions are executed in sequence, errors from previous steps accumulate. A system may perform well on single-step tasks but, after more than a dozen steps, show target drift, missed items, or incorrect state judgments.

5.3 Adaptability to Environmental Change

Object positions, lighting, occlusion levels, container orientations, and object categories need to be varied to observe whether the system can replan rather than relying on fixed templates.

5.4 Recovery Capability

When an action fails, can the system identify the cause? For example, a grasp failure may come from target localization error, insufficient grip force, path collision, or a change in object pose. Different causes should correspond to different recovery strategies.

5.5 Time and Resource Cost

If the robot must wait a long time for the model to return after every action, even a high success rate may not meet the efficiency requirements of warehousing, manufacturing, or service scenarios. Therefore, average task duration, number of model calls, perception waiting time, and local compute load all need to be included in evaluation.

5.6 Safety

The system must be able to restrict dangerous actions and avoid equipment damage, object damage, or personnel risk caused by model misjudgment. Safety policies cannot rely only on the language model’s self-constraints; they should be enforced by independent rule and control modules.

6. Training Data Must Extend from “Understanding the World” to “Interacting with the World”

General models can learn object relationships, spatial descriptions, and task semantics from large-scale image-text, video, and language data, but robots also need to deal with contact information in the real world.

For example, two objects that look similar may have completely different weight and friction characteristics; the same grasping action may produce different results on rigid and flexible objects; and an area that looks clear visually may still restrict the robotic arm’s actual motion due to its own structure or surrounding equipment.

Therefore, robot training data needs to cover:

• Multiple devices and end effectors;
• Objects of different materials, weights, and sizes;
• Contact actions such as grasping, pushing, and rotating;
• Complete processes of both success and failure;
• Sensor variations and occlusion conditions;
• Action interruption, retry, and recovery processes;
• Skill transfer results across different tasks.

Simulation environments can generate large numbers of variation conditions at low cost for testing planning and policies. But differences remain between simulation and reality, especially in friction, flexible deformation, sensor noise, and mechanical error. The final system must be validated on the target device; deliverability cannot be judged solely from simulation results.

7. Enterprises Should Prioritize Tasks with Clear Boundaries

General models do not mean that all robot tasks should immediately run in an open-ended manner. A more prudent approach is to prioritize the following types of tasks:

• A limited variety of objects;
• A relatively fixed workspace;
• Task goals that are easy to verify;
• Failures that do not cause serious consequences;
• Fast recovery through human intervention or the system;
• Clear criteria for determining success.

For example, item sorting, simple loading and unloading, desktop organization, and transport tasks with a limited number of steps are usually more suitable as early deployment scenarios than high-speed assembly, complex two-handed collaboration, or high-risk operations.

During implementation, enterprises should also establish task logs, video records, sensor data, and failure sample libraries. Every failure should answer three questions: Did the model misunderstand the task, was the plan unexecutable, or did the low-level control fail to keep up? Only by locating problems by layer can subsequent optimization avoid blindly adding data or blindly switching models.

8. Future Competition Will Shift from “Larger Models” to “More Reliable Systems”

The development of general models will continue to lower the barrier for robots to handle open-ended instructions, but the final outcome of embodied AI depends on the complete system, not a single model.

More important capabilities in the future include:

• More stable multimodal state estimation;
• More efficient task decomposition and replanning;
• Lower-latency model invocation mechanisms;
• Verifiable skill libraries;
• Action generation with safety boundaries;
• Skill transfer across devices and scenarios;
• Automatic diagnosis of failure causes;
• Collaborative training across simulation, real data, and online feedback.

Therefore, the most practical near-term route is not to let general models directly take over robots, but to establish clear responsibility boundaries: models handle understanding, planning, and decision-making; specialized algorithms handle trajectories and skills; real-time controllers handle execution and protection; and data systems handle logging and continuous improvement.

From this perspective, the real impact of general models on embodied AI is not only that robots can “understand more instructions,” but also that robot systems are moving from fixed-process automation toward adaptive systems that can perceive changes, adjust strategies, and complete tasks under constraints.