Large Models and Embodied AI: Capability Limits and Practical Deployment Paths
This article examines whether general-purpose multimodal large models can control robots directly. It reviews evidence that their kinematics skills may come from memorized robot-specific data, compares response times and token costs with specialized policy models, and outlines a layered architecture that separates high-level planning from real-time control and safety. It also covers practical advice on data strategy and evaluation.
1. Background
Over the past one to two years, multimodal large models that combine vision, language, and reasoning have advanced rapidly. Many in the industry expect these models to bring a qualitative change to Embodied AI. Robots would no longer need hand-written control policies for each task. Instead, they would follow instructions, understand their surroundings, and manipulate objects directly, much as people do.
However, reality falls well short of the hype. Demonstration videos of a general-purpose large model operating a robotic arm, and the data in many papers on “end-to-end” robot policies, both need careful scrutiny. Based on discussions in the public research community and published benchmark results, this article analyzes the actual capabilities and fundamental limitations of general-purpose large models in robot control. It also proposes engineering approaches that are feasible in practice.
2. Core Question: Did Kinematics Ability Really “Emerge”?
2.1 The Logical Gap from Text Training to Robot Capability
A widely shared claim holds that after pretraining on massive amounts of text and image data, large models have “emerged” with the spatial reasoning needed to operate robots. Robotics researchers have identified a key weakness in this argument.
One researcher in the community tested a set of cutting-edge multimodal models. With deep reasoning mode turned off, the models received the joint angle vector of a robotic arm directly, for example q = [0.180, 0.029, 0.137, -2.128, -0.156, 1.973, 0.453]. They were then asked to compute the three-dimensional position of the end effector. This is essentially a forward kinematics (FK) calculation. The reverse problem is inverse kinematics (IK), which means computing joint angles from a given target pose.
Among dozens of evaluated models, only a very small number performed noticeably better than the rest. This is unusual in itself. If these abilities came from general physical and mathematical priors, the better-performing models should not be so rare.
2.2 Evidence of Overfitting: Cracks Appear When DH Parameters Change
The researcher then ran a “rule-change” experiment. The input DH parameters were modified. DH parameters (Denavit-Hartenberg parameters) are the standard method in robotics for describing the relative pose of adjacent links, and each link is defined by four parameters. A model that truly understood kinematic principles should compute results using the new parameters. Instead, the models kept producing results that matched the original DH parameters of one specific robot, a classic seven-axis collaborative robotic arm.
This is direct evidence that the models overfit to a particular robot. Their “kinematics ability” is not a general reasoning skill that emerged from text. It is a pattern memorized from robot-specific training data. In other words:
- This ability is not general. With a robot of a different configuration, performance drops significantly.
- The ability is hard to adjust on demand. Users cannot make the model adapt to their own robotic arm through prompting.
- The apparent generality in demonstrations largely reflects how broadly the training data covers different cases.
2.3 Implications for Developers
This finding has two implications for engineering practice:
- Do not trust out-of-the-box performance. If your robot’s configuration, joint limits, or tool coordinate frame differs from the mainstream platforms in the training data, a general-purpose large model’s control output may be unusable, and even unsafe.
- Data is the bottleneck. Whether a model can control your robot depends on whether it has seen data from robots like yours. Data strategy (collection, annotation, and coverage) therefore deserves more attention than model selection.
3. The Real-Time Bottleneck: Cloud Large Models Cannot Serve as Direct Controllers
3.1 Differences in Latency
Robot control has strict real-time requirements. Servo control cycles in industrial settings are typically on the millisecond scale. Even perception and decision layers usually need to respond within a few hundred milliseconds. Comparing a general-purpose large model used directly as the robot policy (the model outputs an action command at every step) with a specialized policy model reveals a large gap:
- A single inference call from a general-purpose large model can take on the order of ten seconds or more. Inference accounts for more than 90% of total task runtime, so the robot effectively “thinks one step, then stops.”
- A specialized vision-language-action (VLA) policy model has a latency of tens of milliseconds.
- A smaller specialized policy model has a latency of a few hundred milliseconds, which is already usable in many scenarios.
The gap between ten-plus seconds and tens of milliseconds spans more than two orders of magnitude. A cloud-hosted general-purpose large model therefore cannot physically handle a high-frequency control loop.
3.2 Token Costs Matter Too
Latency is not the only issue. In benchmark tests, general-purpose large models completed tasks under different orchestration approaches. The token consumption per successful episode ranged from over 100,000 to tens of millions. Even setting cost aside, this level of resource use means:
- Large-scale concurrent deployment is not feasible.
- Deployment in cost-sensitive commercial scenarios is impractical.
- Long contexts and multi-round calls create unpredictable overhead, which makes budgets hard to forecast.
3.3 Engineering Risks of Network Dependence
Another often overlooked issue is full-chain dependence. If a robot relies on a model hosted in a remote data center, network jitter, bandwidth fluctuations, and service outages all translate directly into robot freezes, or even loss of control. A responsible system needs to answer these questions:
- What is the P99 end-to-end latency in milliseconds? Does it meet the control cycle?
- If the cloud model becomes unavailable, does the robot have a local control stack as a safe fallback?
- Are there local emergency stop and joint limit protections that work independently of the model?
Demonstration videos rarely show these questions, but they are critical to safe operation in real deployments.
4. A More Practical Architecture: General Model Plus Specialized Control Models
General-purpose large models are neither general enough nor real-time enough to serve as controllers. So where should they fit in a larger system? The industry is gradually converging on a layered architecture.
4.1 Layered Responsibilities
- Planning layer (slow, intelligent): A general multimodal large model handles task understanding, task decomposition, semantic grounding (for example, turning “Put the red cup from the table into the dishwasher” into a sequence of subtasks), and anomaly explanation. This layer is called infrequently and can tolerate latency of seconds, which makes full use of the semantic and reasoning strengths of large models.
- **Policy
