Why Embodied AI Still Needs Purpose-Built Models Even When General LLMs Get Stronger

1. The Right Question: Capability Availability Is Not the Same as Capability Fit

Over the past two years, general multimodal LLMs and related digital-side models have started to show potential for robot tasks. In engineering terms, this progress has delivered three practical benefits:

1. Downward transfer of prior knowledge. Instruction understanding and object semantic recognition in open-vocabulary, open-scene settings no longer require per-class labeled datasets.
2. Long-horizon task decomposition and tool orchestration. A task such as “put the cup back in the cabinet” can be decomposed into subgoals and mapped to an existing skill library.
3. Few-shot adaptation. With a small number of demonstrations or natural-language corrections, a policy can adjust its behavioral tendencies.

However, these benefits mainly lower the entry barrier for high-level decision-making. They do not prove that a general model is the most suitable tool for robots. To evaluate whether a tool fits physical work, at least three questions must be answered: Which robot-critical capabilities does it lack? Which unnecessary capabilities does it bring? What costs are associated with these two types of error?

The first question relates to capability gaps. Gaps directly determine the upper limit of task success. The second relates to capability redundancy. Redundancy shows up in latency budgets, compute utilization, maintenance complexity, and safety exposure. General models are strongest in language, code, and open-ended semantics, while the core robot scenario is closed-loop, high-precision, strongly real-time, and irreversible contact work. Their capability distributions do not overlap.

2. The Technical Landscape Is Still Expanding, and No Route Has Been Settled

Moving digital-world capabilities into physical operation has produced several parallel routes, each with a clear capability boundary:

Paradigm	Input	Output	Main gap
Vision-language models	Images + text	Language, categories, regions	No temporal action representation; no contact-process modeling
Vision-language-action models	Images + text + proprioceptive state	Action sequences	Low action frequency, limited precision, long-horizon error accumulation, weak cross-embodiment transfer
Video generation models	Scene + instruction	Future frames	Visual realism does not imply physical consistency; contact forces and rigid-body constraints are hard to guarantee
World models / world-action models	State + action	Future states and consequence prediction	Model bias is amplified during long-horizon prediction; online correction is required

The key point is that robots need both fusion of multi-source heterogeneous information and prediction of action consequences. Fusion determines what the robot “sees,” while consequence prediction determines whether it is safe to act. These two capabilities are currently carried by different paradigms, and any single paradigm is only one slice of the overall capability space. Therefore, the final architecture should not be defined by existing technical categories; it should be defined backward from physical requirements. This is not pessimism about the route, but a reminder to treat classification as a toolbox, not an endpoint.

3. Embodied-Native Design: Reworking Inputs, Outputs, and Learning Objectives

“Embodied-native” is often misunderstood as “attaching an action interface to a digital model.” The real approach is to re-examine three parts.

Input side: Add the native blind spots of the digital world—force/torque, tactile, proprioception, proximity sensing, and temperature. The engineering difficulty is not mainly in algorithms, but in spatial-temporal alignment and calibration: sampling rates across modalities can range from tens of hertz to hundreds of hertz, and calibration drift directly contaminates policy input.

Output side: Define the action space and control frequency clearly, and put safety constraints into the system rather than hoping the model will “learn” them. Whether the output is joint torque, end-effector impedance, or pose trajectories determines whether the model can fit within a real-time budget.

Learning objective: Upgrade from “did the object reach the target?” to “is the contact process controllable?” For the same grasping task, success criteria can include peak contact force, number of slips, pose deviation, and recovery ability. Explicitly modeling failure modes is more valuable than simply raising average success rates.

Based on this, capabilities can be divided into four categories, avoiding the temptation to measure size by parameter count:

embodied_capability_profile:
  must_have:            # Cannot be missing; missing any leads to task failure
    - physical-modality perception (touch/force/distance)
    - high-frequency closed loop and state estimation
    - contact and geometric reasoning
    - safety envelope and failure fallback
    - cross-embodiment transfer
  absorb:               # Reuse general digital-side capabilities
    - open-semantic understanding
    - subgoal decomposition and skill orchestration
    - common-sense priors and anomaly explanation
  optional:             # Enable under conditions
    - video-style future prediction
    - limited code/tool invocation
  exclude:              # Explicitly excluded to avoid wasted compute and latency
    - general software-engineering capability benchmarking
    - long-document writing and open-ended chit-chat
    - deep question answering unrelated to robot operation

The conclusion is “neither too big nor too small”: no critical capability gaps and no unnecessary waste. Optimizing an embodied model toward general programming ability is a typical capability mismatch; conversely, treating distance and tactile perception as optional is even more dangerous.

4. Three Hard Constraints in the Physical World

First: real-time constraints and compute budget. The closed-loop period is the sum of sensor exposure and transmission, preprocessing, perception, policy inference, communication, and actuator response.

# Illustrative latency budget (assuming a 100 Hz end-effector closed loop, 10 ms period)
Sensing/actuation       ≈ 2 ms
Preprocessing + state estimation ≈ 1.5 ms
Policy inference         ≈ 5 ms   # maximum share for learned components
Communication and jitter buffer ≈ 1 ms
Margin                   ≈ 0.5 ms

This means a general large model cannot directly occupy the low-level loop. It can only be placed in the high-level layer of a hierarchical architecture: a slow semantic layer decoupled from a fast control layer, communicating through intent-level interfaces.

Second: data scarcity and long-tail distribution. Teleoperation data is expensive to collect, effective data density is low, and simulation differs from reality in contact dynamics and sensor noise. A practical approach is multi-source composition: pretraining in simulation + fine-tuning with a small amount of real data + active collection, prioritizing the most uncertain scenarios for data acquisition.

Third: irreversible consequences and verifiability. Errors in digital systems are usually rollbackable, but physical work is not. This requires three things: all learned components must be wrapped by deterministic safety monitors; policies must support uncertainty output and conservative fallback; and before deployment, there must be replayable, tiered evaluation.

5. Where Should General LLMs Sit?

+-------------------------------------------+
| Task semantic layer  0.2–2 Hz             |
| General LLM: instruction understanding,   |
| subgoal decomposition, constraint generation |
+-------------------------------------------+
| Skill policy layer  5–20 Hz               |
| Specialized policies + world-model prediction: |
| trajectories, contact parameters          |
+-------------------------------------------+
| Servo control layer  200–1000 Hz          |
| Controllers / impedance control:          |
| torque output, safety limits              |
+-------------------------------------------+

The interface principle is to exchange intent only, not low-level control quantities. High-level outputs should include subgoals, success criteria, constraint ranges, and recoverable states. The middle layer should run a feasibility checker; if validation fails, the system should return to replanning rather than let the model directly generate joint sequences.

6. Engineering and Evaluation Infrastructure

• Data pipeline: Unify time alignment and metadata schema; store raw data in object storage; use message queues for collection tasks and transcription; use containerized batch processing for cleaning and packaging; track collection throughput and frame-drop rate with logging, monitoring, and alerting.
• Evaluation system: Set tiered entry thresholds across single-skill success rate → contact-force metrics → task-level completion → long-horizon task stability → human intervention rate.
• Regression and staged rollout: Use fixed replay sets for version comparison; run in shadow mode first for parallel online observation; then gradually roll out by scenario.

7. Common Pitfalls

1. Using general benchmark scores as the selection target for embodied models.
2. Ignoring cross-embodiment transfer and binding policies to a single hardware setup.
3. Scaling data collection and training without closed-loop evaluation.
4. Outsourcing safety constraints to models instead of encoding them in deterministic layers.
5. Ignoring spatial-temporal alignment and feeding uncalibrated sensor data to policies.
6. Replacing “a more appropriate configuration” with “a larger model,” exhausting the latency budget.

8. Conclusion

As general LLMs become stronger, they make it clearer that the capability boundary issue is not a compute problem, but a configuration problem. General models should take on digital-side strengths such as semantic understanding, task decomposition, and anomaly explanation. Meanwhile, tactile sensing, force control, real-time closed loops, safety fallback, and cross-embodiment transfer must be handled by models redesigned around physical requirements. A practical path is a three-layer decoupling with intent-level interfaces, supported by unified data and evaluation infrastructure, so that every capability is placed at the time scale where it actually works best.