Beyond General-Purpose LLMs: Why Embodied AI Needs Native Models

1. Where the Problem Begins: Capability Spillover Is Not Capability Fit

In the past two years, flagship general-purpose large models have made continued progress in semantic understanding, long-chain reasoning, and code generation, and have even begun to show abilities in decomposing and planning robot tasks. A recurring question therefore emerges: if general-purpose models keep getting stronger, why do robots still need their own models?

My view is direct: a tool can solve a robot problem without being the best tool for robots.

This mismatch appears repeatedly in history. Image classification networks are strong, but they do not replace detection and segmentation. General-purpose speech models are strong, but in-vehicle wake-up, hearing aids, and industrial acoustic monitoring still require dedicated small models. General-purpose compilation is strong, but real-time kernels are still written in assembly. The reason is not that these specialized tools are “more advanced”; their constraint sets are different. Real-time performance, power consumption, failure cost, and explainability are not first-class citizens in general-purpose scenarios.

Therefore, the impact of large models on embodied intelligence cannot be inferred only from the supply side—what existing models can do. It must be worked backward from the demand side—what robot operation actually requires. Three criteria can be derived:

• Necessity: Is this capability actually invoked in a real operation chain? How frequently is it invoked, and how much does it affect task outcomes?
• Sufficiency: Can the current capability supply meet the accuracy and reliability required by the task, rather than merely “looking like it can do it”?
• Acceptable cost: Do the compute, latency, power, and data costs of acquiring a unit of capability fall within deployment boundaries?

2. Four Capability Transfer Routes and Their Shared Blind Spots

The industry is currently trying to transfer digital-world capabilities to the physical world along four main routes:

1. Vision-language models (VLMs): unify scenes and instructions into token sequences. Their strength is semantic grounding and open-vocabulary understanding.
2. Vision-language-action models (VLAs): discretize actions and merge them into the same token space. Their strength is end-to-end perception, reasoning, and action.
3. Video generation models: compress dynamics into pixel distributions. Their strength is providing imaginable and previewable futures.
4. World models / world-action models: couple prediction and decision-making. Their strength is supporting “think before acting.”

All four routes are advancing rapidly. However, when their implicit assumptions are abstracted, several shared blind spots become visible:

• The modality foundation is text- and image-centric. Language and RGB images are the two modalities with the easiest supervision signal on the internet, so models naturally lean toward them.
• Physical modalities lack native channels. Depth, touch, force, and proprioceptive states are often stitched together as “pseudo-images” or compressed into vectors, and geometric and temporal structures are damaged during preprocessing.
• Error and uncertainty are not explicitly represented. Models output point estimates, leaving downstream systems unable to judge whether a prediction is trustworthy.
• Latency is not treated as an architectural constraint. Training targets offline metrics, while deployment requires closed-loop control.
• Embodiment differences are ignored. Policies for different degrees of freedom, end effectors, and workspaces are not directly interchangeable.

These blind spots are not engineering defects of a single route. They are the result of objective functions and data sources. A more durable methodology is therefore embodiment-native design: rather than simply attaching an action interface to a digital model, rethink what a robot’s inputs are, what its outputs are, and what learning objectives it needs. Absorb validated capabilities from digital models, strengthen capabilities unique to robots, and remove capabilities that are not needed.

3. “Not Too Big, Not Too Small”: Define Embodied Models by Capability Configuration, Not Parameter Count

Here, “size” does not mean parameter count. It means capability configuration. It can be formalized as a capability budget five-tuple:

Parameter count / peak compute / inference latency / power and thermal envelope / data acquisition cost

Redundant capability is not free. It creates four types of opportunity cost:

• Latency: Closed-loop control has hard end-to-end inference time requirements. Excessive inference depth directly consumes safety margin.
• Power and memory: Edge compute and battery capacity are limited. Large language capabilities can crowd out resources needed by visual and tactile encoders.
• Failure surface: The more miscellaneous the capabilities, the harder it becomes to enumerate combinations of failure modes, and testing and certification costs grow superlinearly.
• Data cost: Every capability requires corresponding data, and real-robot data is the most expensive kind.

Conversely, the list of capabilities that are necessary but not native to the digital world almost directly maps to real operational failure modes:

Capability	Physical rationale	Typical failure when missing
Distance/depth perception	3D geometry cannot be reliably inferred from RGB alone	Missed grasps, collisions, depth ambiguity
Tactile/force sensing	Contact information exists only during contact	Slippage and dropped objects, damaged workpieces, assembly misalignment
Proprioception	Joint position, velocity, and torque are state variables	Trajectory drift, self-collision
High-frequency temporal alignment	Sensors have different sampling rates and delays	Visual-tactile mismatch, jitter
Coordinate-frame and rigid-transform management	Camera, base, end-effector, and object frames coexist	Systematic localization offsets
Failure detection and safe stopping	Physical actions are irreversible	Accidents, equipment damage

This leads to a clear conclusion: treating “matching the code capabilities of flagship general-purpose models” as the goal of embodied models is a goal mismatch. Full software engineering capability is not a hard requirement for robot operation. In contrast, perception and understanding of physical modalities such as distance and touch are not native to digital-world models, yet they are indispensable when robots perform high-precision, complex operations.

4. Three Hard Constraints from the Physical World

4.1 Input Side: No Sensor Is Error-Free

For embodied foundation models to perform high-precision complex operations, the premise is usable perception of physical modalities. However, no sensor is error-free, and tactile sensing is especially immature. Error sources can be decomposed into:

• Noise and quantization: thermal noise, ADC resolution, and channel crosstalk in pressure/piezoresistive arrays.
• Calibration error and drift: hand-eye extrinsics, temperature drift, deformation, and wear after long-term use.
• Geometric ambiguity: occlusion, reflection, and transparent or specular materials can make depth estimation fail.
• Temporal and spatial misalignment: multi-sensor sampling rates, transmission delays, and clock sources may not be consistent.
• Nonlinearity and hysteresis: tactile elastomers respond differently along loading and unloading paths.

The engineering implication is clear: the model cannot learn only a deterministic mapping. It must also learn the error structure and output confidence.

# Illustrative pseudocode: heteroscedastic uncertainty + precision-weighted fusion
class UncertaintyAwareFusion(nn.Module):
    def forward(self, batch):
        # Each modality branch outputs an estimate and a log variance
        mu_rgb,   logvar_rgb   = self.rgb_branch(batch["image"])
        mu_touch, logvar_touch = self.tactile_branch(batch["tactile"])
        mu_pose,  logvar_pose  = self.proprio_branch(batch["state"])
        # Precision = inverse variance: more reliable sources get higher weight
        p_rgb, p_touch, p_pose = (torch.exp(-logvar_rgb),
                                  torch.exp(-logvar_touch),
                                  torch.exp(-logvar_pose))
        mu  = (p_rgb * mu_rgb + p_touch * mu_touch + p_pose * mu_pose) \
              / (p_rgb + p_touch + p_pose + EPS)
        var = 1.0 / (p_rgb + p_touch + p_pose + EPS)
        return mu, var          # Variance is used downstream as action confidence for fallback decisions

Three supporting practices are needed:

1. Uncertainty modeling: heteroscedastic regression, evidential regression, deep ensembles, or posterior approximation. The core idea is to let downstream systems read a scalar confidence value.
2. Multimodal cross-validation: visual depth estimates and active ranging can constrain each other; visually inferred contact state can be cross-checked against tactile signals. If they disagree, the system should trigger recalibration rather than continue execution.
3. Online calibration: treat hand-eye extrinsics and sensor zero bias as estimable quantities, and correct them slowly using runtime residuals instead of treating factory calibration as a permanent constant.

4.2 Output Side: Actions Are Irreversible, and Closed Loops Are Latency-Sensitive

The second unavoidable problem is the cost of outputs. In the digital world, errors can be rolled back or rerun. In the physical world, errors may damage workpieces, break equipment, or injure people.

Latency must be designed in layers. A typical latency budget is as follows (order-of-magnitude illustration; actual values depend on hardware):

Layer	Typical frequency range	Carried modules	Latency budget
Low-level servo / impedance control	Hundreds of Hz to kHz	Onboard controller, not passing through the large model	Sub-millisecond to millisecond
Mid-level skill policies	Tens of Hz	Multimodal policy models, including visual/tactile encoding	Tens of milliseconds
High-level task planning	≤ 1 Hz	Large-parameter models, network round trips tolerated	Hundreds of milliseconds to seconds

This yields an important inference: the value of large models mainly lies in low- and medium-frequency layers. Directly inserting them into high-frequency closed loops is an architectural mistake. A reasonable form is slow thinking and fast execution, with safety monitoring always online and independent of model output.

def control_step(obs):
    # High frequency: impedance/force control; never passed through the large model
    u_base = low_level_ctrl(obs.joint_state, obs.tactile)
    # Mid frequency: a lightweight policy returns a correction and an uncertainty estimate
    if tick % K == 0:
        du, sigma = policy(obs)
        if sigma > SIGMA_MAX:
            request_replan_or_human(obs)   # Degrade when uncertain; do not guess blindly
        else:
            u_base = u_base + du
    # Safety filtering: independent of the model; hard constraints take priority
    return safety_filter(u_base,
                         joint_limits=LIMITS,
                         torque_max=T_MAX,
                         speed_max=V_MAX,
                         min_separation=D_SAFE)

Output-side constraints must also enter the model: joint limits, torque and power limits, velocity/acceleration smoothness, self-collision and external collision avoidance, and speed and minimum-distance monitoring in human-robot collaboration scenarios. Accordingly, training objectives should explicitly include risk terms—treat violating actions as strong negative samples, and treat safety boundaries as hard constraints rather than tradeable soft penalties.

4.3 Data Side: Real-Robot Data Is Scarce and Cannot Be Freely Parallelized

The third problem is data acquisition and distribution.

• High collection cost: real-robot data consumes device time, requires labeling labor, and needs space and safety investment. It cannot be scaled horizontally like scraping web text.
• Internal inconsistency: different operators have different demonstration styles, and success rates for repeated demonstrations of the same task are not equal. Averaging them directly can teach a “compromised bad habit.”
• Cross-embodiment differences: different degrees of freedom, end effectors, and workspaces prevent policies from transferring unchanged. Representations must explicitly handle embodiment conditions.
• Modality imbalance: RGB and language data are abundant, while tactile and force data are scarce. Device specifications, array layouts, units, and timestamps are also not unified.

The three types of data therefore need division of labor: real-robot demonstrations should be “few but accurate,” simulation and generated data should be “abundant but biased,” and human videos should be “abundant but without action labels.” The correct role of world models and video generation is to serve as dynamic priors and data augmentation—providing imaginable futures for previewing and filtering candidate actions. However, their pixel-level prediction errors accumulate over time, so they cannot directly serve as control inputs. Treating them as the final endpoint mistakes a tool for a goal.

The difficulty of tactile data is especially prominent. Before sensor specifications are standardized, any approach that directly applies fixed convolutional kernels to tactile arrays will bind hardware form into the weights. This is where tactile representation work becomes valuable: first abstract the modality into a device-independent, transferable representation, then discuss capability limits.

5. Tactile Sensing and Distance Perception: Filling a Gap Missing in the Digital World

Why tactile sensing is irreplaceable: the force distribution and micro-slip at the moment of contact exist only during contact. Vision fails under occlusion, transparency, high reflection, and flush surfaces. Tasks such as precision insertion, soft grasping, and cloth manipulation essentially depend on tactile closed loops.

Key design points for representation learning:

• Align the objective with downstream tasks rather than pixel reconstruction. Optimization targets should be evaluable quantities such as slip detection, material/hardness discrimination, assembly alignment, and force-control stability—not how visually appealing the reconstructed tactile image is.
• Abstract arrays into continuous fields. Use a unified representation of contact coordinates + normal/tangential pressure + time, and replace fixed-size convolutional kernels with positional encoding to enable transfer across sensors and array specifications.
• Self-supervised pretraining: apply masked prediction to tactile sequences to learn the temporal structure of contact events; use cross-modal alignment (tactile-visual-proprioceptive) to amplify sparse tactile supervision; use contrastive objectives to distinguish three states: no contact, stable contact, and slip.
• Explicitly model failure modes of distance perception: fuse active ranging and passive stereo vision, and provide a separate degraded path for transparent and specular objects rather than forcing a single model to output one depth value.

Evaluation methods must bind perception metrics to operation metrics: contact detection latency, slip detection rate, insertion alignment error, and drop rate are more persuasive than any offline reconstruction error.

6. Engineering Path: From “Runnable” to “Deliverable”

Architecturally, unify visual patches, tactile field tokens, proprioceptive state tokens, and language instruction tokens into the same sequence space. After the backbone, attach multiple heads: an action head, an uncertainty head, and a safety/failure-detection head. Sparse activation (MoE) can accommodate multiple tasks without significantly increasing inference latency, but memory usage and scheduling complexity must be evaluated for edge deployment.

Training should proceed in four stages to avoid aligning heterogeneous data all at once:

1. Single-modal self-supervised pretraining (vision and touch trained separately, each fully consuming unlabeled data);
2. Weakly supervised cross-modal temporal alignment (dependent on hardware synchronization and unified timestamps);
3. Behavior cloning or offline reinforcement learning on simulation and generated data to learn coarse-grained skills;
4. Fine-tuning with a small amount of real-robot data, combined with online safety evaluation and canary release.

Data and platform infrastructure can be supported by general-purpose capabilities: object storage for raw high-frequency time-series data; data versioning and lineage management for reproducibility; container platforms for elastic distributed training; logging services and monitoring alerts to track success rates and safety events; canary releases and shadow mode for pre-launch comparison of new policies.

The evaluation system should have four layers. Missing any layer can lead to misleading conclusions:

• Perception layer: depth error distribution, tactile classification/regression metrics, and cross-sensor transfer degradation;
• Single-skill layer: success rate, force/position error, and cycle time;
• Long-horizon layer: task completion rate, self-recovery after failure, and number of human interventions;
• System layer: end-to-end latency P99, safety violations per thousand actions, and energy consumption per task.

Three anti-patterns should be actively avoided: replacing physical evaluation with text benchmarks; looking only at average success rate while ignoring tail risk; replacing downstream task benefit with offline reconstruction error.

7. Conclusion: General-Purpose LLMs Are Capability Sources, Not the Final Form

General-purpose large models will continue to improve, and their contribution to embodied intelligence is real and significant: better semantic understanding, more reasonable task decomposition, and stronger code and tool-use capabilities will all be absorbed into embodied systems. But a capability source is not a capability endpoint.

The final form of embodied models should be defined from the needs of the physical world, not constrained by existing technical categories. Three actionable conclusions follow:

1. Elevate physical modalities to the same status as language, rather than treating them as patch-like inputs stitched in.
2. Treat uncertainty as a first-class output, so downstream systems can degrade, request replanning, or hand over to humans.
3. Treat latency, power, and failure cost as model design constraints, not engineering details left for deployment.

The goal is to make the tool “not too big, not too small”: no shortcomings in critical capabilities, and no unnecessary investment in nonessential capabilities. The technical route remains open, and exploration is ongoing, but the direction is clear: define capability boundaries from requirements, rather than selecting from the capability list of existing models.