General-Purpose Large Models in Robotics: Capability Boundaries and Practical System Design
This article examines where general-purpose large models fit in robotics and where they fall short. It breaks robot requirements into semantic, skill, and reflexive layers, then explains why stronger general models do not automatically eliminate the need for robot-specific models. Key constraints include data collection economics, latency budgets, continuous control uncertainty, and fundamentally different evaluation criteria. The article also highlights tactile sensing as a critical but often underexplored modality for contact-rich manipulation, and proposes a layered architecture in which general models handle planning, specialized models handle action generation, and robot controllers enforce safety. Practical guidance for evaluation and deployment is included.
When a general-purpose large model can generate robot action sequences that look reasonable, a sharper question follows: if large models keep getting stronger, why do robots still need their own dedicated models?
This is not an either-or question. It is a question of tool fit. A tool being able to solve a problem does not mean it is the most appropriate tool for that problem. When discussing the impact of general-purpose large models on embodied intelligence, the better question is not “What can existing models already do?” but “What do robots actually need?”
The answer can be understood by breaking the problem down into several key parts.
1. First, Break Down What Robots Actually Need
Robot system requirements can be roughly divided into three layers:
- Semantic layer: understanding instructions, recognizing scenes, and decomposing vague tasks into executable subgoals. For example: “Clear the cups from the table.”
- Skill layer: converting subgoals into trajectories, joint angles, or end-effector pose sequences. This includes manipulation primitives such as grasping, insertion, and turning.
- Reflex layer: millisecond-level closed-loop control that handles sudden contact-force changes, slipping, and posture deviation, and decides whether to stop or continue.
General-purpose large models have a clear advantage in the semantic layer. This comes from long-term training and the world knowledge and language grounding it provides.
However, the skill and reflex layers are about continuous control. They have strict requirements for latency, determinism, and physical consistency. These requirements are structurally different from the strength of cloud-based large models: generating a piece of text in one shot.
2. Usable Is Not the Same as the Best Fit
The industry has been trying to transfer capabilities from the digital world to the physical world: from vision-language models to vision-language-action models, from video generation models to world models.
These directions are all worth exploring, but the technical routes are not yet converged. The real constraints come from four areas.
Data collection economics
The training data for general-purpose large models comes from the internet, where the marginal cost is close to zero.
Robotics is different. One piece of high-quality real-robot manipulation data requires hardware, a physical space, safety boundaries, and people to collect it. If force or tactile sensors are added, both equipment cost and collection time increase further.
As a result, the optimal form of a robot model is often not simply “larger.” It is often a model that can generalize faster with limited data. This points toward designs that emphasize representation reuse, simulation augmentation, and cross-embodiment transfer, rather than merely increasing parameter count.
Latency budgets are a hard constraint
Cloud-side large-model inference typically takes hundreds of milliseconds. Joint-level servo control, however, often needs update frequencies ranging from tens to hundreds of hertz.
This gap cannot be smoothed away by “a little optimization.” It directly determines the system architecture: high-level planning can run in the cloud, but low-level closed-loop control must execute close to the robot body, often using smaller models on the edge.
Cloud-edge division is not a temporary workaround. It is determined by physical constraints.
Action dimensionality is different
Language generation outputs discrete tokens and has relatively high fault tolerance.
Robots output continuous control values. A single wrong command can be amplified by the system dynamics into a hardware collision.
Therefore, robot models need to explicitly model uncertainty. They may need to output distributions or confidence scores over candidate actions, and when confidence is low, proactively request confirmation from a higher level or switch to a safe policy.
This ability to “know what it does not know” is not a core requirement in general text tasks.
Evaluation criteria are completely different
Language models can be scored on benchmarks.
Robots must be accountable in the real world, often for long-horizon tasks. Metrics such as success rate, number of human interventions, single-task duration, and recovery ability after failure cannot be obtained through offline evaluation alone. They must be validated through closed-loop testing on real robots.
3. Put Tactile Feedback Back: The Underrated Last Centimeter
The tactile representation work we recently published was motivated by this issue.
In contact-rich tasks such as plug insertion, screw tightening, and picking up an egg, vision can fail. Once the hand enters the operation area, the target is occluded.
At that point, “can I grasp it?” does not depend on how much you see. It depends on how much contact force you can perceive.
Pure vision-based approaches often fail in the last centimeter on these tasks. The problem is not that the model is not large enough. The problem is that a key modality is missing.
But tactile data is harder to use for three reasons:
- Tactile signals are high-resolution, high-noise time-series data. Feeding them directly into a model creates excessively high dimensionality.
- Different sensors produce outputs with different scales and distributions, making cross-device transfer difficult.
- Tactile annotation depends on human labor, so dataset sizes are much smaller than in vision.
A common industry approach is to first compress tactile data into a low-dimensional, transferable representation space, then combine it with vision and robot state as input to a policy network:
# General pipeline idea for tactile representation# 1. Compress raw high-resolution signal into low-dimensional featurestactile_feat = tactile_encoder(raw_pressure_sequence) # (B, T, H, W) -> (B, D)# 2. Normalize and align distributions to reduce cross-sensor transfer issuestactile_feat = per_sensor_normalize(tactile_feat, calib_params)# 3. Fuse with vision and joint state for manipulation policyobs = concat([visual_feat, tactile_feat, joint_state])action = policy(obs)
There are two engineering points worth emphasizing.
First, calibration and normalization must be managed over the same lifecycle as model training. If the sensor model changes or the measurement range is adjusted, recalibration must be triggered. Otherwise, the representation space can be broken.
Second, the representation should be aligned across modalities. Tactile features should have a corresponding relationship with visual or language concepts such as “slip,” “hard,” or “jammed.” This allows text descriptions to guide tactile policies and reduces annotation dependence.
4. Recommended Architecture: Layered Collaboration, Not Either-or
The conclusion is division of labor.
A practical layered architecture could look like this:
# High level: general-purpose large model handles understanding, planning, and common-sense reasoning (cloud, seconds-level)plan = llm_planner(task_instruction, scene_description)# Mid level: skill/policy model generates actions (device-side, tens of milliseconds)actions = skill_policy(current_obs, plan.current_subgoal)# Low level: closed-loop control handles execution and recovery (robot controller, millisecond-level)execute_with_safety(actions, torque_limit=..., workspace_bound=...)
The responsibilities of each layer should be clear:
- The general-purpose large model provides “what is this, and what should be done?”
- The specialized model handles “how should it be done?”
- The robot controller ensures “it does not cause damage while being done.”
The three layers should be connected through explicit interfaces and state feedback. The upper layer should be able to replan. The lower layer should be able to report exceptions and trigger fallback.
For safety, it is best to design default safeguards. These include torque and speed limits, workspace boundaries, collision detection, an emergency-stop loop, and conservative strategies when confidence is low.
These safeguards should exist independently in the controller layer. They should not depend on the model’s “good judgment.”
5. Evaluation and Deployment Recommendations
Evaluation should cover three dimensions:
- Real-robot success rate, not just simulation success rate
- Long-horizon task completion
- Number of human interventions
It is recommended to build a fixed task set and a failure classification system. Failures should be categorized into perception failures, planning failures, and execution failures. This helps determine whether the root cause is a model problem or a hardware problem.
For deployment, a practical path is:
- First, use simulation plus a small amount of real-robot data to validate the full pipeline.
- Then, introduce tactile sensing for contact-rich tasks.
- Finally, gradually replace the high-level planning component.
The cloud side should handle model training, data processing, and simulation compute. The edge side should handle real-time inference. Containerized services can unify deployment and canary rollout, while logs and monitoring support online metric observation.
For most industrial scenarios, this path has the lowest risk.
Conclusion
The progress of general-purpose large models is clearly good news for embodied intelligence. It makes semantic understanding and task decomposition cheaper.
But robots ultimately operate in continuous control, real physics, and millisecond-level timing. General-purpose models alone cannot solve these problems.
The right approach is to acknowledge the boundary. General models should do what they are good at: cognition and planning. Specialized models should handle robot-body perception and fine-grained control. Long-neglected modalities such as touch should return to the place where they matter.
So, back to the original question: a usable tool is not necessarily the most appropriate tool. And that is exactly why dedicated models for embodied intelligence still have enormous room for innovation.
