Where Do General Multimodal LLMs Get Their Robotics Capabilities?

    General multimodal large models are moving beyond digital tasks and into physical robot workflows, covering environment understanding, spatial reasoning, tool selection, motion planning, task decomposition, and execution feedback. This raises a key question: do these robotics abilities emerge naturally from general intelligence, or do they come from robotics-specific training data? This article works through that debate and offers practical ways to test it, including forward and inverse kinematics probes, spatial memory and long-horizon task evaluation, tool geometry adaptation, and the engineering work around data, evaluation, and safety. The conclusion is that today's robot capability comes from a combination of a general foundation model, robotics domain data, post-training alignment, and engineering constraints rather than from emergence alone.

    1. From the demos: general LLMs are entering the robot closed loop

    A notable shift has appeared in robotics recently: general multimodal large models are starting to enter robot manipulation, simulation design, and task execution. Instead of being limited to question answering, writing, code generation, and information organization in the digital world, they are increasingly taking part in physical-world tasks, including environment understanding, spatial reasoning, tool selection, motion planning, task decomposition, and execution feedback.

    In some demos, developers connect general models to robot simulation and design workflows. The model can read 3D scene information, understand the spatial relationships between parts, identify the positions of tools and target objects, and then adjust the tool’s shape, pose, and contact mode so that a robotic arm can perform fine operations such as bolt removal and component maintenance in a simulated environment. The significance of these demos is that they connect natural-language intent, 3D modeling, mechanical constraints, and robot execution, letting developers state task goals in a way that is closer to natural language and letting the model help generate executable task strategies.

    Another set of demos emphasizes autonomous action in unfamiliar environments. After a robot enters a room it has never seen, it must first explore the environment, identify walkable areas, movable objects, and key obstacles, and record the positions of target objects. Then, when a human gives a vague natural-language instruction such as “bring me that thing we usually use,” the system must combine visual perception, language understanding, and existing spatial memory to infer the target’s location and plan a movement and grasping path. If such abilities can be reproduced reliably, it suggests that the value of general models in robotics goes beyond language interaction and is starting to reach into a more complex loop of perception, decision-making, and execution.

    2. The core debate: emergent general intelligence or robotics data training

    These demos easily lead to one judgment: that general large models have naturally emerged with robotics capabilities. “Emergence” usually refers to a model, after its parameter scale, training data, task complexity, or post-training capability reaches a certain level, displaying abilities that were not explicitly designed. Robot manipulation does rely on many generalizable abilities, such as causal inference, spatial-relationship understanding, tool-function recognition, task decomposition, retry after failure, and cross-scenario transfer. If a general model can transfer these abstract abilities to the physical execution layer, it may indeed show characteristics resembling “robot intelligence.”

    But another explanation is equally plausible: the model may already have been exposed to a large amount of robotics-related data during training. Robotics data is not just video or text; it may also include descriptions of mechanical structures, joint-motion parameters, grasping trajectories, operation logs, simulation control records, sensor data, task-decomposition labels, failure cases, and evaluation feedback, along with a large volume of robotics-related technical documents and tutorials. Once this data enters the training corpus, the model can learn the motion patterns of specific mechanical structures, common ways of using tools, patterns of task-execution steps, and even the kinematic biases of certain common robot platforms.

    So the key question is not a simple either-or between “emergence” and “specialized training.” Rather, we need to judge whether the model can still maintain stable reasoning and execution under unseen robot structures, unfamiliar task environments, and new physical constraints. If it only performs well with familiar bodies, familiar scenes, and familiar task templates, then the so-called robotics capability may come mostly from data memorization. If it can make reasonable inferences under randomly generated new structures, new geometry, and new constraints, then it has stronger abstract transfer ability.

    3. Kinematics probes: detecting whether the model memorized a specific robot body

    An effective way to analyze where a general model’s robotics capability comes from is to use kinematics tasks as probes. Robot kinematics typically involve two basic types of problems.

    The first is forward kinematics. Given the angle of each joint, it computes the position and orientation of the robot’s end effector. Forward kinematics emphasizes forward computation, and its results are relatively deterministic, making it suitable for testing the model’s understanding of chain structures.

    The second is inverse kinematics. Given a target position and orientation, it works backward to find one or more sets of joint angles that satisfy the joint constraints. Inverse kinematics is usually more complex because it may involve multiple solutions, no solution, singular configurations, joint limits, collision constraints, and pose-optimization problems, which brings it closer to real robot-control scenarios.

    Illustrative pseudocode:

    1. def forward_kinematics(joint_angles, kinematic_chain):
    2. transform = identity_transform()
    3. for angle, link in zip(joint_angles, kinematic_chain):
    4. transform = transform.multiply(link.transform_for_angle(angle))
    5. return transform

    Inverse kinematics can be expressed as:

    1. def solve_inverse_kinematics(target_pose, kinematic_chain, joint_limits):
    2. candidates = search_joint_solutions(target_pose, kinematic_chain)
    3. valid = [candidate for candidate in candidates if satisfies_limits(candidate, joint_limits)]
    4. return select_best(valid)

    These tasks are well suited to judging the source of a model’s capability because they are highly sensitive to the robot’s body structure. Different robots differ in link lengths, joint types, number of degrees of freedom, joint-axis directions, mounting methods, and limit ranges, all of which affect the results. If the model has only learned some kind of “motion imitation” from video and language descriptions, it may perform well on common robotic-arm parameters but fail noticeably on randomly generated, never-before-seen robot structures.

    A controlled experiment can be designed as follows:

    1. Select the real kinematic parameters of a common seven-degree-of-freedom robotic arm and ask the model to complete forward-kinematics and inverse-kinematics tasks.
    2. Generate a batch of synthetic robot structures, randomly changing link lengths, joint order, number of degrees of freedom, joint types, and limit conditions, ensuring these structures do not appear in a fixed form in public corpora.
    3. Evaluate the model’s output on both task types, focusing on end-effector position error, orientation error, whether the joint solutions are executable, whether limits are violated, and whether the inverse solution converges within a reasonable range.
    4. Compare the model’s performance on real parameters versus synthetic parameters.

    If the model performs excellently on the real arm’s parameters but drops significantly on the synthetic, unseen structures, it may have memorized the kinematics of specific bodies with limited generalization. If it completes both task types reliably, it likely has strong abstract kinematic reasoning ability. If it performs poorly on both, the current capability is not yet enough to support complex robot control. If it performs worse on real parameters than on synthetic structures, then the evaluation metrics, prompting method, task distribution, and training data should be checked for bias.

    The value of this method is that it breaks “robotics capability” down into verifiable mathematical problems. Demo videos can be affected by scene design, task prompts, editing, and chance, whereas kinematics tests are closer to underlying capability and harder to mask with surface tricks.

    4. Spatial memory and long-horizon tasks: testing generalization in real settings

    Robot deployment relies not only on kinematic computation but also on the ability to continuously model the environment. Fetching an object in an unfamiliar room is a typical combined task. The system must complete several steps: explore the environment, identify passable areas, record object positions, handle occlusion and viewpoint changes, understand vague language instructions, plan a movement path, execute the grasping action, and re-observe and adjust after failure.

    When evaluating such capability, at least four dimensions should be considered.

    First, whether spatial memory is stable. Can the robot still remember the target’s location after moving around the room? If the target object is moved by a person, can the system update its existing memory? If the target is occluded, can the system infer its approximate position based on historical observations?

    Second, whether instruction understanding is sufficient. Vague instructions usually contain referential relationships, such as “that thing we usually use” or “the box in the corner.” The model needs to combine context, task history, scene common sense, and current visual information to determine which object best matches the user’s intent, while avoiding including dangerous, valuable, or immovable items as candidates.

    Third, whether long-horizon execution is reliable. Multi-step tasks are prone to error accumulation. The model needs sub-goal decomposition, progress checking, failure recovery, and retry strategies, rather than relying on a single planning result. Each step should revise the plan based on new observations.

    Fourth, whether behavior is safe. When a robot shares space with people, it cannot pursue only task success rate; it must also consider movement speed, human-robot distance, path smoothness, emergency-stop mechanisms, and exception handling. High-risk scenarios must introduce low-level control constraints and human supervision.

    The high-level model and the low-level control can be layered. The high-level model is responsible for understanding the task, generating sub-goals, and selecting tools, while the low-level control handles path tracking, joint limits, collision avoidance, and real-time stability. A simplified execution loop is shown below:

    1. while not task_done:
    2. scene = perception.get_scene()
    3. goal = language_to_goal(user_instruction, scene, memory)
    4. subgoals = planner.decompose(goal, robot_constraints)
    5. for subgoal in subgoals:
    6. plan = motion_planner.plan(scene, subgoal)
    7. action = low_level_controller.execute(plan)
    8. result = monitor.evaluate(action, scene)
    9. if failed(result):
    10. memory.update(result.failure_reason)
    11. replan = True
    12. break

    This layered design avoids placing the entire reliability burden on the large model. Large models excel at semantic understanding, task decomposition, and common-sense inference, but they are not naturally reliable with real-time physical constraints; low-level control excels at stability and safety but struggles with open-ended language tasks. Combining the two is what brings the system closer to an engineering-ready form.

    5. Tool use: geometric understanding, contact constraints, and action adaptation

    In simulation and design workflows, general models participating in tool-shape adjustment is another direction worth analyzing in depth. In real robot tasks, a tool is not an abstract label but something with dimensions, shape, grasping points, contact regions, and force constraints. Take a wrench-type tool as an example: the system must not only recognize that “this is a wrench” but also understand the opening width, handle length, insertion direction, reachable space, and rotation trajectory. If the tool does not match the target bolt’s position, the model must adjust the tool’s pose or even request a different tool.

    This capability can be tested through several kinds of tasks:

    • Given a bolt’s position and a tool model, output the tool’s grasping point and insertion direction;
    • When the tool collides with or is occluded by a component, adjust the path;
    • When the tool size is incompatible, recognize the failure and propose an alternative;
    • Among multiple tools, select the most suitable one based on task semantics;
    • Under contact-force constraints, judge whether tightening, prying, or removal actions can be completed.

    If the model has only memorized the fixed action sequences in demo videos, it will fail when the tool geometry changes. Real capability should be reflected in adaptability to new geometric relationships, new constraint conditions, and new task goals. Therefore, evaluation should actively change tool sizes, bolt positions, occlusion conditions, and task goals rather than merely repeating the original demo.

    From a technical perspective, tool-use capability involves several layers working together: the vision layer identifies the tool and the target; the geometry layer understands spatial relationships; the semantic layer judges the tool’s function; the planning layer generates the action sequence; and the control layer executes the contact and force control. General large models may provide strong support in the earlier layers, but the final layer still requires the robot control system to ensure safety and stability.

    6. From demo validation to engineering deployment: data, evaluation, and safety must be built together

    Even if a general model does have a certain level of robotics capability, an enterprise cannot immediately put it into production control. Robot systems operate in the physical world, where any mistake can cause equipment damage or personal injury. Engineering deployment must proceed along three main lines: data, evaluation, and safety.

    First, build a high-quality data closed loop. Robotics data is not just collected video; it also includes task goals, success and failure trajectories, sensor data, joint states, collision records, human-takeover logs, and scene metadata. Data governance should cover collection, cleaning, deduplication, anonymization, labeling, version management, and playback verification. For scenarios involving people, privacy compliance and access control also deserve attention.

    Second, build a reproducible evaluation set. The evaluation set should include both basic capabilities and combined tasks. Basic capabilities include kinematic computation, geometric reasoning, tool selection, and spatial memory; combined tasks include fetching, tidying, assembly, inspection, and human-robot collaboration. Each task should define success criteria, failure types, and safety metrics. Do not look only at the demo success rate; also look at recovery after failure, average task duration, number of collisions, frequency of human intervention, and result stability.

    Third, run cross-domain validation between simulation and reality. Simulation environments can generate large numbers of scenarios at low cost, but there is a gap with the real world, such as friction coefficients, sensor noise, lighting changes, load inertia, communication latency, and contact uncertainty. Techniques such as domain randomization, noise injection, real-data calibration, and evaluation processes that gradually approach the real environment can be used. For critical actions, validate in simulation first, then trial-run in a low-risk environment, and only then move into production.

    Fourth, safety mechanisms must be built in from the start. A robot system should have joint limits, speed limits, collision detection, an emergency-stop channel, exception rollback, and a human-takeover interface. The high-level model’s output cannot be treated directly as control commands; it should be filtered through a safety validator. For high-risk actions, multi-sensor confirmation, redundant judgment, and conservative strategies should be used.

    When building such a system on the cloud, general cloud-platform capabilities can help. For example, object storage can hold multimodal training data and runtime logs, a message queue can handle asynchronous tasks and status notifications, a container platform can deploy simulation, inference, and evaluation services, a logging service can record the full execution chain, and monitoring and alerting can surface abnormal tasks and performance fluctuations. This can improve the system’s observability and scalability, but the cloud platform is only infrastructure; true reliability still depends on model capability, data quality, and engineering governance.

    7. Conclusion: a general foundation and robotics data jointly shape capability

    General multimodal large models entering robotics do bring new possibilities. They may bring natural-language understanding, spatial reasoning, tool use, and multimodal perception into the robot task layer, reduce traditional robot systems’ dependence on fixed scripts, and change the way developers interact with robots. But demo success does not mean that general intelligence has naturally emerged with robotics capability. A more reasonable explanation is that the general foundation provides abstract abilities, robotics data provides domain knowledge, post-training provides task alignment, and the engineering system provides execution constraints. Together these four form the basis of today’s robot intelligence.

    Judging whether a model truly has robotics capability cannot rely on a single demo; instead, look at whether it can stay stable under unseen robot structures, new environments, new tools, and vague instructions. Kinematics probes, spatial-memory tests, tool-geometry adaptation, long-horizon task evaluation, and safety metrics are all more reliable judgment tools than a single-point demo. For enterprise developers, the key is not to chase concepts but to build a verifiable, reproducible, and scalable robot evaluation and deployment system. Only by testing model capability against real physical constraints can we tell how far it can really go.