General-Purpose Models Taking Over Robotics: Capability Shift, Data Bottlenecks, and VLA Engineering Practice
Summary: An international AI lab's decision to dissolve its dedicated robotics team and fold those researchers into core research and product groups is easy to read as "abandoning robotics." Look at it from a technical angle instead and the story changes: the boundary between robotics-specific models and general-purpose models is being erased by models that are simply more capable. This article walks through that main thread — how general-purpose and specialized models compare, the long-standing data bottleneck in robotics, the architecture and training of Vision-Language-Action (VLA) models, and engineering practices such as hierarchical decision-making, data engineering, and system evaluation. It closes with what this kind of organizational change means for developers and enterprises: not whether general models will replace robotics, but which layer of your system should be handled by a general model and which still needs a specialized module.
Industry media recently reported that a leading international AI lab has confirmed it is dissolving its standalone robotics team and moving those staff into core research and product application departments. The reason the lab gave: general-purpose models have demonstrated the ability to execute some robotic tasks directly, and the company wants to further connect robotics research with general-purpose model research.
An organization whose goal is general artificial intelligence pulling back a robotics-specific team is easy to read as “giving up on robotics.” But if you switch the lens from organization to technology, another fact appears: robotics research is not less important, and what is really happening is that the original division of labor between “robotics-specific models” and “general-purpose models” is being broken down by general-purpose models that are now more capable.
This article reviews that main thread from a technical perspective, including a capability comparison between general-purpose and specialized models, the long-standing data bottleneck in robotics research, the architecture and training approach of Vision-Language-Action (VLA) models, and engineering practices such as hierarchical decision-making, data engineering, and system evaluation. It ends with the implications of this kind of organizational change for developers and enterprises.
1. The signal behind the change: general-purpose models doing robotics directly
A landmark data point comes from third-party evaluation. A simulation test made public by a compute and chip vendor showed that a new-generation general-purpose multimodal model could generate robot actions directly from camera images and natural-language instructions, without any dedicated robotics training, achieving a 66.3% success rate on grasping tasks. When an agent system was wrapped around it to handle task decomposition, progress tracking, and failure retries, the success rate rose further to 74.8%.
For robotics researchers, the significance of these numbers lies in the comparison target: this was a model with no dedicated robotics training, used in its “general” form, and it clearly outperformed the contemporary results of traditional robotics-specific models in the test. Understanding the scene, deciding the next step, and correcting course based on execution results — capabilities that used to require robotics researchers to design and train each one individually — are moving inside the coverage radius of general-purpose models.
The organizational change is therefore better understood as a “shift in the center of capability gravity”: as general-purpose models become usable out of the box on more and more tasks, continuing to maintain robotics as an independent product line alongside language, code, and vision yields diminishing marginal returns. The more sensible approach is to treat robotics as one “embodied output modality” of a general-purpose model, where the general model supplies understanding, planning, and error correction, while the robotics team focuses on data, control, and system integration.
2. General-purpose vs. specialized robotics models: the center of gravity has moved
2.1 Specialized models: fine-grained control, limited generalization
Traditional robotics stacks usually adopt a layered “perception-planning-control” structure built from several specialized modules. Perception identifies and localizes objects, motion planning generates collision-free paths, the control module tracks trajectories in closed loop at a relatively high frequency, and the task layer uses state machines or behavior trees to decide execution order.
# Illustrative code: a "pick and place" pipeline assembled from a specialized model / classic stackdef pick_and_place(sensors, robot):# Perception: point cloud segmentation and pose estimation to localize the target objecttarget_pose = estimate_object_pose(sensors.depth, sensors.rgb)# Task layer: order is decided by fixed rules and depends on manual orchestrationsteps = ["approach", "grasp", "lift", "move_to_bin", "place"]for step in steps:if step in ("approach", "move_to_bin"):path = motion_planner.plan(robot.ee_pose, waypoints[step])robot.follow_trajectory(path) # high-frequency closed-loop trackingelif step == "grasp":robot.move_to(pre_grasp_offset) # pre-grasp poserobot.close_gripper()if not tactile_detects_object():return "grasp_failed"elif step == "place":robot.open_gripper()return "success"
This kind of stack is efficient and reliable in structured settings, but it generalizes poorly, for three main reasons:
- Module coupling: perception errors accumulate up the chain, and a failure in one module is hard to adjust for globally.
- High rule maintenance cost: task order and grasping strategies are largely hand-written, so any change of scene or object requires re-tuning and re-orchestration.
- Perception depends on calibration: pose estimation typically relies on camera calibration and object models, and accuracy degrades noticeably with transparent, reflective, heavily occluded, or unfamiliar objects.
2.2 General-purpose models: understanding, decision-making, and correction in one model
The advantage of a general-purpose multimodal model is that it handles observation and language with a single model. Take the instruction “put the red mug on the table into the grey storage box”:
- Semantic localization: “red mug” and “grey storage box” come straight from the instruction text; no predefined object categories are needed.
- Common-sense reasoning: it can infer “grasp the mug first, then approach the storage box,” and it can also understand implicit constraints such as “hold gently and avoid knocking other objects over.”
- Progress awareness: it can recognize that “the mug is already hovering above the storage box” and therefore decide whether the next move is to descend or to translate sideways.
The following illustrative code contrasts sharply with the traditional pipeline:
# Illustrative code: a general-purpose multimodal model performing "semantic localization + conditional judgment" directlydef vla_step(image, robot_state, instruction, history):obs = {"image": image,"state": robot_state, # joint angles, gripper opening, and other proprioception"history": history, # recent actions and completed steps}plan = multimodal_model.reason(obs, "Analyze the current scene, decide the next step, and judge whether the previous step succeeded")if plan.step == "grasp" and not plan.prev_step_ok:plan = multimodal_model.replan(obs, plan.failure_reason) # replan after a failurereturn plan.action
2.3 Acting still requires “embodied experience”
Understanding the scene and planning the steps is not the same as controlling the motion precisely. Where the grasp point should be, when the gripper should close, and how much force to apply at the moment of contact all require large amounts of “observation-action-result” data to learn well. That is also why VLA models still have to be trained on robot trajectory data on top of general vision-language ability. The reasonable conclusion is not “specialized models are obsolete” but rather that the center of gravity has shifted: high-level understanding and decision-making go to the general model, while low-level control still depends on a specifically trained policy.
3. The data bottleneck: a problem robotics research cannot sidestep
3.1 Pretraining: the foundation of general capability, and exactly what robotics lacks
Pretraining means first learning reusable knowledge and representations from massive general-purpose data, then transferring them to specific tasks. Language models learn grammar, facts, and reasoning patterns by predicting the next token, and vision models learn object and scene structure from large volumes of images — all of which happens entirely in the digital world.
Robotics, by contrast, has to pair “what is seen” with “what is done”: where the hand should move, when the gripper should close, and whether an object will be pushed or roll away after contact. This kind of correspondence can only be obtained through interaction. The difference in how the two data types are collected is stark:
| Comparison dimension | General data (language / vision) | Robot interaction data |
|---|---|---|
| Data form | Static content such as text and images | “Observation-action-result” temporal trajectories |
| Collection method | Natural accumulation on the internet | Teleoperation, autonomous collection, simulation |
| Collection cost | Close to zero marginal cost | Hardware, facilities, operators, maintenance |
| Content diversity | Massive topics, inherently broad coverage | Limited by tasks, objects, and scenes |
| Data scale | Hundreds of millions to tens of billions of items | Typically hundreds to tens of thousands of trajectories |
Notably, robotics is also continuously absorbing pretrained knowledge from the internet. The Vision-Language-Action (VLA) approach proposed by one research team jointly trains the pretrained weights of a general vision-language model with robot trajectory data. Language and vision knowledge improve the model’s ability to understand new objects and new instructions, while robot data is responsible for establishing the correspondence between observation and action. Combining the two is precisely a data strategy of “general pretraining as the base, specialized data for alignment.”
3.2 A classic case: reinforcement learning and domain randomization
In 2019, a leading international AI lab published a widely followed study: reinforcement learning was used to train a neural network to control a five-fingered robotic hand to solve a Rubik’s cube, and the resulting capability was then transferred to a real robotic hand. Reinforcement learning improves control through a loop of “trial and error, receiving a reward, updating the policy,” letting the robot refine its control according to the results of its actions. The technical points can be summarized as follows:
- Task decomposition: solving the cube was broken into subtasks such as rotating the center sections, reorienting the cube as a whole, and solving it layer by layer.
- Large-scale parallel simulation: many copies can run simultaneously in a virtual environment, quickly accumulating interaction data.
- Domain randomization: parameters such as cube size and finger friction are randomized to force the policy to adapt to a wide range of physical conditions.
- Real-robot fine-tuning: training continues on a real robotic hand to cover the parts that simulation cannot model completely.
Illustrative code for domain randomization:
# Illustrative code: domain randomization during simulation trainingdef domain_randomize(scene, rng):# Sample different physical parameters each time the scene is reset per episodescene.cube.size = rng.uniform(0.05, 0.08) # object sizescene.cube.mass = rng.uniform(0.03, 0.10) # object massscene.finger.friction = rng.uniform(0.5, 1.5) # finger friction coefficientscene.action_delay = rng.uniform(0.0, 0.05) # communication and control delay (seconds)scene.observation_noise = rng.normal(0, 0.01) # sensor noisereturn scene
The core idea of domain randomization is this: rather than making the simulation approximate one particular “real world,” make it cover a wide enough range of conditions to force the policy to learn robust behavior that does not depend on exact physical parameters, thereby narrowing the sim-to-real gap.
However, a single-task policy only solves “how to turn one Rubik’s cube.” Restricted collection methods, small data scale, and a lack of cross-task generality eventually led that line of research to be paused for lack of data. Over the same period, data reserves in language, code, and imagery grew rapidly at very low cost, making them far better suited to supporting the long-term general-AI route. This case illustrates precisely that what determines progress in robotics research is not only the algorithm itself — the way data is supplied matters just as much.
3.3 Where does the data come from: a four-layer pyramid
The industry has gradually converged on a multi-tier data sourcing strategy that can be summarized as a pyramid:
- Bottom layer: internet data. Provides the model with language and vision common sense, for example “a cup should be held upright” or “eggs are fragile.” Large in scale and cheap, but lacking action labels.
- Second layer: human video. Shows how people complete tasks, from which manipulation strategies and object interaction patterns can be learned. Relatively large in scale, but lacking precise action labels, and human hands differ in morphology from a mechanical gripper.
- Third layer: simulation data. Controllable in scale, collectible in parallel, and supportive of domain randomization and automatic task generation. The difficulty lies in physical modeling and visual realism.
- Top layer: real-robot data. The highest quality, with observation and action in strict correspondence, but the most expensive to collect — and the key to a model truly “learning to use its hands.”
For a team, a sensible data strategy is not to maximize a single source but to organize data along the proportions of the pyramid: internet data secures semantic understanding, video and simulation data expand scenario coverage, and real-robot data aligns the details of action.
4. VLA models: “translating” internet knowledge into actions
4.1 Structure and inference flow
A VLA (Vision-Language-Action) model is a paradigm that unifies vision, language, and action in a single model. A typical structure has four parts:
- Vision encoder: converts camera images into visual features.
- Language model backbone: processes natural-language instructions and models them jointly with visual features, usually initialized from a general vision-language model.
- State encoder: encodes the robot’s own state, such as joint angles and gripper opening.
- Action head: decodes the model’s internal representation into concrete actions, such as end-effector pose deltas, joint targets, or discrete action tokens.
# Illustrative code: one inference pass of a VLA modeldef vla_predict(image, joint_state, instruction, history):visual_tokens = vision_encoder(image) # image encodingtext_tokens = tokenize(instruction) # instruction encodingstate_tokens = state_encoder(joint_state) # proprioceptive state encoding# The general backbone processes all three inputs togetherfeatures = vla_backbone(visual_tokens, text_tokens, state_tokens, history)# Decode into an action sequence the robot can executereturn action_head(features) # pose delta / joint targets / action tokens
At inference time the model can output a sequence of actions for the next several steps at once (so-called “action chunking”), which is then handed to a low-level controller for high-frequency tracking. This both reduces the number of inference calls, mitigating the impact of large-model inference latency, and ensures temporal coherence between actions.
4.2 Training: general knowledge as the base, robot data for alignment
VLA training is generally split into two stages:
- Pretraining alignment: starting from a general vision-language model, continue learning joint visual-linguistic representations on image-text data so that the model “understands the scene and understands the instruction.”
- Action fine-tuning: train the action head on robot trajectory data so the model maps “the scene it understands” to “the correct action.” Common approaches include discretizing actions into tokens and treating prediction as classification, or using a continuous action regression loss.
In the end, the model has two kinds of capability at once: understanding unseen objects and newly phrased instructions (from general knowledge), and knowing how the robot should actually move in its current state (from robot data).
4.3 Engineering considerations
- Observation space design: the combination of multi-camera views, depth information, and a history window of actions directly affects perception under occlusion and during fast motion.
- Action representation: end-effector pose deltas are friendlier to controllers, joint-space targets are more direct, and discrete action tokens make unified modeling easier — each has trade-offs.
- Real-time behavior: model inference usually cannot match the frequency of low-level control, so it must work together with a high-frequency controller, or reduce call frequency through action chunking.
- Failure injection: including failures, retries, and unexpected contacts in training data lets the model learn to handle “non-ideal states” rather than only the smooth path shown in demonstrations.
5. Hierarchical architecture: the general model “thinks,” the control policy “moves”
In the test described earlier, adding the agent system raised the success rate from 66.3% to 74.8%. It did not strengthen the model’s own “hand-eye ability”; it compensated for the weaknesses of the end-to-end approach on long-horizon tasks:
- Task decomposition: breaking “tidy up the desk” into executable subtasks such as “pick up the mug, put it in the storage box, straighten the book.”
- Progress tracking: combining action history with the latest frame to judge which step is current.
- Failure diagnosis and retry: when a grasp slips, deciding whether to change the grasp point or adjust the approach angle.
This is a typical “slow thinking, fast execution” hierarchical architecture:
- Task layer: the general model handles task planning, progress assessment, and exception handling.
- Skill layer: short skills such as grasping, moving, and placing are executed by specifically trained visuomotor policies.
- Control layer: joint-level controllers run closed loops at high frequency to guarantee real-time behavior and safety constraints.
# Illustrative code: hierarchical architecture with general-model planning + skill execution + failure retrydef robot_task_loop(instruction, model, skills, limit=5):plan = model.plan(instruction) # task layer: decompose into a list of subtasksfor attempt in range(limit):for step in plan.remaining():# Skill layer: call a trained visuomotor policy to complete one subtaskoutcome = skills[step.skill](step.params)# Task layer: judge success or failure from the latest frame and the execution resultverdict = model.check(step, outcome)if verdict.done:return "success"if not verdict.ok:# On failure, revise the plan using the frame and the failure reasonplan = model.revise(plan, step, verdict.reason)break # go back to the outer loop and re-execute the remaining stepsreturn "max_retries_exceeded"
The illustrative code above omits implementation details such as error handling; in real engineering, model outputs also have to go through structured validation and safety checks. What must be emphasized is that the value of the agent layer rests on the premise that the low-level skills work: if the grasping policy itself cannot reliably complete a single grasp, no amount of retries will make up for it. The improvement from 66.3% to 74.8% shows that there is still considerable room in task planning and error correction, while the remaining failures most likely come from low-level execution itself.
6. From research to system: key considerations for engineering deployment
6.1 Real-time behavior and latency
Robot control is latency-sensitive, and the inference latency of a large model determines which layer of decision-making it can participate in. A common practice is to place model inference at the task layer operating on a “seconds” timescale and leave millisecond-scale control to a lightweight controller. Fallback strategies should also be designed for network jitter and model timeouts, so that the system always has one working control loop.
6.2 Safety and reliability
“Can understand and can plan” is not the same as “ready for commercial use.” Industrial and service scenarios have explicit safety requirements, which typically include:
- A safety layer independent of the model, with torque and velocity limits and collision detection;
- Emergency stop and human takeover mechanisms;
- Complete operation logs, enabling incident tracing and model iteration.
The safety layer should not depend on the model “not making mistakes” but should serve as the last line of defense.
6.3 Data loop and cloud engineering
Because a gap exists between simulation and real robots, failure cases produced during real-robot operation need to be fed back into training to form a data loop. On the cloud side, common industry capabilities can be used to build the supporting infrastructure:
- Object storage: centrally storing trajectories, images, and logs;
- Message queues: buffering data sent back from devices;
- Container platforms: elastic scheduling of batch simulation and model training jobs;
- Logging services and monitoring/alerting: tracking device online rates, task success rates, and inference latency.
Baidu AI Cloud, for example, offers general-purpose products in these categories that can cover data processing and training scheduling needs of this kind. Teams can compare options through their own testing against their constraints on latency, cost, and data compliance before selecting.
6.4 Evaluation methodology
Evaluation conclusions depend heavily on test conditions, and the following dimensions can be used as a reference:
- Success rate: distinguishing first-attempt success from success after multiple attempts;
- Generalization: varying object categories, backgrounds, lighting, and instruction phrasing;
- Efficiency: task completion time and number of action steps;
- Failure mode classification: perception errors, planning errors, execution failures, safety incidents;
- Reproduction conditions: publishing the robot model, scene configuration, and evaluation scripts.
7. Implications for developers and enterprises
- Re-examine the “specialized vs. general” boundary: as general-model capabilities keep spilling over, systems assembled from rules and specialized models need to be assessed for their potential to migrate to general-purpose models.
- Put data strategy at the center: in robotics deployment, algorithm selection is only the starting point; how data is acquired and how efficient the data loop is often determine the ceiling of a project.
- Adopt hierarchical design: let the general model handle “understanding and decision-making” and let specifically trained modules handle “execution and control,” rather than trying to solve every problem with one large model.
- Prioritize reusing existing capabilities: general models, general data, and evaluation benchmarks are all reusable assets. A team reorganization like this lab’s is essentially about reducing duplicated work and concentrating resources on a reusable foundation.
Conclusion
A leading international AI lab has again dissolved its standalone robotics team, which shows that “general-purpose models doing robotics directly” has moved from concept to quantifiable test results. But the underlying logic of this trend is still the core proposition that robotics research has wrestled with for years: data determines the ceiling of capability, and general knowledge determines the boundary of generalization.
For developers and teams, the more valuable question is not “will general-purpose models replace robotics” but “in my system, which layer should the general model own, and which layer still needs a specialized module.” Drawing that boundary clearly matters more than chasing any single model.
