Large Models Enter the Physical World: Deconstructing Closed-Loop Correction for Real-Robot Manipulation

1. Event Background: A Capability Leap and a Talent Move

Public reports have presented a paired set of signals: a capability leap and a talent move.

A leading model provider released a new flagship model. On a widely used robotic-arm manipulation simulation benchmark, the model improved average task success rate from about 27.6% in the previous generation to about 64.9%. A mid-tier model from the same generation scored 39.4%, while a score of about 29.8% was reported for another mainstream model family.

More notable than the benchmark jump was what happened next. Third-party developers tested the model with a real robotic arm in a blind evaluation. After taking over the device, the model completed a single 62-step checkers game. During the task, physical contact deviations and end-effector deformation occurred frequently, yet the model continued to correct its actions.

At the same time, a dexterous manipulation engineer from a leading humanoid robot project confirmed that they would join the large-model team. Their background covers data collection, policy training, model evaluation, and real-robot deployment. They previously worked in a robotics research lab at a chip manufacturer, focusing on contact-rich manipulation and Sim-to-Real transfer.

Together, these two signals carry more information than either one alone:

• The capability-side signal is that general-purpose models are beginning to demonstrate long-horizon, closed-loop execution of physical tasks.
• The organization-side signal is that this capability cannot be obtained through general-purpose pre-training alone; it requires closing the loop between hardware, data, and deployment.

A single benchmark score improvement should still be treated as an early signal. There remains a substantial distance between such a result and direct usability in production environments. Later sections provide a quantitative perspective.

2. What Robotic-Manipulation Simulation Benchmarks Test and Miss

The evaluation logic of robotic-arm manipulation simulation benchmarks is essentially to decompose a complex task into a set of automatically checkable success conditions: whether the target object reaches the target pose, whether it falls into a container, and whether a sequence of actions is completed in the required order.

These benchmarks measure the combined capability of the chain from perception, to reasoning, to planning, to invoking low-level policies. They do not measure the performance of the physical robot itself.

A cross-generational improvement such as the jump from 27.6% to 64.9% is usually contributed by four overlapping sub-capabilities:

• Spatial and geometric reasoning: Inferring graspable poses, occlusion relationships, and reachability from pixels or point clouds, instead of applying fixed grasp templates.
• Task decomposition and subgoal generation: Expanding abstract instructions, such as “move the piece to the specified square,” into executable subgoal sequences and checking correctness under sequential dependencies.
• Failure detection: Recognizing that an action did not succeed. This is a key dividing line from open-loop execution.
• Long-horizon consistency: Maintaining task state, avoiding repeated actions, and not forgetting constraints across dozens of steps.

What benchmarks often do not cover is precisely the main cost of real deployment: calibration errors, latency jitter, sensor noise, grasp slip, workpiece tolerances, lighting and reflection changes, and the safety requirement that tasks may be slow but must never damage equipment or people.

Understanding this boundary is a prerequisite for all subsequent engineering decisions.

3. Why the 62-Step Real-Robot Run Matters: Success Decay over Long Horizons

A high success rate for a single action does not guarantee that an entire task can be completed. If each sub-step succeeds with probability p, and the steps are approximately independent, the success probability of the whole task is approximately p raised to the power n:

• When p = 0.99 and n = 62, the whole-task success rate is about 54%.
• When p = 0.999, the whole-task success rate reaches about 94%.

In other words, for a 62-step task to have a practically usable completion rate, per-step reliability must enter the “three-nines” range.

This is where closed-loop correction becomes valuable. Its role is not to raise the success rate of a single step. Its role is to cut off the multiplicative decay caused by accumulated errors and transform a chain-like structure, where one failure causes the entire task to collapse, into a branch-like structure where failures can be locally rolled back and recovered.

The typical disturbances faced in the blind test included:

• A piece not landing in the expected square.
• Unintended contact between the gripper and the edge of a piece, causing the piece to shift.
• Pose estimation drift caused by deformation of the end effector and fixtures.

An effective response path is to first identify the cause of the failure, then choose a recovery action, such as realigning, fine-tuning the approach pose, or replaying the previous sub-step when necessary, rather than mechanically retrying the same trajectory.

A minimal closed-loop control skeleton is shown below to illustrate the division of responsibilities across components:

while not task.done():
    obs      = sensors.capture()          # Multimodal observations: RGB-D, joint states, force/tactile
    state    = estimator.fuse(obs)        # Hand-eye calibration + time synchronization + target pose estimation
    if monitor.detect_anomaly(state):     # Failure detection: pose jumps, force limit exceeded, sub-step timeout
        subgoal = planner.replan(state)   # High-level replanning: local rollback rather than blind retry
    else:
        subgoal = planner.current_subgoal()
    action = motion_policy(state, subgoal)   # Low level: motion planning / learned manipulation policy
    action = safety_layer.clip(action)       # Speed limits, force caps, workspace and self-collision constraints
    robot.execute(action)
    metrics.log(state, action, monitor.flags)  # Full logging for offline attribution and regression evaluation

4. Contact-Rich Manipulation and Sim-to-Real: Four Engineering Priorities

Layered Architecture and Clear Responsibility Boundaries

General-purpose models are well suited to high-level responsibilities: understanding instructions, maintaining task state, detecting failures, and replanning. Low-level control should be handled by specialized modules with guaranteed real-time behavior, such as trajectory generation, impedance or compliant control, and grasp fine-tuning.

Having a large model directly output high-frequency joint commands is not cost-effective when latency, determinism, and safety are considered. Decoupling the two layers through a subgoal interface also makes each part easier to evaluate and replace separately.

The Often Underestimated Engineering Cost of the Observation Layer

Several observation-layer issues are easy to underestimate: hand-eye calibration accuracy, time synchronization between camera and joint data, point-cloud denoising, and coordinate system alignment. If any of these are not handled well, even strong high-level reasoning will be amplified by noise.

In engineering practice, it is better to make calibration and synchronization automatically verifiable checks rather than one-time configuration.

Standard Methods for Sim-to-Real Transfer

Common Sim-to-Real methods include:

• Domain randomization: Randomizing mass, friction, lighting, camera intrinsics, and latency ranges.
• System identification: Feeding measured parameters back into simulation.
• Real-robot data fine-tuning: Including iterative dataset aggregation, where newly discovered failure cases during deployment are added to the training set.
• Incorporating contact-force information.

Contact-rich tasks share a common feature: decision-making based on vision alone is often insufficient for stable success. Force or tactile feedback often determines whether the task succeeds.

Safety and Compliance Boundaries

Safety and compliance boundaries should include at least the following:

• Joint speed and torque limits.
• Workspace constraints and personnel isolation.
• Collision and gripping protection.
• An emergency-stop chain.
• Gray release after simulation rehearsal.
• Traceable logs for every real-robot run.

These are not optional items. They are prerequisites for entering real-world scenarios.

5. What the Talent Signal Reveals

Based on the public résumé, the newly joined engineer’s capability distribution maps directly to four stages of embodied deployment: data collection, policy training, model evaluation, and real-robot deployment.

This chain reveals the most common gap when general-purpose model teams enter the physical world. The gap is not a lack of algorithms, but a lack of people who can define evaluation sets, perform detailed failure attribution, and stably deploy policies onto real robots.

The implication for engineering teams is direct: embodied capability should be organized as a joint structure of “model team plus hardware team,” sharing the same data specifications, evaluation standards, and deployment workflows, rather than the model side delivering an interface while the hardware side adapts on its own.

6. Actionable Deployment Roadmap and Evaluation Checklist

It is advisable to proceed in three stages, with clear entry conditions for each stage:

1. Simulation stage: The goal is task-level success rate and the distribution of failure modes. This stage should produce reusable scenario libraries and randomization configurations, and automatically archive failure samples.
2. Controlled real-robot stage: Use fixed fixtures, fixed lighting, and limited speed. The goal is to verify how large the Sim-to-Real transfer gap is and to collect the first batch of real-robot data for fine-tuning.
3. Semi-open stage: Introduce safety constraints such as workpiece tolerances, occlusion, and the presence of people, and gradually increase the proportion of real-world scenarios through gray release and rate limiting.

Evaluation records should be structured and traceable to support horizontal comparison and regression testing:

{
  "task": "board_game_62steps",
  "steps_completed": 47,
  "termination": "recovered_and_completed",
  "failure_modes": ["contact_slip", "pose_drift"],
  "recovery_attempts": 3,
  "recovery_success_rate": 0.67,
  "safety_events": 0
}

The classification statistics in failure_modes and the recovery_success_rate are often more diagnostic than the overall success rate. The former indicates where improvements should be made, while the latter directly measures closed-loop correction capability.

At the infrastructure level, such projects typically require storage and retrieval of large multimodal datasets, elastic compute for training and inference, version consistency management for devices and models, and runtime logging, monitoring, and alerting.

Baidu AI Cloud can provide generic cloud building blocks such as object storage, a container platform, message queues, logging, and monitoring/alerting to support data loops and deployment operations. Specific choices should still be evaluated based on actual data scale and real-time requirements.

7. Common Pitfalls and Conclusion

Three pitfalls should be avoided:

• Treating a simulation benchmark score as equivalent to usability. There are multiple gaps involving calibration, latency, safety, and tolerances.
• Being misled by selective bias in demonstration videos. Long-horizon statistics should be conducted using the same failure-classification criteria.
• Allowing a large model to directly perform high-frequency low-level control, which sacrifices determinism and safety.

Overall, the real technical signal in this event is that general-purpose models have crossed a threshold on the perception-planning-correction loop for long-horizon tasks. The engineering signal is that the sources of capability are expanding from pure software to data and hardware.

For teams preparing to enter this space, a reasonable order is to first build the evaluation system, data loop, and safety boundaries, and then discuss model selection and scenario expansion.