Perception, Planning, and Error Recovery for Long-Horizon Embodied Manipulation

Long-horizon embodied manipulation is not “chaining together single-step grasp models.” A complete task often involves dozens of contact events: approach, alignment, grasping, pushing, release, and replanning. Millimeter-level deviations at any step, gripper compliance, table friction, or camera extrinsic drift can invalidate the preconditions for every subsequent action. The engineering problem is to maintain task progress under uncertain physical contact, and to pull the policy back onto a feasible trajectory with interpretable state after a mistake.

1. Define Executable Task States First

A long-horizon sequence must be able to answer three questions: which subgoal are we currently in, which physical constraints still hold, and which checkpoint can we return to after a failure?

Write the task as a finite state machine rather than a natural language instruction. Each state should at least include:

• The relative pose interval between the target object and the end effector
• The contact mode (no contact, point contact, surface contact, closed grasp)
• Allowed force or torque bounds, plus timeouts
• Success and failure predicates
• The previous stable state to fall back to

A minimal structure can look like this:

state: align_piece
pre: gripper_open AND piece_visible
success: lateral_error < tol AND contact_force in [f_min, f_max]
fail: timeout OR slip OR occlusion_frames > N
recover: reopen -> retreat -> redetect -> align_piece

Predicates must be measurable by sensors—avoid “it looks about right.” Visual detection confidence, abrupt end-effector force changes, joint tracking errors, and gripper open/close feedback should all feed the same state machine, not just sit in logs for post-hoc review.

2. Layered Control: Planning Handles Progress, Low-Level Handles Contact

General-purpose large models or task planners are well suited to producing subgoals and tool-call sequences; contact-rich actions should still be executed by short-horizon policies or motion controllers. Merging them into one layer leads to two failure modes: the planner over-intervenes in per-frame joint commands, amplifying latency and hallucinations; or the low-level policy lacks task context and achieves local success while breaking global, board-like constraints (e.g., knocking an already-placed object out of its target zone).

A recommended three-layer architecture:

1. Task layer: maintains the subgoal graph, checkpoints, and termination conditions; replans only on state transitions or anomalies.
2. Skill layer: short-horizon policies for grasping, pushing, insertion, and placement. Inputs are local observations and relative target poses; outputs are end-effector deltas or joint targets.
3. Safety layer: force, velocity, workspace, and self-collision limits, with highest priority and the ability to stop independently.

The task layer’s output should be “goals and constraints,” not frame-by-frame trajectories. The skill layer operates in closed loop within those constraints. When a physical contact error occurs, the safety layer first limits damage, the skill layer attempts local recovery, and only if local recovery fails is the event escalated to the task layer.

3. Recovery Should Target the Error Source, Not Blindly Retry

Failures on real hardware fall roughly into four categories, and the recovery actions should differ accordingly.

Perception errors: missed detections, wrong associations, depth holes, extrinsic changes. Recovery is re-observation: change the viewpoint, adjust lighting, or wait for stable frames, and filter single-frame detections with multi-frame geometric consistency. Do not keep closing the loop on a wrong target.

Geometry and calibration errors: outdated tool coordinate frames, camera-to-base transforms, or gripper fingertip models. These manifest as systematic bias toward one side. Recovery should trigger a calibration check or online bias estimation, not larger motion amplitudes.

Contact and deformation: object slipping, pose changes after pushing, flexible-part bending, actuator compliance. Force signals often precede vision. Use force transients as events—retreat to a contact-free safe pose, then re-estimate the object pose.

Policy out-of-distribution: friction, lighting, and occlusion conditions rarely seen in simulation. Local retries should have a hard cap; once exceeded, degrade to a more conservative skill or request a human reset checkpoint to avoid oscillating in a bad contact.

Recovery logs should record: the triggering predicate, the state rolled back to, and whether recovery changed the topology of the object set (whether objects were knocked out of their stations). Counting only “final success” masks unacceptable mid-task collisions.

4. Sim-to-Real: Transfer the Distribution, Not a Single Success Rate

Simulation is good for amplifying contact samples and auto-labeling failure types, but it cannot replace real-hardware closed loops. When narrowing the gap, prioritize aligning these quantities:

• Observations: image noise, latency, detection dropouts, and depth bias—injected in simulation according to real-world statistics
• Actions: control frequency, communication latency, tracking error, and gripper actuation delay
• Contact: randomization ranges for friction, mass, and elasticity should cover real-world measurements, not be arbitrarily widened
• Reset: the object pose distribution at the start of each trajectory should match how objects are actually staged on site

Evaluation should be bucketed, not reported as a single average success rate. At minimum, bucket by sub-skill, initial perturbation magnitude, occlusion level, and whether a contact interruption occurred. Long-horizon tasks should also track: the first failing step, recovery success rate, whether recovery introduced new violating contacts, and the number of human interventions. A high average success rate with no mid-task recovery capability means the system still cannot run unattended.

A real-hardware blind test protocol must be written in advance: no online recalibration, no manually repositioning objects (unless the protocol counts manual reset as failure), frozen sensor and policy versions, and replayable failure videos. Otherwise a “successful run” cannot be reproduced.

5. Data and Training Loop

Contact-rich skills depend on diverse but clean data. During collection, synchronously record images, proprioception, force/torque, gripper state, and timestamps, with aligned clocks. Demonstrations should cover near-failure samples—slight slips, off-center grasps, re-alignment—not just perfect trajectories. Policies trained only on perfect trajectories often fail to recover on real hardware.

Keep the same set of success predicates between training and deployment. For offline metrics, use simulation rollouts plus real-hardware spot checks; in production, run shadow-mode comparisons between “the model’s suggested subgoals” and “what the controller actually executes,” examine the divergence distribution first, then gradually expand skill-layer permissions. Any expansion should be instantly reversible to scripted, conservative actions.

6. Safety and Engineering Boundaries

Long-horizon manipulation will, by default, produce unexpected contacts. Workspace limits, force thresholds, emergency stops, and velocity scaling must be independent of the planning model. When the model times out, emits unparseable output, or detection confidence stays persistently low, the actuation side should move to a known safe posture rather than hold the last frame’s increment. Logging, tracing, and alerting enable post-hoc attribution of failures to perception, calibration, contact, or policy. Object storage is suitable for retaining aligned multimodal clips, while monitoring and alerting suit real-time notifications for force overruns and consecutive recovery failures. These are generic engineering capabilities that do not depend on any single model’s proprietary interfaces.

Team responsibilities should also be clear: robotics engineers own calibration, the safety layer, and real-hardware protocols; algorithm engineers own state predicates, skill policies, and bucketed evaluation; application teams own whether the task graph matches on-site cycle times. Treating “the model can manipulate objects” as the definition of done typically stalls at the first contact error.

7. Deployment Checklist

Before going live, confirm at least:

• Every subgoal has sensor-measurable success and failure conditions
• Contact errors are classified, and each class has a fallback state and a retry cap
• The safety layer does not depend on the task planner being alive
• Simulation randomization ranges are grounded in real-hardware measurements
• The blind-test protocol freezes versions and measures recovery quality, not just final outcomes
• Failure samples can be replayed down to the specific predicate, not just as a video

The core capability of long-horizon embodied manipulation is knowing, while the physical world keeps violating ideal assumptions, which step you are in, why you failed, and what the smallest safe recovery step is. Getting the state machine, layered control, typed error recovery, and reproducible evaluation right brings you closer to a maintainable system than chasing a single impressive demo.