Can Robots Generalize Compositionally? Evaluating Compositional Generalization in Embodied AI

1. The Core Problem: Knowing Two Actions Does Not Mean Knowing Their Composition

In embodied AI, a common but easily overlooked question is whether a robot that has learned to “pick up a cup” and “open a drawer” can complete a new task such as “place the cup in the drawer and then close it” without dedicated training.

For humans, this is close to common sense. For an intelligent agent, however, it involves multiple capabilities, including skill representation, task planning, spatial reasoning, state tracking, and execution feedback. Success on an individual action only shows that the agent has local manipulation ability under specific observation and execution conditions. Success on a composite task requires the agent to understand action dependencies, changes in object state, and recovery strategies after failure.

Therefore, the ability to “create a fourth action” is not simply a matter of stacking actions together. It is a transferable capability: can the agent reorganize learned skill elements into a new task it has not seen before? This capability is commonly referred to as compositional generalization.

2. The Gap in Current Evaluation: One-Off Tasks Cannot Measure Transfer

Existing embodied AI benchmarks have traditionally focused on individual capabilities such as scene understanding, navigation, grasping, and manipulation. A typical evaluation gives an agent a task, records its execution, and measures success rate or completion time. This type of one-off evaluation can answer whether the agent performs well on the current task, but it does not clearly show whether previous experience helps it perform better on a new task.

Continual learning has received more attention in non-embodied settings, including question answering, code generation, and digital interface automation. These environments generally do not require an agent to handle three-dimensional space, physical collisions, viewpoint changes, or execution errors. As a result, their evaluation methods cannot be transferred directly to robotic scenarios.

This creates a gap: embodied agents need to accumulate experience over time, but the field still lacks evaluation systems that can measure how effectively that experience transfers. Addressing this gap requires converting the abstract concept of continual learning into an operational and measurable experimental process.

3. Using Compositional Generalization as a Measure

One practical approach is to first train an agent on a set of simple tasks so that it can acquire relevant skills. The agent is then evaluated on new tasks that have not appeared during training but are composed of previously learned skills. Performance before and after learning can be compared to determine whether the agent is actually using prior experience.

Task hierarchy is central to this approach. Tasks can be divided into three levels:

1. Atomic action level: A single object interaction, such as approaching, grasping, placing, opening, or closing. This is the smallest skill unit and can be further divided into perception, approach, and execution steps.
2. Basic task level: A practical task composed of two or three atomic actions, such as “place the cup on the table” or “open the drawer and retrieve the object.” These tasks introduce action ordering and state changes.
3. Compositional task level: A new task made up of multiple basic tasks or atomic actions, such as “place the cup in the drawer and then close it.” The specific composition does not appear during training, although its component skills have already been learned.

Training and test sets must be strictly separated. A test composition should not appear directly in the training tasks, while its component actions may appear at the atomic or basic-task level. This distinction helps separate memorization of task templates from genuine skill recombination.

4. Designing the Evaluation Protocol: Training, Testing, and Distribution Shifts

A rigorous compositional generalization evaluation should include at least the following stages:

• Skill learning stage: Train the agent on atomic actions and basic tasks. Record success rate, execution steps, and failure types.
• Compositional testing stage: Evaluate the agent on unseen compositional tasks and measure both completion rate and execution efficiency.
• Control experiments: Use a baseline without the relevant component skills to measure transfer gains. Also include a control group that has seen the same composition template to rule out simple memorization.
• Distribution-shift testing: Vary object locations, lighting, viewpoints, obstacles, and initial states to determine whether compositional ability depends on a specific environment.
• Forgetting evaluation: Re-evaluate atomic actions after compositional testing to determine whether previously learned skills have degraded.

Common metrics include compositional task success rate, generalization gap, transfer gain, step efficiency, invalid-action rate, recovery success rate, and forgetting rate. Metrics should be reported by task complexity rather than relying only on a single average, which can hide important failure patterns.

Here:

• Generalization gap is the difference between success rates on training tasks and compositional tasks.
• Transfer gain is the improvement achieved by learning the component skills compared with the baseline.
• Forgetting rate measures the degradation of previously learned skills after later training or evaluation.

5. Practical Engineering Recommendations

In real-world system development, the evaluation framework should be decoupled from the training pipeline. Tasks should use a structured description that specifies the objects, initial state, target state, permitted actions, and success criteria. Execution logs should retain observations, actions, state changes, and failure causes to support later analysis.

A compositional task should also not be reduced to a “long instruction.” A long instruction may simply contain more steps, whereas compositional generalization focuses on novel recombination of skill elements. The evaluation set should cover multiple types of composition, including sequential composition, conditional composition, parallel constraints, and failure recovery.

A simplified implementation is shown below:

def evaluate_compositional_generalization(agent, atomic_tasks, base_tasks, compositional_tasks):
    agent.train(atomic_tasks + base_tasks)
    base_metrics = agent.evaluate(base_tasks)
    comp_metrics = agent.evaluate(compositional_tasks)
    forget_metrics = agent.evaluate(atomic_tasks)
    return {
        "base_success": base_metrics.success_rate,
        "composition_success": comp_metrics.success_rate,
        "generalization_gap": base_metrics.success_rate - comp_metrics.success_rate,
        "forgetting": base_metrics.success_rate - forget_metrics.success_rate,
        "step_efficiency": comp_metrics.steps / comp_metrics.reference_steps
    }

6. Challenges and Future Directions

Compositional generalization evaluation still faces several challenges. The composition space grows rapidly as the number of skills increases, making exhaustive testing difficult. Real environments introduce noise and long-tail failures. Success criteria must be standardized across different tasks. In addition, further research is needed to determine whether an agent’s skill representation supports interpretable recombination.

The direction is nevertheless clear: only by turning “generalizing from one task to another” into a reproducible evaluation process can embodied AI move from demonstrations on isolated tasks toward long-term autonomous operation.

For developers and organizations, establishing task hierarchies, separating training from testing, and tracking transfer-related metrics early on will provide greater long-term value than focusing solely on success rates for individual tasks.