Real-World Experiments
Each task is evaluated under No-Change, Background, Light, and Camera Viewpoint scenarios.
(a), (b). The IL policy fails to complete manipulation tasks (e.g., open the drawer) under visual domain shifts (e.g., background color changes). (c). StructPolicy first uses StructCon to construct a Structure Map that encodes fine-grained object structures invariant to visual domain shifts. It then leverages StructEncoder to extract domain-invariant map features that indicate how objects are structured and where them can be manipulated. By providing these features to the IL policy, StructPolicy equips IL policy with a comprehensive understanding of objects to enhance the robustness against visual domain shifts, thereby improving manipulation accuracy.
Abstract
Imitation Learning (IL) offers an effective approach for robot manipulation by learning a mapping from visual inputs to actions. However, this paradigm suffers from poor robustness under visual domain shifts (e.g., lighting conditions, camera viewpoints, etc.), often failing to perform manipulation. Our key insight in tackling this problem is to construct a Structure Map encoding the fine-grained object structures that remain invariant across visual domains and vital for manipulation. Based on this insight, we propose StructPolicy, a method elaborately designed to incorporate the domain-invariant Structure Map into the IL policy through two modules: StructCon and StructEncoder. StructCon is an automated Structure Map construction module that leverages structure primitives to enable flexible transformation and composition to form a wide range of objects. StructEncoder is a hierarchical network that efficiently captures structural relationships and affordance from Structure Map. This provides the IL policy with both domain-invariant and structurally-aware features, guiding it toward robust manipulation under visual domain shifts. We extensively evaluate StructPolicy across 49 manipulation tasks in multiple benchmarks and diverse real-world tasks under various visual changes. The results demonstrate consistent and significant performance improvements across all tasks, validating that StructPolicy enhances the effectiveness and robustness against visual domain shifts of the IL policy, improving manipulation accuracy.
Overview of StructPolicy. StructPolicy serves as a plug-and-play module that enhances standard imitation learning (IL) policies with StructCon and StructEncoder. Given a task description, an RGB image of the environment, and IL policy input, the StructCon (a) first constructs a Structure Map that encodes object structures and derived affordance. This Structure Map is then processed by the StructEncoder (b) to extract two types of features: the structure feature ($f_s$) and the corase gripper pose feature ($f_p$). The resulting features are fused into map features $f_m$, which are concatenated with the visual feature $f_v$ from the IL policy (c) to predict the final action.