The Dex-One2Many framework, detailed in a paper published on arXiv, enables robots to acquire dexterous manipulation policies that generalize to new object poses, goal positions, and grasps, even when these were not present in the initial human demonstration. Traditional approaches often rely on strict motion imitation, which limits a robot's ability to adapt to varied scenarios. Conversely, pure reinforcement learning (RL) can generalize broadly but struggles with the extensive exploration required for high-dimensional, multi-stage tasks.
Dex-One2Many bridges this gap by abstracting a human demonstration video into sequential scene graphs. These graphs serve two primary functions: they act as generative constraints for sampling diverse reset states within a simulated environment, and they provide dense rewards for each stage of a manipulation task. By constraining relations rather than precise poses, the reset states generated cover a wider range of object poses and grasps than those shown in the original video. Initializing each stage from these states with dense rewards streamlines the exploration process, making it more efficient and guided.
The framework operates on a real-to-sim-to-real pipeline. The initial human video demonstration is used to create a simulated environment where the robot learns. This learning occurs entirely within the simulation, which then allows for zero-shot transfer to a physical multi-fingered robot hand. In tests across five different tool-use and manipulation tasks, Dex-One2Many outperformed baseline methods. It showed a 6.5% improvement in configurations that were similar to the demonstration and a 71% improvement in scenarios that were entirely new to the robot.
This research builds on previous work in learning dexterous manipulation from human data and real-to-sim-to-real pipelines. Other frameworks like DexMan and DexWeave have also explored similar concepts, focusing on converting human visual demonstrations into robot skills and transferring human motion and interaction structures. The core challenge in this field remains the "reality gap," the discrepancy between simulated physics and real-world conditions, which can hinder the transfer of learned policies to physical robots. Dex-One2Many's approach to guiding RL through abstracted scene graphs offers a method to enhance both exploration efficiency and generalization capabilities, contributing to the ongoing effort to make robots more adaptable in complex manipulation tasks.