A new reinforcement learning framework, Accelerating Dexterity via Pre-Training (ADEPT), aims to enhance the ability of robots with many degrees of freedom to perform complex, contact-rich manipulation tasks. The framework, detailed in a paper published on arXiv, utilizes a two-stage approach: pre-training a dexterous policy in simulation on a generic object reposing task, followed by post-training for specific downstream tasks. This method allows robots to acquire skills that are otherwise challenging to learn from scratch, particularly for multi-fingered robot hands.
Dexterous manipulation with high degree-of-freedom (DoF) robotic systems, such as arms equipped with anthropomorphic multi-fingered hands, presents a persistent challenge in robot learning. These systems involve high-dimensional state and action spaces, coupled with frequent contact interactions, making it difficult to discover useful behaviors using traditional reinforcement learning (RL) methods. Even with large-scale GPU simulation, policies trained for one task often do not transfer effectively to others, necessitating a restart of the learning process for each new task.
ADEPT addresses this by first pre-training a foundational arm-hand-object dexterity on a generic object reposing task. This pre-trained policy can then generalize and solve the reposing phase of subsequent tasks without needing to be explicitly trained on them. However, directly fine-tuning this pre-trained policy with standard RL techniques often leads to a degradation of its capabilities. The researchers identified several reasons for this "catastrophic forgetting," including mismatched observation spaces, misaligned value estimates, and excessive policy drift during transfer.
To counteract these issues, ADEPT incorporates a specialized post-training framework. This framework combines behavior-cloning distillation to adapt the actor, frozen-actor critic warm-up to recalibrate the value function, and conservative on-policy updates to limit policy drift. This multi-pronged approach allows the system to specialize the pre-trained behaviors for contact-rich tasks without losing the foundational skills.
The framework also introduces a joint configuration-space Geometric Fabric controller. This controller is positioned between the RL policy and the robot to safely utilize the robot's full kinematic dexterity. Following post-training, the specialized "teacher" policy is distilled into a stereo-RGB "student" policy. This student policy is designed for zero-shot sim-to-real transfer, meaning it can be deployed directly in the real world without further fine-tuning.
The researchers demonstrated ADEPT's capabilities on a 23-DoF KUKA–Allegro arm-hand system. The system successfully performed a complex, long-horizon FMB peg insertion task directly from stereo RGB images, starting from various initial states. This task involved grasping, in-hand reorientation, and insertion, highlighting the framework's ability to manage intricate manipulation sequences.
The development of ADEPT aligns with a broader shift in robotics, where reinforcement learning is increasingly used for post-training and adaptation rather than training from scratch. This approach, which often starts with a pre-trained or imitation-trained policy, leverages rewards, human feedback, and on-robot data to refine and improve robot performance. Previous research, such as DexGraspRL, has also explored pre-training policies using imitation learning and then refining them with direct reinforcement learning in real-world scenarios. The ability to bridge the "reality gap" between simulation and physical robots remains a central challenge in robotics, with various methods, including domain randomization and neural-augmented simulations, being explored to improve sim-to-real transfer. ADEPT contributes to this effort by providing a structured method for transferring complex dexterous skills from simulation to real-world applications.