A new generalist humanoid policy, VioLA, has been developed to enable robots to learn complex whole-body tasks from human demonstrations. The policy, detailed in a paper published on arXiv, addresses the challenges of large, coupled action spaces and the scarcity of robot-specific training data by predicting body and hand motion latents instead of direct joint commands. This method allows VioLA to leverage a vast pool of human motion data, significantly expanding its training resources.
The VioLA framework utilizes pretrained motion encoders that translate both human and robot movements into a shared latent space. This shared representation means that human video recordings can directly serve as supervision for the robot's actions. The training dataset for VioLA comprises 140.6 million frames, totaling 781 hours of data, with human recordings constituting 93.2% of this pool. This high proportion of human data shifts the bottleneck for humanoid training away from costly teleoperation towards readily available human video content.
On a real Unitree G1 humanoid robot, VioLA demonstrated 100% success in locomotion tasks and 88.6% success in manipulation tasks without requiring any task-specific fine-tuning. These tasks included walking, bending, sitting, closing a laptop, hanging a coat, and rotating a chair. In comparison, existing generalist policies like GR00T N1.7 and Ψ0 achieved 16.7% and 0% success, respectively, for locomotion, and significantly lower rates for manipulation.
The core of VioLA's innovation lies in its hierarchical decomposition of control. The generalist policy is responsible for predicting 128-dimensional motion latents, with 64 dimensions for body motion and 64 for hand motion. These predictions are then executed by frozen, pretrained controllers, such as SONIC for body control and a separate module for hands, which convert the latents into coordinated joint commands at a rate of 50 Hz. This separation means the policy does not need to learn lower-level aspects like balance, coordination, or individual joint control; instead, it focuses on selecting appropriate task-level motions.
The researchers highlight that the conventional approach of having a policy directly predict joint commands for humanoids often conflates the problems of coordination and task selection, making policies harder to train and more reliant on extensive robot-specific data. By abstracting joint-level control, VioLA simplifies the learning problem for the generalist policy.
A variant of VioLA trained solely on human demonstrations, without any robot data in the policy training phase, achieved 40% locomotion success on the real robot. This result indicates that encoded human motion can provide effective action supervision for real-robot behaviors. The researchers stated that code and checkpoints for VioLA will be released.