Researchers have developed a new robot learning model, Zero-WAM, that can generalize to unseen manipulation tasks by learning from human demonstration videos. This system employs an in-context learning paradigm, a method previously successful in large language models, to enable robots to execute novel tasks without requiring task-specific training data or parameter updates. The model treats human videos as visual specifications for intended tasks, allowing a single policy to interpret these demonstrations and predict future actions.
The Zero-WAM framework is built upon a causal video-action model. Unlike traditional methods that rely on extensive datasets for each new task, Zero-WAM leverages human videos as a form of "in-context" guidance. This approach is particularly beneficial for robotic manipulation, where language instructions alone can be ambiguous. Human videos provide rich visual cues about the desired object interactions, the sequence of actions, and the temporal evolution of a task. The researchers generated a dataset of 74,200 human-robot in-context learning pairs across 8,600 tasks to train the model. This dataset, named HumanGen, was created by converting sampled robot trajectories into semantically aligned human video instructions.
A key challenge addressed by Zero-WAM is the scarcity of task-rich paired human-robot data. The team developed a pipeline to convert robot trajectories into human video demonstrations, effectively scaling the data needed for this in-context learning approach. This method allows the model to learn from a diverse range of tasks, enhancing its ability to generalize to entirely new scenarios. The model's training involves an "in-context future chunk prediction" objective. This specific objective helps prevent the model from relying too heavily on its own past actions or language instructions, thereby strengthening its ability to accurately interpret and follow the human video prompts for predicting longer-term task outcomes.
The researchers demonstrated Zero-WAM's capabilities in both simulated environments, such as RoboTwin 2.0, and real-world configurations. In simulations, the model achieved a 46.95% average success rate on seven unseen tasks, significantly outperforming existing leading baselines by up to 35.97 percentage points. This performance highlights the model's capacity for zero-shot cross-task generalization, meaning it can perform tasks it has never encountered during training. The system's ability to generalize extends to real-world scenarios without the need for collecting corresponding robot data or updating model parameters for each new task.
This work builds upon the concept of World Action Models (WAMs), which learn physical dynamics by predicting future world states and actions. Previous research in this area, such as DreamZero, has shown that WAMs can improve generalization by jointly modeling video and action. However, Zero-WAM specifically focuses on utilizing human videos as the primary in-context prompt for task specification, differentiating it from models that primarily rely on language instructions or purely self-generated data. The effectiveness of using human demonstrations as a visual interface for robots is further supported by other research exploring unified action geometries and cross-embodiment transfer, suggesting a broader trend towards more intuitive and visual methods for robot instruction.
The development of Zero-WAM represents a step towards more adaptable and general-purpose robots that can learn and execute a wide array of tasks with minimal human intervention beyond providing a demonstration. The system's success in zero-shot generalization suggests potential applications in complex manufacturing, logistics, and domestic assistance where task variability is high.