Researchers have developed a new system named RAPID (Robot Agentic Programming from Demonstrations) that can generate, verify, and refine robot programs from a single visual human demonstration. This framework aims to bridge the gap between AI coding agents, which have shown success in complex programming tasks, and robotic systems. RAPID automatically infers essential components for creating functional robot programs, including testable task specifications, executable action primitives, and an interactive environment for program execution and verification.
The core of RAPID's methodology lies in its object-centric relational program representation. This approach focuses on the underlying structure of a demonstrated strategy rather than a precise sequence of movements. Action primitives are expressed as trajectory-optimization programs designed to achieve object-level motion effects. These primitives are then composed through relational constraints that adapt to scene-specific geometry during runtime. This design allows the generated programs to generalize across variations in object pose, shape, material, and environment.
To create these programs, RAPID first reconstructs a 3D scene from a single RGB-D frame using vision-language models. This reconstructed environment serves as a sandbox for program verification. The system defines manipulation primitives, each with a semantic goal, a relational interface specifying involved objects and their geometric relations, a success predicate for autonomous verification, a geometry resolver, and an optimization cost. The framework then optimizes trajectories to achieve these goals.
Verification and refinement occur through the generation of "feasible scene variants." RAPID perturbs object poses and physical properties to create these variants. The coding agent iteratively executes candidate programs against these variants, diagnoses failures based on the defined success predicates, and refines the code until the strategy consistently succeeds across the synthesized verification set.
The researchers evaluated RAPID through experiments in simulation on eight contact-rich nonprehensile manipulation tasks and general prehensile manipulation tasks within the LIBERO-Pro benchmark. The system was also successfully deployed on a real Franka arm to perform the eight nonprehensile tasks. In all tested scenarios, RAPID demonstrated strong performance and generalization capabilities. The system's ability to generalize means that the generated programs are reusable and can adapt to new scenes with minimal adaptation, requiring only a "lightweight semantic binding" to associate objects in a new scene with the strategy's roles.