Researchers at the Massachusetts Institute of Technology (MIT) and New York University have introduced FLEX, an AI-assisted learning platform designed to help robots interpret human commands with greater nuance. The system allows robots to infer broader intentions from natural language instructions, moving beyond simply executing literal commands. This development could pave the way for more intuitive and interactive household and assistive robots.
FLEX operates through a three-step process to enhance a robot's motion-planning capabilities. The first step, "Add," involves the robot incorporating a feature or object mentioned in a user's command into its awareness. For instance, if a robot is instructed not to drop a bowling ball on a mug, the mug is added to a list of items to avoid. This initial step is similar to previous methods of robot instruction.
The subsequent "Generalize" step utilizes a large language model (LLM), such as GPT-4o, to infer broader rules or patterns from the specific command. The LLM, trained on vast amounts of text data, identifies commonalities and generalizations. For example, if a robot is told not to spill coffee on a laptop, the LLM can infer that liquids can damage electronic devices. This allows the robot to generalize the instruction to avoid other electronics, like a nearby phone, even if not explicitly mentioned. The final "Mask" step employs computer vision to remove objects from the robot's awareness that are no longer present in the environment, keeping the feature list current. This entire process takes approximately five seconds to complete.
The FLEX system was tested in simulated scenarios involving tasks such as serving coffee, moving a candle, and carrying a bowling ball. In these simulations, users provided natural language commands like "Don't spill on my laptop" or "Stay away from the stack of paper." The robots equipped with FLEX not only followed these literal instructions but also reasoned at a higher level to avoid related categories, such as other electronics, flammable items, or fragile objects, without explicit instruction for each. This approach contrasts with older methods that required users to manually add each object to avoid, which could become cumbersome.
The research team, led by Andreea Bobu from MIT's Collaborative Learning and Autonomy Research (CLEAR) Lab, aims to develop robots that can understand the underlying intentions behind human speech. This work is scheduled for presentation at the upcoming Conference on Robot Learning.