A new model called UniWALM, developed by researchers from Brown University, Amazon, Northeastern University, and Google Research, unifies language understanding, visual perception, and motor control for embodied robots. The model learns a shared representation across these three modalities, allowing them to inform and enhance one another. This contrasts with previous methods that treated these components separately.
Unlike prior architectures that might attach separate vision-language models to a robot's action backbone, UniWALM integrates all three within a single mixture-of-transformers architecture. This allows for synchronous denoising across modalities, meaning the model can process and generate language, action, and video data concurrently. The model's design enables it to decode subtask instructions and textual memory in the same pass as generating actions, eliminating the need for a distinct planning stage.
The researchers posed two key questions that UniWALM aims to answer. First, they investigated whether jointly generating language could enrich the shared representation and improve a robot's manipulation capabilities and support memory-dependent tasks. Second, they explored whether a robot's policy could use its own generated language, re-consumed across steps, as a form of working memory for tasks requiring recall.
UniWALM's approach treats each modality as both a conditioning signal and a supervision target. This means that language can guide action and perception, while actions and perception can, in turn, refine language generation. This bidirectional influence is central to the model's ability to create a more cohesive understanding of the robot's environment and tasks.
To ensure the reliability of the action stream within this richer objective, the researchers introduced a training objective called Flow Matching with Correction (FMC). This method trains the model to recover from perturbed action trajectories without incurring additional computational costs during inference.
In evaluations, UniWALM demonstrated consistent improvements over baseline models, including FastWAM, on compositional pick-and-place tasks in the MolmoSpaces environment. It also showed strong performance on the memory-dependent subset of RMBench, achieving 34.5% success without pre-trained language or reasoning models. This performance surpassed that of setups using external keyframe memory.
Furthermore, on a real-world "Find-the-Item" task, UniWALM's ability to generate textual memory significantly boosted success rates from 11.1% to 61.1%. This improvement was notable when compared to state-of-the-art vision-language-action (VLA) models, which performed at or below chance levels (<=33.3%).
The development of UniWALM represents a step towards "any-to-any" modeling across language, video, and actions, suggesting a future where robots can more fluidly integrate diverse forms of information for complex tasks.