A new benchmark named BEAKER has been developed to evaluate artificial intelligence agents intended for use in self-driving chemical laboratories (SDCLs). The benchmark, detailed in a paper released on arXiv, aims to assess how well AI systems can perceive, reason about, and act within a laboratory environment, considering physical, procedural, and safety constraints.
SDCLs represent a move toward embodied automation in chemical experimentation, where machines must not only understand scientific concepts but also execute physical tasks. While multimodal large language models (MLLMs) are seen as potential "brains" for these systems, existing benchmarks have largely focused on safety or general scientific knowledge, leaving the execution-oriented embodied capabilities underexplored. BEAKER seeks to fill this gap by providing a more comprehensive evaluation.
The BEAKER benchmark comprises 1,000 question-answer samples that have been curated by experts. These samples cover 11 different types of image and video tasks. The tasks are designed to test key judgments required for chemical laboratory execution, including grounding visual information to physical actions (affordance grounding), planning movement trajectories, understanding the state of equipment and materials, connecting apparatus, reasoning through operational processes, and interpreting experimental results.
All questions and annotations within BEAKER were created and reviewed by human experts in both chemistry and embodied AI. The development process deliberately excluded AI-assisted generation or annotation to ensure the quality and integrity of the benchmark.
The researchers evaluated 28 different MLLMs on the BEAKER benchmark. These models included general-purpose, scientific, chemical, and embodied MLLMs. The results indicated that while current AI models can recognize laboratory objects and understand states, they still struggle with grounding experimental knowledge into actionable tasks. Specifically, they face difficulties in identifying operable regions, planning feasible movement paths, and performing long-horizon procedural reasoning.
BEAKER's findings highlight a significant gap between the perception abilities of multimodal models in laboratory settings and their capacity for action-grounded experimental understanding. The benchmark is intended to serve as a diagnostic tool for future development of MLLMs intended for SDCL applications.
Self-driving laboratories are an emerging area in chemistry and materials science, aiming to accelerate discovery through integrated AI, robotics, and digital technologies. These automated systems can reshape research and development pipelines, potentially speeding up the time to market for new products. Benchmarking efforts in this field are crucial for quantifying the acceleration of learning and reducing the number of experiments needed to achieve specific goals. Previous work has explored various aspects of autonomous labs, including human-robot interaction in shared spaces and the use of Bayesian optimization for experiment selection. [cite:3, cite:2] However, the specific evaluation of embodied AI agents' ability to perform complex experimental reasoning and actions within these environments remains an area requiring further development.