A new automated framework called LongChang has been developed to assess the capacity of large language models (LLMs) to autonomously discover hidden physical laws by exploring simulated environments with altered physics. The framework, introduced by researchers from institutions including the Hong Kong University of Science and Technology (Guangzhou) and Zhejiang University, places LLMs in simulated worlds where one or more physical laws differ from those of our reality. These LLMs act as explorers, tasked with manipulating objects and constructing tools to investigate their own questions without any predefined research objectives or hints about the underlying altered laws.
The LongChang framework employs multiple LLM agents that work collaboratively. One agent generates and reviews simulated scenes, another reviews tool designs, and a third evaluates reports for correctness and experimental support, all while adhering to the altered physical laws and the record of exploration. This process allows the entire cycle to run without human intervention, guiding scene evolution for further exploration.
In tests involving 12 different LLMs, only GPT-6-Astra achieved a discovery rate of 16.7%. The remaining 11 models failed to discover any new physical laws, scoring 0%. Even for GPT-6-Astra, the discovery rate indicates only a partial identification of the altered physical laws, and exploration records revealed uninvestigated anomalies. The evaluation distinguished between correctly guessing a law and providing experimental support for it; models needed to satisfy both criteria to be considered successful. For instance, to differentiate between two potential explanations for an observed phenomenon, a model would need to design an experiment where the two explanations predict different outcomes.
While previous research has explored LLMs in scientific discovery, often providing them with established scientific literature or canonical problems, LongChang aims to evaluate a more fundamental capability: the ability to notice the unexpected and pursue it through investigation. Other related work includes frameworks that integrate LLMs into multi-agent systems for automated discovery of physical laws by combining literature review, hypothesis generation, and symbolic regression. Another project, PhyNex, uses an LLM-guided agent with computational tools to explore solution spaces for scientific tasks, matching or exceeding human-designed approaches in certain areas. Additionally, the Physics-LLM project is developing a toolkit to support researchers' workflows across the scientific discovery lifecycle using LLMs and agent-based AI. However, these efforts often involve more structured guidance or focus on known scientific domains rather than the open-ended, free exploration tested by LongChang.