Training artificial intelligence systems that interact with humans often relies on multi-agent reinforcement learning, where one AI acts as a policy and another, typically a large language model (LLM), simulates user behavior. However, a recent paper published on arXiv, "One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL," demonstrates that this common approach systematically fails to generalize to real-world scenarios or unseen simulators. The researchers attribute this failure to "simulator collapse," a phenomenon where the simulator LLM becomes mode-collapsed, producing a narrow range of behaviors.
When an LLM policy is trained against such a restricted simulator, it overfits to specific strategies that exploit the simulator's dominant behavioral patterns. This overfitting results in a policy that performs poorly when encountering more diverse or unpredictable real users. The paper formalizes this collapse theoretically, providing a framework to understand how typicality bias in preference datasets contributes to mode collapse during LLM alignment. This bias causes the LLM to favor the most common or "typical" responses, suppressing the broader diversity it learned during pre-training.
To counteract simulator collapse, the researchers propose two complementary solutions: Verbalized Sampling at inference time and Co-Training during the training phase. Verbalized Sampling aims to broaden the simulator's behavior by encouraging it to sample from a more diverse, verbalized response distribution. Instead of generating a single "best" answer, the LLM is prompted to articulate multiple possible responses along with their associated probabilities. This technique forces the model to reveal its inherent uncertainty and tap into the wider range of ideas present in its pre-trained weights, thereby restoring diversity without requiring retraining. Experiments have shown that Verbalized Sampling can increase response diversity by 1.6 to 2.1 times over direct prompting, while maintaining or improving quality and factual accuracy.
The second proposed solution, Co-Training, involves a paradigm where multiple LLMs are trained together using multi-agent reinforcement learning. This approach moves away from relying on a single, static simulator. Instead, the LLMs generate their own responses and engage in discussions to collaboratively enhance the final output. A verifier then scores the final output, and these scores serve as rewards that are maximized through multi-agent reinforcement learning. This co-training process also includes incentives to encourage corrective and persuasive outputs during the discussions. Previous research, such as the MAPoRL (Multi-Agent Post-co-training for collaborative LLMs with Reinforcement Learning) framework, has demonstrated that multi-agent co-training can boost collaboration performance across benchmarks and generalize to unseen domains, unlike training individual LLMs alone.
The concept of multi-agent reinforcement learning itself presents challenges, including long-horizon decision-making and interacting with stochastic environments. Researchers have observed that vanilla adaptations from single-turn reinforcement learning methods often lead to instability and "Echo Traps," where agents overfit to locally rewarded reasoning patterns, resulting in repetitive and deterministic responses. Ensuring diverse initial states, multiple responses per state, and high rollout frequency are identified as key factors for stable multi-agent reinforcement learning.
The findings suggest that the diversity of simulated user behavior is critical for developing generalizable AI policies in human-AI interaction settings. Relying on a single, "frozen" simulator, even if it is a powerful LLM, limits the policy's exposure to the full spectrum of human responses, leading to brittle performance. The proposed methods, Verbalized Sampling and Co-Training, offer pathways to introduce the necessary diversity, allowing AI policies to learn more robust and adaptable interaction strategies.